DEV Community: Paul Chen

Harnessing the Nondeterministic Machine: Practical Lessons in Agentic AI Hardening

Paul Chen — Fri, 16 Oct 2026 18:17:31 +0000

There is a specific kind of confidence failure that shows up in agentic AI systems. It is not the dramatic hallucination - the confident fabrication of a person who never existed. It is quieter than that. The model understands the task. It understands the data. And then it decides it has a better idea.

We ran into this repeatedly while building the content health maintenance workflow for Synthadoc, an open-source AI-native wiki engine. The workflow's job sounds modest: scan wiki pages for structural problems like duplicate headings or thin sections, propose targeted fixes, and let the user approve them before applying anything. What we found, and eventually fixed, is a catalogue of ways an agent can go wrong even when the model is capable and the intent is clear - and why the solution is never just a better instruction.

The Problem We Were Solving

Synthadoc ingests raw documents - PDFs, web pages, transcripts, spreadsheets - and uses an LLM to synthesize them into a structured Markdown wiki. Over time, the same topic gets ingested from multiple sources. Without intervention, a single wiki page about Alan Turing can accumulate ten copies of ## Bletchley Park and Cryptanalysis and eight copies of ## Early Life and Education. The page is coherent in isolation but structurally broken: duplicate content, contradictory emphasis, and thin sections that exist only because a heading was created before the body text was written.

The content health scanner addresses this. It is deliberately a pure-Python heuristic with no LLM dependency - it detects duplicate_section, empty_section, thin_section, unclosed_code_block, orphaned_heading, and broken_list_nesting through deterministic regex. This part is reliable. The next part - using an LLM to fix the issues - is where the story gets interesting.

What We Intended the Workflow to Do

The design was explicit. For each flagged page:

Load the page content and the list of detected issues
Send both to the LLM with a carefully constructed prompt
Show the user a side-by-side BEFORE/AFTER diff
Wait for approval before writing anything
Snapshot the previous version so a rollback is possible

Every step in this design has a purpose. The BEFORE/AFTER review is a human gate, not a formality. The snapshot is a reversibility guarantee. The heuristic scanner runs before the LLM so the issue list is precise, not speculative.

What actually happened when we first ran it told us exactly which parts of our hardening were missing.

What Went Wrong

Failure 1: The Model Rewrote the Page

The first problem emerged immediately. The LLM received the page content and a list of issues. There were twelve structural problems on the page about the Open Source Movement - three duplicate sections and nine thin sections. The AFTER content we received back bore no structural resemblance to the original. The original heading ## Richard Stallman and GNU (1983) was gone. The heading ## The Cathedral and the Bazaar (1999) was gone. In their place: ## Historical Framing and Origins, ### Research Computing Culture and Principled Advocacy, ### GNU Toolchain and the Missing Kernel. The content was factually reasonable. It was also something the user never asked for and could not safely approve, because we had lost all traceability between the original and the proposed version.

The model had, in effect, written its own wiki page from its pre-trained knowledge, using our wiki page as a rough reference. The word "fix" in our prompt was interpreted by the model as license for rewriting.

Failure 2: The Model Kept the Wrong Duplicate

For a page about computing history, the scanner flagged ## Moore's Law appearing three times. Two occurrences were thin summaries of about forty words each. The third was a detailed paragraph with four footnote citations to the source material. Our initial prompt said "keep the first occurrence." The model faithfully kept the first - one of the forty-word summaries - and discarded the version with citations.

Technically correct. Practically, the wrong choice. We had a rule that was precise but not meaningful for the actual goal.

Failure 3: CLI-Provider Artifacts in the Proposed Content

Synthadoc supports multiple LLM backends. One of them is a CLI-based provider - an agentic tool (like opencode or Claude Code) that is itself an agent with its own tool-calling loop. When we routed the content health workflow through this provider, the response text included shell command output: <|func_arg_name|>command<|func_arg_value|>cat /Users/user1/wikis/alan-turing.md. The model had read the actual file from disk, bypassing our carefully constructed context, and its internal tool invocations leaked into the response text we were about to write to the wiki.

Failure 4: Planning Notes Presented as Final Content

In some runs, the proposed content contained lines like:

### LISP (1958) - actually LISP is more third gen but...
### Introduction paragraph (existing)
Expand to 50+ words

These were the model's internal planning notes - draft scaffolding that it had not cleaned up before terminating its response. These notes would have been written to the wiki without detection. It is possible that a user reviewing a diff in a modal at speed might miss them.

Failure 5: Heading Renamed to "Correct" a Date

After tightening the prompt, one run still returned a subtle violation. The original heading was ## The Cathedral and the Bazaar (1999). The proposed content had ## The Cathedral and the Bazaar - the year dropped. The model "knew" the essay was presented in 1997 and silently corrected what it interpreted as an error in the user's own heading. It was an editorial judgment we had not authorized.

The Failure Mode Map

These failures cluster into two categories that are worth naming precisely.

Category A - DRIFT: Model understands task but applies wrong judgment
Category B - HALLUCINATION: Model substitutes its own knowledge for the task

It is important to distinguish. As a result of drift, the right process is being used to make wrong decisions - "keep the first" instead of "keep the best." As a result of hallucination, the right process is being replaced completely - generating a page from weights instead of fixing the one we hand it. Different mitigations are required for each.

How We Hardened the System and Built the Harness

These fixes fall into two distinct layers. Hardening addresses what the model produces: tighter prompts, output parsing, provider-specific guards. Harness engineering addresses what the model can affect: human gates, rollback, scope exclusions. Hardening reduces the probability of a bad output; the harness limits the consequence of one that gets through. Both are required - hardening alone cannot guarantee zero failures, and a harness without hardening just moves the review burden to the human gate.

Hardening the Agent

1. Artifact Extraction as an Input Boundary

The CLI-provider issue required a structural fix, not a prompt fix. We introduced _extract_md_from_response() - a pre-processing step that strips everything before the first # H1 heading in the response. CLI-provider tool call output always appears before the actual content begins. By treating the first H1 as the clean boundary, we recover the markdown content without the artifact noise.

This is an example of boundary enforcement at the output interface: treat the model's raw response as untrusted input to a parser, not as a value ready to write.

2. Draft Detection as a Post-Output Gate

For planning notes, we added _is_draft() - a regex that catches linguistic markers of incomplete thinking: TODO, FIXME, Expand to, Hmm,, Wait,, I'll, Let me, existing content. If any match, the proposed content is flagged and the page is skipped automatically with a clear explanation in the UI.

_DRAFT_RE = re.compile(
    r"^\s*(TODO|FIXME|Expand to\b|Hmm[,.\s]|Wait[,.\s]|I'll\b|I will\b|Let me\b"
    r"|existing content|combine existing|add more content|placeholder)",
    re.IGNORECASE | re.MULTILINE,
)

This is an output validation gate - a deterministic check applied after the model responds and before the result reaches the user approval screen. The model cannot disable it by rephrasing the note.

3. Surgical Prompt Constraints

The original prompt said: "Fix the structural issues listed below." That sentence, to a language model, is nearly unlimited scope. We replaced it with an explicit constraint hierarchy:

ABSOLUTE RULES - violating any one is a failure:
  • PRESERVE every existing section heading EXACTLY - same text, same # level,
    including any dates or years in parentheses. Do not 'correct' headings.
  • PRESERVE the section ORDER exactly as in the original.
  • Do NOT add new top-level or second-level sections.
  • Do NOT rename, merge, split, or reorder any sections.
  • Do NOT alter factual claims in sections you are not fixing.

PERMITTED CHANGES (only these, only where the issue list directs):
  1. Duplicate sections: keep the most complete occurrence (most words and/or
     citations); delete the others. Place the kept version where the first
     occurrence was. Do NOT merge or rewrite - keep it verbatim.
  2. Thin/empty sections: add factual prose UNDER the existing heading.
     The heading text must remain unchanged.

Changing from "fix issues" to "permitted changes" is intentional. The model cannot rationalize around a list of what it is allowed to do when permissions are enumerated rather than prohibitions.

4. Issue Count Threshold for Nondeterministic Providers

Even with the tighter prompt, pages with more than six issues on a CLI-provider run produced hallucinated new sections. The CLI provider is itself an agentic tool with its own reasoning loop - it does not behave like a simple completion endpoint, and the compound nondeterminism (our model + its internal model) overwhelms a long issue list.

We added a hard skip at six issues for the CLI path:

# CLI providers (opencode, claude-code) are agentic - they read files,
# plan before writing, and hallucinate new sections when the issue list
# is long. API providers follow the prompt reliably and don't need this
# guard. This path is CLI-only, so the cap applies only there.
_MAX_ISSUES_AUTO_FIX = 6

API-key providers (Anthropic, OpenAI) use a separate code path with no cap, since they are stateless completion endpoints that see the prompt exactly as we construct it. Rather than a model capability, this is a property of the architecture.

Building the Harness

5. System Page Exclusion

Thin sections on pages named index and dashboard were correctly identified by the scanner. These are navigation pages, not content pages - thin sections are structurally appropriate. The workflow would propose expanding them without exclusion, which the user would reject correctly, wasting their time.

We use the existing SYSTEM_PAGE_SLUGS constant to exclude them from the fix loop. No duplicate definition, no magic string. The scanner still runs on them - the issues appear in lint reports - but the auto-fix workflow skips them.

6. The Human Gate as a Structural Requirement

Every proposed fix must pass through a BEFORE/AFTER diff review before anything is written. This is not a feature - it is the architectural guarantee that no automated process can corrupt wiki content without explicit consent. The confirm card shows:

The page slug and issue summary
BEFORE (first 2000 chars of the original)
AFTER (first 2000 chars of the proposed content)
"Approve" / "Skip this page" buttons with a 5-minute timeout (auto-decline)

The scrollability of code blocks in the review modal was a concrete UX issue we had to address - users need to be able to read what they are approving. Gates that are theoretically present but practically unusable are not gates.

7. Snapshot and Rollback as a Safety Net

Before any approved fix is written, snapshot_if_changed() takes a versioned copy of the original content. If the approved result turns out to be wrong, the user can request a rollback by name: "roll back alan-turing." The workflow restores the snapshot, re-runs the scanner, and updates the issue list to reflect the restored state. It is not a rollback to recover from system failure - it is a first-class workflow feature that acknowledges the model may produce something subtly wrong that was not detected during review.

The Broader Principle: Both Layers Are Architectural, Not Textual

The pattern we followed across all of these fixes is consistent: every control - whether it hardens the agent's output or constrains the harness around it - is implemented in code, not in prose.

"Do not hallucinate" in a system prompt is a behavioral suggestion. The model considers it, weighs it against other pressures, and sometimes overrides it. That is not a flaw, it is how language models work. The token prediction process does not have a hard stop for "this would be a hallucination."

Both hardening and harness engineering mean building controls that are not subject to the model's interpretation:

Layer	Control	What it does	Why it cannot be bypassed
Hardening	`_extract_md_from_response()`	Strips pre-H1 content	Runs on raw response bytes, before any model interpretation
Hardening	`_is_draft()` regex	Blocks planning-note content	Deterministic regex match - not in the model's loop
Hardening	PERMITTED CHANGES prompt	Enumerates allowed scope	Narrows the action space; drift must contradict the list
Hardening	Issue count cap	Skips complex pages	Hard code gate; model never sees the page
Harness	System page exclusion	Skips non-content pages	Slug lookup against a constant; not in the prompt
Harness	BEFORE/AFTER human gate	Requires user approval	No write path bypasses this
Harness	Snapshot / rollback	Preserves prior state	Written before the fix is applied

These controls form layers. A fix that evades the prompt constraints is caught by the draft detector or the human gate. A fix that passes the human gate can be rolled back. No single layer is sufficient. The layers together are.

Why This Matters Beyond Wiki Maintenance

Incorrectly updating a wiki page is a recoverable mistake. A user clicks rollback, the page is restored, and no lasting harm is done. But the failure modes we observed - a model substituting its own knowledge for the task, a model including its internal tool calls in the output, a model silently correcting data it considered wrong - are not domain-specific. The same failure patterns appear in:

Agentic coding tools that refactor code the user did not ask to touch
Document automation systems that "improve" a legal clause beyond the approved scope
Data pipelines where an LLM normalizes values using world knowledge rather than the provided schema
Financial report generators that extrapolate figures not present in the source data

In each case, the consequence of unhardened agentic behavior scales with the stakes of the domain. A hallucinated section in a personal wiki is recoverable. A hallucinated figure in a financial disclosure is not.

The argument for hardening is not that models are bad. It is that models are nondeterministic, and production systems require deterministic guarantees at the boundaries where AI output meets persistent state.

What to Apply From This

If you are building an agentic system that reads and modifies persistent data, consider these as minimum requirements:

Boundary enforcement. Treat the model's output as untrusted input at every handoff. Parse it. Validate it. Strip what does not belong before it reaches the next step.

Deterministic post-output gates. Add validation logic that runs after the model responds and is not subject to the model's behavior. Draft detection, schema validation, output format checks. These are deterministic; the model cannot reason around them.

Enumerated permissions over enumerated prohibitions. In prompts, tell the model what it is allowed to do, not only what it must avoid. The scope of permitted actions is easier to reason about than an open-ended task with a list of exceptions.

Acknowledge provider heterogeneity. Not all "model" configurations behave the same way. A CLI-based agentic provider and a stateless API completion endpoint have fundamentally different behavioral properties. Design your hardening layer to account for both, or constrain which providers can reach which actions.

Human gates as structural requirements. For any workflow that modifies persistent state, a human review step is not a UX nicety - it is a control. Make it usable. If the review UI cannot display the full diff, the gate is not functioning as intended.

Reversibility as a first-class design constraint. Snapshots and rollback should be designed in from the start, not added after the first production incident. Assume the model will sometimes produce a subtly wrong result that passes review, and design the recovery path accordingly.

Closing

It took multiple iterations for Synthadoc's content health maintenance workflow to behave predictably. Rather than rephrasing the prompt, each iteration addressed a specific failure mode. As a result, the team trusts the workflow - not because the model is perfect, but because the system is designed to catch and contain errors.

Synthadoc is open source. The content health scanner, the resolver workflow, and the draft detection logic are all available at Synthadoc git repo. If you are building AI-native systems that write back to persistent state, the patterns described here are worth examining in the context of real code, not just as abstractions.

The nondeterministic machine can be harnessed. The harness is engineering, not prompting.

Four Platforms for Developer Writing: What Each One Does Best

Paul Chen — Sun, 27 Sep 2026 18:26:36 +0000

Four Platforms for Developer Writing: What Each One Does Best

I've published on LinkedIn, Medium, Dev.to, and CoderLegion. People regularly ask me which platform to use. The honest answer: it depends on what you're trying to accomplish. Here's how I think about each one.

LinkedIn - Entry Point, Not Destination

LinkedIn is best used as a conversation starter. Put a provocative question, a key takeaway, or a short observation in the post, then include the link to the full article in the first comment. This uses LinkedIn's strengths - professional reach, cross-border visibility - without asking it to do what it's bad at: hosting deep technical content. Think of it as the top of a funnel, not where the real discussion takes place.

Medium - for Storytelling and Long-Form Thought Leadership

Medium is ideal for writing that's meant to endure: career reflections, architectural post-mortems, in-depth essays. The Partner Program adds a monetisation path. The main challenge is discoverability, posts get buried quickly unless you're already in a publication or have an existing audience driving traffic. Medium rewards writers who bring their own readers; it does little to surface new voices on its own.

Dev.to - for Tutorials, Community Discussion, and Learning in Public

Dev.to is one of the most developer-focused writing platforms available. It's open, free, and has a genuinely engaged community that spans everything from students to senior engineers. The tag system makes technical content easy to find, and posts are indexed quickly by search engines, which gives your writing a longer shelf life than most social feeds.

The real strength is the format fit: "how I built X," debugging stories, beginner tutorials, TIL posts, and learning-in-public threads all feel natural here. Series support lets you structure multi-part content cleanly. Cross-posting with a canonical URL is well-supported, so you can syndicate from your own blog without SEO penalty.

The trade-off is volume - the platform is active, so standing out requires either strong tags, an early engagement spike, or consistent posting over time. That said, the comments section tends to attract thoughtful responses from people who actually read the post, which is not something you can take for granted elsewhere.

CoderLegion - for Early Visibility Without an Existing Audience

CoderLegion's Trending and Promotion tags give newer posts and newer members real home-page visibility from day one, which most platforms don't offer. People tend to actually read and comment rather than scroll past, which makes it a good place to get early feedback on an idea before you've built an audience.

The Developer's Journey sharing group is worth highlighting specifically: it's designed for developers at any level to share progress and learn from each other's experiences, which creates a supportive environment that's harder to find on larger platforms.

If you're just starting out or want genuine peer feedback rather than chasing an algorithm, CoderLegion is worth trying.

My Rough Rule of Thumb

Role	Platform
Entry point / professional amplifier	LinkedIn
Technical tutorials, how-tos, learning in public	Dev.to
Long-form essays, thought leadership	Medium
Community feedback + early visibility	CoderLegion

Most experienced developers end up cross-posting to two or three. Write once, adapt the framing, post everywhere that fits.

Your thoughs

Where do you prefer to publish? I'm curious which platform gives you the most useful engagement, drop it in the comments.

30 Domain Templates for Your LLM Knowledge Base: Skip the Setup, Start Capturing

Paul Chen — Tue, 15 Sep 2026 01:17:15 +0000

The hardest part isn't the LLM layer

The hardest part of building an LLM knowledge base for a specific domain isn't wiring up the models, it's encoding how that domain thinks. A compliance analyst's knowledge base needs every obligation mapped to a specific regulation, every control to an owner, every exception to an approver. A clinical wiki needs evidence grades on every claim. A DevOps runbook system needs incident timelines and rollback steps. The query routing, intake form fields, and quality standards are all different — and none of that comes pre-configured.

Most teams spend the first two weeks not capturing knowledge but arguing about structure: what pages should exist, how routing should decompose, which source types need their own forms. By the time those questions are settled, content has already been ingested into the wrong shape and needs to be redone.

synthadoc install --template solves this in one command.

What a Domain Template Actually Provides

A domain template is not a demo or a sample dataset. It is a pre-made structural skeleton for a specific knowledge domain - everything that governs how knowledge is organized, ingested, and queried, without any of the actual content.

One install command gives you six structural components:

1. Query routing table (ROUTING.md)

The map that tells the query engine which branches of your wiki to search based on the nature of the question. A finance/investment template routes "capital structure questions" to debt financing and covenant pages, while routing "valuation questions" to financial models and comparable companies. Writing this from scratch requires domain knowledge most developers building a knowledge system do not have — and the gap shows up as poor retrieval quality until someone with that expertise intervenes.

2. Domain-tuned agent guidelines

Each template ships with CLAUDE.md, AGENTS.md, and GEMINI.md in the wiki root — the standard files that AI coding assistants (Claude Code, Gemini CLI, GitHub Copilot Agents) read automatically when active in a directory. The domain guidelines are injected into each file, so any AI tool you already use becomes domain-aware the moment it opens the wiki.

Here is what the clinical healthcare template puts in those files:

Cite clinical guidelines by issuing body, title, year, and recommendation strength (Grade A/B/C or Level 1/2/3). Note evidence quality for every clinical claim: RCT, meta-analysis, cohort study, expert consensus, or case report. Never store identifiable patient data in wiki pages.

Compare that to the compliance template:

Map every compliance obligation to the specific regulation, rule, or contractual requirement that creates it. Track controls with the control owner, testing frequency, and last test date. Distinguish between legal requirements (non-negotiable) and internal policy requirements (adjustable). Note jurisdiction and regulated entity for every obligation.

When a compliance analyst has Claude Code running in their wiki directory and asks it to help write a page or interpret a lint warning, it already knows that regulatory citations need jurisdiction-level specificity and that control owners must be named — without configuring anything.

3. Scaffold stub pages

8–12 stub pages covering the core knowledge areas of the domain — empty frames with the right titles, frontmatter, and a  zone marker that separates content you write (preserved) from content the weekly scaffold job refreshes (generated). The structural skeleton maintains itself without ever overwriting your edits.

4. Blank intake forms (template-*.md)

Structured markdown forms in raw_sources/ for each document type common to the domain. A finance/investment install includes forms for deal memos, company profiles, and due diligence checklists — the same fields a professional would populate in any standard memo, now wired directly into the ingest pipeline.

5. Pre-validated seed sources (seeds.md)

Curated ingest commands for public-domain reference sources - all checked for accessibility and domain appropriateness before the template was released. For finance/investment, these include SEC investor education materials and Federal Reserve H.15 interest rate releases. For healthcare/clinical, clinical guideline registries and evidence-based medicine references. Real domain context before your proprietary content lands.

6. Scheduled maintenance jobs

A weekly lint run (Sunday 2:00 AM) and a weekly scaffold job (Sunday 3:00 AM), pre-registered at install time. Neither requires manual intervention once the wiki is running.

Template Library: 30 Domains, 9 Categories

The selection reflects domains where knowledge is both high-stakes (wrong answers have real consequences) and naturally document-heavy (the source material already exists - it is just not queryable). Finance, healthcare, and legal are the clearest examples. Technology and research are the next tier, where the cost of an engineer not knowing the current incident runbook, or a researcher duplicating a failed experiment, is concrete and measurable.

From Install to First Query

Getting a domain-template wiki to the point where it can answer real questions takes less than an hour.

Step 1: Install

synthadoc templates list          # browse all 30 options
synthadoc install my-wiki --target ~/wikis --template finance/investment
synthadoc serve my-wiki

The install creates a standard wiki then applies the template: routing table, agent guidelines, stub pages, intake forms, and seeds. Staging is enabled by default — every ingested page goes to a candidates queue for review before entering the live wiki.

Step 2: Ingest seed sources and your own documents

# From seeds.md — authoritative public reference material
synthadoc ingest "https://www.investor.gov/introduction-investing" -w my-wiki

# Your own sources — batch ingest everything in raw_sources/ at once
synthadoc ingest raw_sources/ --batch -w my-wiki

Step 3: Review candidates in Obsidian

Open the Obsidian plugin and run Synthadoc: Candidates: review candidate pages... from the Command Palette. The modal lists every staged page with its slug, colour-coded confidence badge (high / medium / low), and ingest timestamp — promote what passes your bar, discard what does not.

Step 4: Run lint

synthadoc lint run -w my-wiki

Lint validates every page against the domain guidelines, checks citations, and flags orphans and contradictions. Clean draft pages are promoted to active automatically. After lint, your wiki is queryable.

What Happens After

Once the wiki is active, maintenance is driven by the web UI rather than by you remembering to check things. After every query or lint run, the engine reads its own output and pre-fills the input box with the highest-priority next action — if it found stale pages, the prompt says "Re-ingest these pages"; if re-ingest just finished, it says "Run lint to promote them"; if contradictions surfaced, it queues the resolver. You confirm and it runs. No command to remember, no separate dashboard to check.

Behind that interface, the agentic layer handles:

Contradiction resolution — conflicting pages are identified, original sources retrieved, resolution proposed interactively
Re-ingest on source updates — stale pages refreshed with a before/after diff for your approval
Broken wikilink repair — dead links scanned and repaired automatically
Citation integrity — claims audited against source documents; drift flagged for re-ingest
Sensitive data handling — sanitizer strips PII at ingest time; synthadoc retract rolls back a source and all pages citing it if something slips through
Scheduled lint and scaffold — weekly jobs keep the knowledge graph healthy without intervention

The goal: after the first few days of populating your wiki, the agentic layer handles the maintenance work that would otherwise require a dedicated knowledge engineer.

Quick Reference

synthadoc templates list                          # browse the library
synthadoc install <name> --template <cat/domain>  # install with a template
synthadoc templates sync -w <wiki>                # sync after an upgrade

Release notes (v1.3.3)
Full template docs: template/readme in the installed package

Synthadoc is an open-source LLM knowledge compilation engine, it ingests your sources, compiles them into a queryable, citation-backed wiki, and keeps it consistent and self-growing over time. The domain template library ships in every install, no additional packages required.

Hallucination Is an Architecture Problem, Not Only a Prompt Problem

Paul Chen — Fri, 28 Aug 2026 19:06:47 +0000

If you've built a knowledge base on top of an LLM, you've probably noticed something counterintuitive: the outputs rarely contain flat-out fabrications. What they contain is drift — the model summarizes accurately but slightly overstates a confidence level, rounds a number, or drops a qualifier. "As of 2023" quietly becomes "currently." A causal relationship is inferred from correlation.

These are small errors individually. Across a knowledge base that gets queried repeatedly, they compound.

Prompt instructions help. Don't make things up, answer only from the provided context, never fabricate — these do nudge model behavior in the right direction. We use them too. But they leave enforcement entirely up to the model doing the generating. A language model producing text token-by-token has no reliable internal mechanism to audit its own faithfulness to a source.

We built Synthadoc — an LLM-based knowledge base compiler — around a different bet: hallucination is an architecture problem, not only a prompt problem. A prompt fix is hoping the model behaves. An architecture fix is verifying that it did.

Three Layers

The approach has three distinct layers, each targeting a different failure mode:

Domain scoping at ingest — Only on-domain content enters the knowledge base. Out-of-scope material never becomes a wiki page, so it can never become the basis for an answer.
Retrieval-grounded generation — Answers are built from retrieved pages with an explicit positive constraint: answer from only these pages, cite every claim. Positive constraints ("do only Y") are harder to rationalize around than negative ones ("don't do X").
Citation faithfulness audit — A separate LLM pass examines each claim against the source lines it cites and classifies the relationship.

None of these is a complete solution alone. Together they make hallucination detectable and traceable rather than just discouraged.

Layer 1: Content Gate at Ingest

Every wiki in Synthadoc has a purpose.md that defines its domain. When a new source arrives, the ingest agent reads this before deciding what to do with it:

Wiki scope (from purpose.md):
{content of purpose.md}

action="skip" means the source is completely OUTSIDE the wiki's domain.
…

The agent then chooses: skip, create, update, or flag. Off-domain content is dropped at the door.

When sources conflict, the flag path runs: the ingest agent issues a flag action and the page moves to a contradicted lifecycle state, automatically excluded from retrieval until the conflict is resolved. A contradicted page is one where the system can't be confident what's true — serving answers from it would be hallucination with extra steps.

Resolving a contradiction requires an explicit human approval. An interactive agentic workflow surfaces the conflict, shows a diff of the proposed edit, and blocks writing until a human accepts it. The gate is unconditional.

Layer 2: Grounded Generation

At query time, Synthadoc retrieves the most relevant pages using hybrid search (BM25 + optional vector re-ranking) and builds a synthesis prompt that includes those pages verbatim:

Answer using ONLY these wiki pages. Cite with [[PageTitle]].
Extract and include all specific facts — dates, years, numbers, and names —
even when they appear briefly or in passing.

Question: {question}
Pages: {retrieved_wiki_pages}

The purpose.md domain description is prepended to the pages block as a reminder during synthesis.

When retrieval confidence falls below threshold, the system surfaces the gap rather than guessing. The response notes that the wiki lacks dedicated coverage on that topic and generates enrichment suggestions — search queries and domain-scoped URLs. Any pages retrieved flow through the normal ingest pipeline and stay in draft state, out of the search corpus, until a human promotes them to active.

What grounding doesn't solve: A model can still hallucinate within retrieved content — misattributing which page said what, inventing a specific number close to but not the same as the source, or synthesizing a conclusion that no single page actually states. Grounding reduces the search space. It doesn't eliminate drift.

Layer 3: Citation Faithfulness Audit

This is the layer that makes the system verifiable. Each claim in an answer carries a citation marker linking it to specific source lines. A separate LLM pass then audits whether the claim is actually supported:

Verdict	Meaning
`supported`	The claim is directly and accurately supported by the cited source lines.
`drift`	The claim is related to the source but overstated, simplified, or shifted in meaning.
`hallucination`	The source lines do not support the claim, or directly contradict it.

The auditor sees only the claim and the raw source lines — not the original question. This matters: the same model that wrote the answer is likely to defend it when asked "is this correct?" Separation of duties removes that incentive.

What we actually see: drift is the most common finding. Specific dates that don't appear in the source, an attribution switched between two people in the same paragraph, a qualifier silently dropped — these are the findings that surface. True hallucination, where source lines don't support the claim at all, is less common in well-grounded RAG. But it happens, and when it does the audit tells you exactly which source line to check.

Audit results are cached per-page with staleness detection. If a page is re-ingested from updated source material, the cache invalidates and the next audit reflects the new content.

Two Things This Still Doesn't Solve

Synthesis across pages. A conclusion drawn by combining facts from multiple pages may be accurate at each step and still misleading in combination. The citation audit operates per-citation — cross-page inferences that no single page actually states fall outside its scope. The practical mitigation: cited pages are always explicit, so synthesized conclusions can be verified. And if a conclusion matters enough to keep, saving it as a wiki page subjects it to future source ingestion — any new source that disputes it triggers the flag action.

Uniform source error. When all available sources agree on the same wrong answer, no contradiction is detected and Synthadoc answers confidently from incorrect material. Two mechanisms reduce this risk: adversarial lint, which uses the model's general knowledge to flag claims that clearly contradict well-established facts; and re-ingesting from updated authoritative sources, which triggers the flag action against outdated pages. Neither is a complete solution.

The View From Here

The hallucination debate is usually framed as a model capability question: will the next model be more faithful? For a production knowledge system, that's the wrong frame. Even a perfectly faithful model can only be as accurate as the content it's grounded in. And even a slightly drifting model can be made accountable if its output is systematically audited.

The architecture is a feedback loop: domain scoping controls what enters; lifecycle states control what gets queried; retrieval grounds generation; citation audit classifies the output; the contradiction resolver closes the loop. When a conflict is found, a human resolves it, the page is corrected, and the knowledge base improves. Errors become visible rather than accumulating silently.

That's a more defensible position than any number of ALL CAPS instructions.

Synthadoc is open-source. The citation faithfulness audit, contradiction resolver, and lifecycle system described here are all in the follwoing repository.

→ github.com/axoviq-ai/synthadoc

[Boost]

Paul Chen — Tue, 11 Aug 2026 14:08:59 +0000

Paul Chen

Aug 10

From Manual Commands to Intent-Driven Maintenance: The Agentic Workflow

#ai #agents #llm #automation

4 min read

From Manual Commands to Intent-Driven Maintenance: The Agentic Workflow

Paul Chen — Mon, 10 Aug 2026 22:15:47 +0000

Here's a maintenance loop that every wiki eventually produces.

A source document gets updated. The wiki page derived from it goes stale. You run synthadoc ingest to reprocess the file. You wait. You run lint to check if the page passes quality checks. You wait again. If it passes, lint promotes it to active. If it doesn't, you diagnose the issue and repeat. Now multiply that by twelve stale pages.

It's a well-defined sequence. It's also tedious — each step blocks on the previous one, you have to stay present for the whole thing, and the surface area for forgetting a step is real.

Synthadoc v1.2 ships an agentic maintenance workflow that replaces this loop with a single prompt.

What the agent actually does

When you send "re-ingest stale pages," the ActionAgent doesn't just parse your intent and hand it to a fixed function. It runs a tool-call loop: the model emits a structured tool call, the agent executes it, returns the result to the model, and the model decides what to call next. This continues until the model produces a plain-text summary with no pending tool calls.

For the stale-page workflow, the sequence looks like this:

find_stale_pages : returns every stale page with its source file path
confirm : sends a confirmation request to the UI listing the scope; blocks until you approve or decline
ingest_source (×N) : force-reingests one source at a time, polling until each job reaches a terminal state
run_lint : enqueues a full lint pass
poll_job : waits for the lint job to complete
get_page_states : checks the final lifecycle state of every reingested page
Plain-text summary with every outcome, lint result, and page states

The model drives all of it. If one ingest fails mid-workflow, the agent continues with the remaining pages and reports the failure in the summary. No hardcoded branching, the failure handling comes from the system prompt the model operates under.

The confirmation gate

The confirm tool is not a courtesy. Before the agent writes anything, it sends a confirm_request event to the UI carrying the full scope — which pages, which source files — and blocks until you respond. The 120-second timeout defaults to declined rather than approved.

This matters because "re-ingest stale pages" and "re-ingest all pages, including ones I deliberately left stale" look identical to a system without a confirmation step. The gate keeps you in control of what gets rewritten.

Real-time progress

A workflow that takes three minutes and shows output only at the end feels like it hung. The agentic workflow emits tool_progress SSE events at each tool step — "Re-ingesting: alan-turing.md", "Ingest running... (7s)", "✓ alan-turing re-ingested", "Running lint check..." — so the UI stays live throughout.

The same events surface in the Obsidian plugin's query modal. After the workflow completes, the web UI pre-fills the chat textarea with the logical next action — "Run lint to promote re-ingested pages" — so the next step is one click, not a typed command.

Works from anywhere

The same agentic loop runs regardless of where you initiate it:

Web UI: type the prompt in the chat interface, watch progress stream in. Or from the knowledge graph, right-click any stale page node and trigger a per-slug reingest directly from the graph view
CLI: synthadoc query "re-ingest stale pages" runs the same agent, same confirmation prompt, same result

The CLI path surfaces tool_progress events as terminal lines and handles the confirm gate interactively with a [Yes/Cancel] prompt, so the workflow is fully usable without a browser.

Pluggable by design

The workflow is built on an abstract interface: AgenticWorkflow declares build_system_prompt(), build_initial_message(), and get_tool_fns(). The loop machinery — tool dispatch, result injection, termination detection, the 30-call limit, the confirmation timeout — lives in the action agent and is inherited by every workflow.

Adding a new workflow means implementing three methods and writing a system prompt. The existing infrastructure handles the rest.

See it running

The walkthrough below covers the full loop end to end: two stale pages, the confirmation gate, per-page ingest progress, lint auto-run, and the content snapshot diff in Obsidian showing exactly what changed between versions.

📺 Synthadoc Agentic Maintenance Workflow — YouTube

What we learned building this

System prompts are the policy layer. The rules governing the workflow — discover scope first, always confirm before writing, process pages one at a time, run lint after all ingests — live in the system prompt, not in code. When we needed to adjust the ordering for the per-slug path, it was four lines of text, not a code change.

Per-step progress makes everything feel faster. The actual duration doesn't change. The perceived latency does.

One failure shouldn't stop the batch. If a source file disappears between discovery and ingest, the workflow reports the failure and keeps going. Wiring that in from the start saved a real debugging session later.

Synthadoc CE v1.2.0 is open source. The agentic workflow, the pluggable interface, and the full SSE protocol are all in the repo.

Which LLM Gives You the Best Accuracy-to-Cost Ratio for Knowledge Base Queries?

Paul Chen — Fri, 10 Jul 2026 18:47:51 +0000

When you build a knowledge base powered by an LLM, you eventually have to answer a practical question: which model should the query agent actually use?

This post shares our findings from a structured evaluation across five models. The short version: the cheapest viable option also won on accuracy.

Background

Synthadoc is an open-source LLM wiki compilation engine. It reads your raw source documents — PDFs, spreadsheets, web pages, Word files — and synthesizes them into a structured, cross-referenced wiki at ingest time. Knowledge is compiled once, not re-summarized on every query.

The QueryAgent sits at the front of every question. It decomposes the query, runs parallel BM25 retrieval across the compiled wiki, assembles a proportional context window, and calls the configured LLM to synthesize an answer with inline citations. It's the most token-intensive path in the system, and the one where model choice has the most visible impact on both quality and cost.

Why We Ran This Evaluation

Synthadoc supports a wide range of LLM providers: Anthropic, MiniMax, DeepSeek, Qwen, Gemini, Groq, and local Ollama. Choosing between them used to be a matter of intuition. We wanted data.

The specific question: is a more expensive model meaningfully more accurate, or is there a sweet spot that gives you most of the quality at a fraction of the cost?

The Evaluation Setup

We used the AquaFlow Capital scenario, a private equity M&A due diligence wiki built from eight deal documents (LBO model mechanics, quality of earnings, covenant framework, ESG standards, legal DD process, and exit valuation benchmarks).

We ran 15 structured questions across three complexity tiers:

English - Medium (Q1–Q5): factual retrieval, single-page lookups
English - High (Q6–Q10): multi-source synthesis, numerical reasoning
Cross-lingual Chinese (Q11–Q15): same questions in Chinese, requiring CJK retrieval

Five models were evaluated: MiniMax-Think (M3), Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1 (V3), and Qwen Plus.

Each answer was graded on a 0–100 scale against a ground-truth rubric. PASS threshold: 85%.

What We Found

The scatter tells the story clearly.

MiniMax-Think (M3) sits alone in the top-left, highest accuracy at the lowest cost. That's the ideal quadrant, and no other model comes close to occupying it.

Model	Accuracy	Est. cost / 1K queries
MiniMax-Think (M3)	92%	~$25–40
Claude Opus 4.8	89%	~$1,050
Claude Sonnet 4.6	86%	~$210
DeepSeek-R1 (V3)	78%	~$35–55
Qwen Plus	73%	~$30–50

Claude Opus 4.8 scored 89%, just 3 percentage points below MiniMax-Think, but at roughly 5× the cost of Sonnet and 26× the cost of MiniMax. For a query workload measured in thousands of questions, that gap is significant.

Claude Sonnet 4.6 at 86% is the best option if you're locked to the Anthropic Claude or its API and need reliable citation quality.

DeepSeek-R1 at ~$40/1K queries is a solid budget pick for exploratory or conceptual synthesis queries, where synthesis depth matters more than precision.

Takeaway

For the AquaFlow domain - dense financial documents, multilingual queries, numerical reasoning - MiniMax-Think (M3) is the clear default. It wins on accuracy and cost simultaneously. That's a rare combination.

Model choice isn't just a quality decision. At scale, it's a cost architecture decision.

The full report includes per-question breakdowns, cost math, WARN pattern analysis (where each model fails and why), and a per-question line chart across all 15 questions:

👉Full evaluation report on GitHub

Interfaces Layer Design Pattern for AI Knowledgebase

Paul Chen — Tue, 23 Jun 2026 16:01:30 +0000

▶ Watch on YouTube

Most AI knowledge tools make a quiet assumption: the model is the worker, and the human is the caller.

You ask, the system retrieves pages, the LLM synthesises them, you get an answer. Clean pipeline. One direction.

That assumption breaks the moment a reasoning model needs to use the knowledge base as a tool - not to answer a question, but to do work. Search for related pages. Read a contradicted entry. Write a resolution. Commit a lifecycle transition. Then search again based on what it found.

Now the model is the caller. The knowledge layer is the called. Same storage. Opposite direction.

This is the coordination problem that most knowledge tools never have to solve, because they pick one mode and stay there. Building a system that handles both cleanly turns out to be the interesting design challenge.

The Four-Interface Pattern

The clearest way to see the tension is through the four surfaces a well-designed knowledge system needs to support:

CLI : a human (or script) calls the system, which calls an LLM, which returns an answer. Streaming for humans, structured output for pipes. The model is a service.

IDE/App Plugin : same direction, richer context. The plugin renders wikilinks as navigation, exposes a lifecycle dashboard for human decisions. Still: human calls system calls LLM.

Web Chat UI : conversational, multi-turn, natural language operations. "Run lint." "Show me stale pages." The system interprets these and dispatches them. Still: human calls system calls LLM.

MCP : a reasoning model connects to the knowledge layer as a set of tools. It decides what to search, what to read, when to write, which lifecycle state to commit. The system is now memory. The model is now the brain.

The insight is that the first three interfaces share a common structure. The fourth inverts it entirely.

Why the Memory Layer Needs Opinions

Here is where the design gets non-obvious.

When a reasoning model can freely read, write, and transition pages, you need the storage layer to enforce constraints the model might not. A lifecycle transition graph that blocks invalid state changes. An immutable audit trail that records every edit and its stated reason, regardless of whether a human or an agent made it. Provenance tracking that ties every page to its source material.

These aren't features for the human users. They're the constraints that make it safe to connect a reasoning model to the same base a human is actively maintaining.

A model can call write_page then lifecycle_transition. What it cannot do is move a page from stale to contradicted - that transition isn't in the allowed graph. The memory layer enforces this the same way it enforces it for a CLI command. It doesn't care which surface issued the instruction.

The audit trail records triggered_by: mcp and the model's stated reason. Run the lifecycle log the next morning and the full chain is there, whether it was a human clicking in a plugin or an agent running at 2 AM.

The Hint Engine Doesn't Disappear

One question that comes up: if a reasoning model generates its own next steps dynamically, what happens to the deterministic hint engine that suggests follow-up queries?

It serves the human-facing interfaces. No LLM cost, instant suggestions based on current wiki health, contextual to what the user just asked. A reasoning model doesn't need that, it generates its own chains based on what it finds. Two different users, two different workflows, same storage layer.

The coexistence is the point. The four-interface pattern isn't about choosing between human-driven and agent-driven workflows. It's about a knowledge layer that supports both without leaking the distinction into the storage model.

Reference Implementation

The pattern described here is implemented in Synthadoc, an open-source AI knowledge base tool. The video above walks through all four interfaces against a live demo wiki - CLI queries, Obsidian plugin lifecycle management, web chat multi-turn conversation, and connecting Claude Code as a reasoning brain over MCP.

If you're building something similar, the part worth paying attention to isn't the interface count, it's the audit trail and transition graph. Those are what make the memory layer safe to connect to a model that can write, not just read.

Google's OKF Is Proposing a Standard for How AI Agents Consume Domain Knowledge - Here's How Synthadoc Already Fits

Paul Chen — Mon, 15 Jun 2026 14:03:53 +0000

On June 12, 2026, Google Cloud published Open Knowledge Format (OKF) v0.1 - a vendor-neutral specification for how AI agents should consume structured knowledge. If you've been thinking about how to give your agents reliable, domain-specific memory beyond RAG over raw documents, this is worth your attention.

We built Synthadoc to solve exactly this problem, and the OKF spec reads like a formal description of what we built. Independent convergence on the same design is usually a sign that both teams found something real.

This post breaks down what OKF is, where Synthadoc already aligns with it, how we're converging on the remaining gaps, and why we think Synthadoc is one of the most important implementations the OKF ecosystem needs.

What Problem OKF Is Solving

Organizations today fragment their knowledge across metadata catalogs, wikis, Notion pages, code comments, and internal documents. When you want an AI agent to reason over that knowledge, you face a choice: either build a custom pipeline for each source, or dump everything into a vector store and hope similarity search finds the right facts.

OKF proposes a third path: a portable, file-based format for curated knowledge that any AI agent can consume without vendor-specific APIs or custom parsers. The format is deliberately minimal - markdown files with YAML frontmatter, organized in directories whose paths represent concept identity. Standard markdown links between files create a traversable knowledge graph.

The only required field is type. Everything else - title, description, tags, timestamp, resource - is optional. The philosophy is producer/consumer independence: the same format works whether you generated it from a data catalog, a wiki, or a code annotation pipeline.

This is a strong design philosophy. It means OKF bundles are readable by humans in any text editor, versionable in git, and consumable by any agent that can read files.

Google's reference implementation

Alongside the spec, Google shipped three things: an Enrichment Agent that ingests BigQuery metadata and emits OKF bundles (built on Google's Agent Development Kit with Gemini as the backend), a Visualizer that renders any OKF bundle as a self-contained interactive HTML file using Cytoscape.js, and three pre-built sample bundles for GA4 e-commerce, Stack Overflow, and Bitcoin datasets.

The Enrichment Agent is worth studying. It is a working OKF producer, a concrete proof that the format is implementable, not just specifiable. But its source material is structured: BigQuery tables, schemas, foreign-key relationships, data catalog metadata. This is one important slice of the OKF producer space.

The other slice - the harder one for most organisations - is unstructured source material: internal PDFs, research papers, spreadsheets, meeting notes, product documentation. That is the gap Synthadoc fills. Google's Enrichment Agent and Synthadoc are complementary producers in the same ecosystem, covering different source types with the same output format.

	Google Enrichment Agent	Synthadoc
Source material	BigQuery datasets, data catalogs	PDFs, XLSX, PNG, plain text
OKF role	Producer	Producer
Lifecycle management	None	5-state (draft → archived)
Conflict resolution	None	Contradicted state + auto-resolve
Human editorial gate	None	draft → active transition
LLM backend	Gemini only	Gemini, Anthropic, OpenAI, Groq, Qwen, Ollama, Deepseek, Minimax

Synthadoc's Design Was Already There

Synthadoc compiles raw documents (PDFs, spreadsheets, images, Youtube video, plain text) into a structured wiki - a collection of markdown files in a directory, each with YAML frontmatter, linked together by wikilinks. That wiki is then queryable by both humans (through Obsidian or a web UI) and AI agents (through the API).

Here's what a Synthadoc wiki page looks like:

---
title: Alan Turing
tags: [biography, theory, cryptography]
status: active
confidence: high
created: 2026-04-08
sources:
  - file: turing-biography.pdf
    hash: sha256:9f3a...
    ingested: 2026-04-08
---

# Alan Turing

Alan Mathison Turing (1912–1954) was an English mathematician and computer scientist...

The [[von-neumann-architecture]] that underlies modern computers shares conceptual roots
with Turing's stored-program ideas. Turing also influenced [[programming-languages-overview]]
by formalising what it means for a problem to be computable.

When we mapped this against the OKF specification, the alignment was immediate:

OKF requirement	Synthadoc
Markdown + YAML frontmatter	Native format for all wiki pages
`index.md` reserved for navigation	`wiki/index.md` in every wiki — auto-generated and maintained
Hierarchical concept organization	`ROUTING.md` groups pages into topic branches (`People`, `Hardware`, etc.) - logical hierarchy via manifest rather than physical directory structure
Markdown links as knowledge graph edges	`[[wikilinks]]` compile to a traversable knowledge graph
`tags` field	First-class frontmatter field on every page
`timestamp` (ISO 8601)	`created` and lifecycle timestamps throughout
No proprietary SDK required	Standard files, readable by any tool
Git-compatible	Every wiki is a git-managed directory

The core insight OKF encodes that knowledge for agents should live in plain files with structured metadata, not locked in a database, is the same insight Synthadoc was built on.

Converging with OKF

Synthadoc was designed independently before OKF was published, so a few intentional design choices don't yet match the spec exactly. Here's what they are and how we're closing each gap.

Wikilinks → standard markdown links.
OKF uses [Alan Turing](people/alan-turing.md); Synthadoc uses [[alan-turing]]. The graph structure is identical - only the syntax differs. We're adding an OKF export mode that rewrites wikilinks to standard markdown links at export time, so any Synthadoc wiki can be published as a fully spec-compliant OKF bundle.

Adding the type field.
OKF's only required field is type , a classifier for what kind of knowledge the page contains (concept, person, technology, event). Synthadoc's frontmatter uses status for lifecycle state and confidence for epistemic weight - complementary metadata, not a replacement. We're adding type to the schema in the next release. With that one field, every Synthadoc wiki page will be OKF-compliant out of the box.

sources → resource alignment.
OKF's resource field is a single URL pointing to an external source. Synthadoc's sources array goes deeper: each entry includes the file path, SHA-256 hash, file size, and ingestion date, and individual claims carry line-level citations. This is a superset of what OKF specifies, not a conflict. The OKF export will surface the primary source URL in the resource field while preserving Synthadoc's full provenance in the body.

These three changes give Synthadoc full OKF compliance while keeping the richer metadata that makes the wiki useful for humans and auditors, not just agents.

Where Synthadoc Goes Further

OKF v0.1 explicitly lists several open design questions - problems the specification hasn't solved yet. These are interesting to us because they're exactly what Synthadoc was designed to handle.

Contradiction resolution. When two source documents say conflicting things about the same topic, OKF has no answer yet. Synthadoc flags the affected page as contradicted and can automatically propose a reconciled view using LLM-assisted lint. The resolution is recorded in the page's lifecycle audit trail.

Stale information management. OKF acknowledges this as an open problem. Synthadoc has a stale lifecycle state triggered when a source document changes after initial ingest. Scheduled lint runs detect drift and surface pages that need review.

Adversarial review. OKF doesn't address how to audit knowledge for unsupported claims or overconfident assertions. Synthadoc's lint pass runs an adversarial review on each page, flagging claims that lack source support and writing findings directly into lint_warnings frontmatter.

These aren't listed as criticisms of OKF v0.1 is explicitly a "starting point for community feedback." They're listed because they represent the class of problems any serious knowledge management system needs to solve as it matures.

Synthadoc as an OKF Producer: Bridging Agents to Domain Knowledge

Here's the framing we find most compelling.

OKF describes a consumption format , a way for AI agents to read structured knowledge. What it doesn't describe is how that knowledge gets created, validated, kept current, and made queryable. That's the production problem.

Synthadoc solves the production problem. It ingests raw documents from your organization, compiles them into a structured wiki, validates each page through adversarial lint, manages the page lifecycle as sources change over time, and serves the result through a query API.

The output of Synthadoc, a directory of markdown files with YAML frontmatter, linked by wikilinks, with an index.md, is naturally aligned with what OKF describes as agent-consumable knowledge.

This means you can think of Synthadoc wikis as OKF-compatible knowledge bundles that cover the full lifecycle:

If you're building an agent that needs domain expertise, a customer support agent that needs product knowledge, a research agent that needs a scientific literature summary - a coding assistant that needs your internal architecture decisions - Synthadoc turns your existing documents into a structured, validated, self-maintaining knowledge base the agent can trust.

Our Perspective: What the Standard Still Needs

OKF v0.1 is a strong starting point, and we think the direction is right. But having built in this space, we have a few opinions worth putting on record, not as criticism, but as a contribution to where v0.2 should go.

A format standard without a quality standard is garbage-in, garbage-out.
If any tool can produce an OKF bundle, agents will inevitably consume contradictory, stale, or unsupported content formatted as valid OKF. The spec defines how knowledge is structured, but not how it is validated. The ecosystem needs production tooling that enforces quality - adversarial review, conflict detection, lifecycle management - before knowledge reaches an agent. A format alone doesn't make a knowledge base trustworthy.

Agent memory needs trust signals, not just content.
OKF sepc currently gives an agent no way to know how confident to be in what it's reading. A page written two years ago from a single unverified source should carry different weight than one reviewed last week, cross-referenced against multiple documents, and cleared by an adversarial lint pass. confidence, status, and claim-level provenance need to become first-class concepts in the spec, not optional extensions, if agents are to make consequential decisions from OKF knowledge.

Domain-specific knowledge is where this matters most.
General facts are already baked into LLM weights. What agents actually need is proprietary, time-sensitive, domain-specific knowledge that cannot be in training data - internal architecture decisions, evolving product specifications, specialist research. This is precisely where OKF is most valuable, and also where freshness and accuracy matter most. A standard that doesn't address knowledge currency will struggle in exactly the use case it's most suited for.

The human editorial layer is non-negotiable for consequential decisions.
Fully automated pipelines - ingest everything, trust everything - break down when agents act on the results. Knowledge intended for agent consumption needs a defined editorial state: has a human reviewed this page and cleared it for use? Synthadoc's draft → active lifecycle transition is exactly this gate. We think OKF should define a concept of editorial provenance so consumers can distinguish machine-generated drafts from human-validated knowledge.

Synthadoc v0.8.0

The OKF alignment discussed in this post ships alongside Synthadoc v0.8.0, released June 12, the same day as OKF v0.1. The release focuses on three areas:

Multi-turn conversation : the web UI maintains full session history across turns. Follow-up questions resolve correctly through context injection and automatic phrase rewriting. Long sessions compress older turns into summaries rather than discarding them. Sessions persist server-side and survive page refreshes; the sidebar shows history as a collapsible tree with turn-count badges.
Qwen DashScope + Claude Opus 4.8 : Qwen cloud models (qwen-plus, qwen-max, qwq-32b) now work via DashScope's OpenAI-compatible endpoint. New DashScope accounts receive 1 million free tokens valid for 90 days. Claude Opus 4.8 is supported for maximum answer quality.
Conversational job operations : the Action Agent handles live job queries directly in chat. Ask for job status and receive a real-time table; job IDs appear as clickable chips for drill-down. Multi-status filters ("show failed and skipped jobs") work across turns.

If you find Synthadoc useful, a ⭐ on GitHub helps the project reach more people: https://github.com/axoviq-ai/synthadoc.

References

OKF v0.1 Specification : GoogleCloudPlatform/knowledge-catalog
Synthadoc : github.com/axoviq-ai/synthadoc

Synthadoc: A Self-Invalidating Query Cache

Paul Chen — Thu, 11 Jun 2026 02:12:28 +0000

Most LLM queries feel expensive the second time. You've already asked your wiki "What is Moore's Law?" this morning. The answer hasn't changed - nothing in your wiki has changed - but if you ask again this afternoon, Synthadoc will hit the retrieval pipeline, build a prompt, call the LLM, and wait 2–10 seconds for an answer you already have.

v0.7.0 eliminates that. A query cache stores the result the first time, and serves it instantly on every repeat - with one rule: the cache invalidates itself automatically whenever your wiki changes. No manual flush, no TTL to configure. You never serve a stale answer, and you never pay twice for the same question.

This post covers the cache design and the performance benchmarks we ran against it. The streaming query pipeline and web chat UI that shipped alongside it in v0.7.0 are covered in the companion post: Synthadoc: Streaming Queries and a Local Web Chat UI.

Query Caching: When Not to Call the LLM

The cache design started from a specific observation: most queries against a domain wiki are repeated. A team maintaining a knowledge base about their software architecture will ask the same questions dozens of times - during onboarding, during incident reviews, during planning sessions. Every one of those calls hits the LLM and incurs both latency and cost.

The cache eliminates that. But the tricky part is knowing when to invalidate it.

Cache Key Design

key = SHA-256(normalized_question + "|" + wiki_epoch + "|" + provider_model)

Three components:

Normalized question - lowercased, whitespace-collapsed. "What is Moore's Law?" and "what is moore's law?" hit the same cache entry.

Wiki epoch - an integer counter on the server instance. It starts at 0 on startup and increments on every ingest job completion and every lifecycle state transition. When the epoch changes, the cache key for every question changes. Prior entries don't get deleted immediately, they just become unreachable. Old entries are cleaned up in a background sweep (entries more than 5 epochs behind current, or older than 7 days).

Provider/model - "openai/gpt-4o-mini" or "anthropic/claude-sonnet-4-6". Switching models invalidates the cache. A cached answer from a smaller model shouldn't surface when you've upgraded to a better one.

The epoch approach is what makes invalidation automatic. You don't call "invalidate cache" after an ingest, the epoch bump does it implicitly. Any query after a wiki change computes a new key that has never been seen, misses the cache, and calls the LLM fresh. The previous answer doesn't need to be deleted; it simply ceases to be looked up.

Diagram: Cache Lookup, Hit, Miss, and Epoch Invalidation

Measured Results

Rather than leaving the latency claims as estimates, we wrote a full performance test suite against the cache layer. All numbers below come from running pytest tests/performance/test_query_cache_perf.py locally on a Windows development machine with an SSD. Linux bare-metal numbers are consistently 30–40% better.

Chart 1 - Cache read latency distribution (500 reads, 200 cached entries)

P50 = 0.26ms, P95 = 0.34ms, P99 = 0.41ms against a 10ms SLO. The distribution is extremely tight - the persistent connection eliminates the per-call connection-open overhead that was the main source of outliers. Every percentile sits well inside the budget with headroom to spare.

Chart 2 - Cache hit vs miss latency at varying LLM speeds

The left panel uses a log scale because the gap is so large it can't be shown linearly. Cache hit P50 stays flat at ~0.25ms regardless of LLM speed - one shared persistent connection makes the hit path a pure queue-and-execute SQLite read with no file-open cost. The miss path scales directly with LLM latency. The right panel shows the resulting speedup factor:

Simulated LLM speed	Cache miss P50	Cache hit P50	Speedup
50ms (fast provider)	95ms	0.29ms	~330×
200ms (mid provider)	235ms	0.32ms	~730×
500ms (slow provider)	544ms	0.24ms	~2270×
2000ms (reasoning model)	2055ms	0.26ms	~7900×

The cache hit time is so small relative to any real LLM that the ratio is dominated entirely by provider latency. At reasoning models (o3-mini, MiniMax M2) a single saved round-trip reclaims 15–30 seconds of wall time.

Chart 3 - Concurrent readers: persistent connection scaling curve

Single reader: 0.5ms P95. Ten readers: 2.0ms. Twenty-five: 3.8ms. Fifty: 7.8ms. One hundred: 14.9ms. The curve is smooth and monotonically increasing - no Windows spikes, no non-monotonic jitter. All concurrent reads queue through one shared aiosqlite background thread; the connection-open overhead that caused the old instability is simply not there. For a local single-user tool the realistic ceiling is n=5–10 concurrent reads, where P95 is under 2ms. Even at n=100 the tail is well inside a 50ms budget.

Chart 4 - Cache vs no-cache throughput (queries/second)

The throughput advantage starts at 75.7× at n=1 and compresses to 4.1× at n=100. The compression is expected: asyncio.gather() parallelises the simulated LLM calls so the no-cache path scales nearly linearly with concurrency. The cache path, sharing one connection, serializes through the aiosqlite queue and grows sublinearly. But critically, the cache always wins by a wide margin - 4.1× at n=100 is far better than the 1.3× seen before the persistent connection fix. At realistic single-user concurrency (n=1–5), the advantage is 33–76×.

Estimated Latency Gains

A typical Synthadoc query against a mid-size wiki has two latency components:

Phase 1 (BM25 retrieval): 100–200ms. This runs regardless of cache.
Phase 2 (LLM synthesis): 2–10 seconds depending on provider, model, and answer length.

A cache hit skips Phase 2 entirely. The server reads the cached result_json from SQLite (~0.26ms P50 on SSD via a persistent connection), then emits a synthetic SSE burst at full network speed. The client receives what looks like a live streamed response, but the entire burst completes in under 100ms instead of waiting 2–10 seconds for the LLM. With a reasoning model provider, that gap widens to 15–30 seconds per query, the cache makes those queries feel instant.

The Cache Is Shared Across All Three Surfaces

CLI, Obsidian plugin, and Web Chat UI all share the same cache.db. If you ran synthadoc query "..." from the CLI this morning and the wiki hasn't changed, opening the Obsidian modal and asking the same question will hit the cache. The key is identical - same normalized question, same epoch, same model.

# Drop the entire cache - both LLM response cache and query cache
synthadoc cache clear
Cache cleared: 47 entries removed.

What Makes This Architecturally Different

The caching architecture differs from the typical approach of setting an explicit TTL (cache for 24 hours, or cache for one week). TTL-based caches are almost always wrong at the edges: they're either too short (you evict answers that are still valid) or too long (you serve stale content after a wiki update). Epoch-based invalidation is event-driven — the cache is valid until something in the wiki changes, exactly. You don't think about expiry. You ingest new content, and the next query automatically goes to the LLM. Every query after that hits the cache again until the next change.

Quick Demo

Query caching is covered in the quick-start guide against the history-of-computing demo wiki:

Query caching: Step 23 - Query caching

The full thing runs locally in about ten minutes:

git clone https://github.com/axoviq-ai/synthadoc.git
pip install -e ".[dev]"
synthadoc install history-of-computing --target ~/wikis --demo
synthadoc plugin install history-of-computing
synthadoc web   # opens browser

If you find Synthadoc useful, a ⭐ on GitHub helps the project reach more people: https://github.com/axoviq-ai/synthadoc.

[Boost]

Paul Chen — Wed, 10 Jun 2026 02:56:51 +0000

Paul Chen

Jun 8

Synthadoc: Streaming Queries and a Local Web Chat UI

#ai #llm #architecture #cli

9 min read

Synthadoc: Streaming Queries and a Local Web Chat UI

Paul Chen — Mon, 08 Jun 2026 21:02:24 +0000

There's a moment every Synthadoc user hits eventually. You've got forty or fifty compiled pages, a nightly ingest schedule running, lint keeping everything healthy. And then you open a terminal, type synthadoc query "...", and wait. The BM25 retrieval is instant. But then the cursor blinks. The LLM is thinking. You wait four seconds, six seconds, eight seconds. The answer eventually appears, all at once, like a curtain dropping.

That wait is fine the first time. It gets annoying on the tenth query when you're in a research session and you already know the answer is coming - you just want to read it as it forms, not stare at a blinking cursor.

v0.7.0 improves that. Streaming query output across all three query surfaces, and a local web chat UI that understands the health of your wiki. The architecture behind each of these turned out to be more interesting than I expected when we started building them. (A companion post covers the third feature in this release: a self-invalidating query cache.)

Diagram 1: What Changed in v0.7.0: The Architecture at a Glance

The diagram below maps the full Synthadoc architecture as it stands after v0.7.0. Items marked [NEW] are additions in this release; everything else was already present. The two features in this post touch two separate layers: the access layer gains a new client, and the engine gains new agents.

Two additions are the focus of this post:

Query Web UI (Access Layer): the new synthadoc web browser client using HTTP + SSE
Query (stream) · Action · Hint Engine (Agents): streaming query pipeline, live command execution, deterministic hint generation

Everything flows through the same server process per wiki. The CLI, Obsidian plugin, and Web Chat UI are all thin clients talking to the same HTTP + SSE endpoint. There's no separate service for the web UI. The MCP Server (shown as optional with a dashed border) is a fourth access path for AI tools like Claude Desktop or Cursor — it exposes the same wiki operations over the Model Context Protocol and requires opt-in setup.

Three Ways to Query Your Wiki and When to Use Each

Before getting into the streaming mechanics, it's worth laying out the three query surfaces Synthadoc now supports. All three can answer the same question, the difference is workflow fit.

CLI: when the query is part of a larger workflow

The CLI is where you go when a query isn't just a question, it's a step in something automated. The obvious case is CI/CD: a post-ingest job that queries the wiki to verify a newly compiled page before promoting it to active. Less obvious is using it as part of an agent integration, where an external orchestrator issues queries and parses the structured JSON output.

# Stream to terminal - tokens appear as the LLM generates them
synthadoc query "What were the main causes of the 2008 financial crisis?"

# Script mode - waits for full response, stdout is clean for piping
synthadoc query "Summarize page: moore's-law" --no-stream | jq .

# Force LLM call even if cache has a result - useful when wiki just changed
synthadoc query "What changed in the latest ingest?" --no-cache

The --no-stream flag is specifically for automation. Streaming output is beautiful on a terminal and disruptive in a pipeline. A script that parses stdout doesn't want token-by-token delivery, it wants a complete JSON blob when the query is done. --no-stream gives it that.

Obsidian Plugin: when you're in a research session

The Obsidian plugin exists for a different moment: you're writing a note, you need to check a claim against your wiki, and you don't want to leave Obsidian. The query modal (Ctrl/Cmd+P → Synthadoc: Query: ask the wiki...) is the right tool here. It renders [[wikilinks]] as clickable links, which means an answer that references related pages becomes navigable instantly.

The streaming behaviour in the Obsidian plugin mirrors the CLI, tokens appear as they arrive, citations follow at the end. The bypass cache checkbox is visible in the modal, unchecked by default. For researchers doing active ingest sessions, checking it once gets you fresh output without reaching for the terminal.

Web Chat UI: when you want a session, not a one-shot query

synthadoc web is the new entry. It opens a local chat interface in your browser, nothing leaves your machine, no cloud service, no authentication. It's designed for the kind of session that's too exploratory for the CLI and too long for the Obsidian modal.

Each turn is an independent query - the same cache applies here as in the CLI and Obsidian plugin. The chat history is displayed in the browser, but prior messages are not yet injected into the LLM prompt; multi-turn context injection is planned for a future release.

What the web UI adds over the other surfaces: operational commands. You can type "run lint", "show wiki status", "what pages are orphan pages?" or "schedule ingest every night at 9 PM" directly in the chat, and the Action Agent parses those and executes them live against your wiki, with results shown inline.

The screenshot above shows a live session against the history-of-computing demo wiki. The response to "What changed in the wiki this week?" includes a date-indexed ingest table, current lifecycle counts (Active: 80, Draft/Stale/Contradicted/Archived: all zero), and three action chips - "Activate a draft page", "Archive a stale page", "Restore an archived page to draft" - rendered inline as clickable buttons. The left panel shows prior session queries, allowing you to jump back into an earlier thread.

Streaming: The Architecture Behind a Two-Phase Response

Every Synthadoc query goes through two phases. Phase 1 is retrieval: BM25 search, routing, sub-question decomposition if needed. This is synchronous and fast, typically 100–200ms. Phase 2 is synthesis: the LLM generates an answer from the retrieved pages. This is where the latency lives.

The decision to stream only Phase 2 was deliberate. Phase 1 finishes before the first LLM token could possibly arrive, there's no partial retrieval state worth exposing. So the SSE protocol is clean:

The status events let the UI give immediate feedback. The user knows within 150ms whether the wiki found relevant pages or not before any LLM latency has accumulated. "sources: 3" in the synthesizing event tells them the answer is backed by three pages before they've read a single word of it.

The gap event fires only when the wiki doesn't have enough to answer confidently. Instead of a vague "I don't know," it returns suggested_searches - concrete ingest strings the user can use to fill the gap. These are generated by a secondary LLM call that decomposes the original question into targeted search queries - the same decomposition that drives sub-question retrieval, reused here to produce actionable ingest suggestions.

Provider Streaming Behavior

Not all providers stream in the same sense. API-based providers - OpenAI, Anthropic, Gemini, Ollama - emit tokens as they are generated, so the CLI and web UI render them character-by-character in real time. The latency shown in the SSE sequence above (one token every ~20ms) is what these providers deliver.

CLI subprocess providers - Claude Code (claude-code) and Opencode (opencode) - work differently. They run as child processes and write their output only when the process exits, so there is no per-token stream to intercept. Synthadoc runs the subprocess to completion, then emits the result word-by-word through the same SSE pipe. The words arrive in a rapid burst rather than a gradual flow - the total wait is the same, but the perceived streaming effect is a short pause followed by the full answer appearing almost at once.

If you are using a CLI subprocess provider and queries are timing out, increase the default timeout:

synthadoc query "..." --timeout 180

The default is 60 seconds, which is sufficient for API providers but may be short for subprocess providers on complex queries.

Session Management

Sessions live server-side in audit.db, in two tables: chat_sessions and chat_messages. The React UI stores only the session_id in memory - it's React state, not localStorage. This means sessions don't survive page reload, and every new browser tab starts fresh. This is a deliberate design choice: a session is tied to one exploratory thread, not your entire browsing history.

Chat messages are stored to audit.db after each turn, but prior messages are not yet injected into the LLM prompt, each query is answered independently. The session record is used for mode persistence and hint rotation, not for conversational context. Multi-turn prompt injection is planned for a future release.

Diagram 2: Web Query Flow: Client to Server, Session to Stream

The diagram below traces a complete web UI query round-trip, from the user typing a question to the hint chips updating after the response. The left column is the browser; the right column is the server.

A few things worth highlighting in this flow. The session_id lives only in React state - close the tab and it's gone. The mode determined at POST /sessions (step 1) persists for the lifetime of that tab and shapes hint generation at every done event (step 3 and 4). The HintEngine never calls the LLM - it reads the answer content and the session mode and applies deterministic rules to generate the three chips.

Adaptive Hints: No LLM Required

The hint chips - three clickable suggestions rendered below the chat input - update after every response. They're generated by a deterministic HintEngine, not an LLM. No API call, no extra cost.

The engine first classifies the wiki's health state when the session is created:

Mode	Condition	Initial hints
`NEW_WIKI`	Fewer than 5 pages	Guide user toward first ingest
`EXPLORER`	First session, healthy wiki	Offer tour queries
`HEALTH_CHECK`	Stale or contradicted pages exist	Surface lint and lifecycle actions
`POWER_USER`	Returning user, healthy wiki	Context-sensitive topic suggestions

After each assistant response, the done SSE event carries a next_hints array, three suggestions computed from the answer content and session mode. If the answer mentioned a specific page, the hints might suggest a follow-up on a related page. If the answer triggered a knowledge gap, the hints offer the suggested_searches as clickable options.

The design principle here is that hints should reflect where you are in the conversation, not where you were when you opened the browser. A user on a HEALTH_CHECK session who just asked about contradicted pages shouldn't see generic "try querying about X" chips, they should see "run lint", "list orphan pages", "archive contradicted page". The mode carries through the session, shaping every hint update.

What Makes This Architecturally Different

Most streaming chat interfaces work the same way: user sends a message, server calls the LLM, tokens stream back. There's no retrieval, no structured knowledge, no notion of whether the answer is backed by reviewed sources.

Synthadoc's streaming pipeline is a two-phase system where the first phase is a structured knowledge retrieval against a compiled, lifecycle-tracked wiki. The tokens you receive as they arrive are not hallucinated filler - they're synthesized from pages that passed lint, have known provenance, and carry a lifecycle state that tells you when they were last reviewed. The sources: N in the status event isn't decorative. It tells you before the first word of the answer how much of your wiki was relevant.

The session mode detection adds something I haven't seen elsewhere: the server classifies your wiki's health state when you open a session and uses that classification to shape every hint update for the rest of the session. A HEALTH_CHECK session doesn't give you generic "explore your wiki" prompts, it gives you "these pages need attention." The hints aren't cosmetic. They're a live triage system for wiki health.

Quick Demo

The CLI streaming and web UI are covered in the quick-start guide against the history-of-computing demo wiki:

CLI streaming: Step 5 - Query the pre-built wiki
Web Chat UI: Step 22 - Use the web chat UI

The full thing runs locally in about ten minutes:

git clone https://github.com/axoviq-ai/synthadoc.git
pip install -e ".[dev]"
synthadoc install history-of-computing --target ~/wikis --demo
synthadoc plugin install history-of-computing
synthadoc web   # opens browser

If you find Synthadoc useful, a ⭐ on GitHub helps the project reach more people: https://github.com/axoviq-ai/synthadoc.