DEV Community: Paul Chen

Interfaces Layer Design Pattern for AI Knowledgebase

Paul Chen — Tue, 23 Jun 2026 16:01:30 +0000

Most AI knowledge tools make a quiet assumption: the model is the worker, and the human is the caller.

You ask, the system retrieves pages, the LLM synthesises them, you get an answer. Clean pipeline. One direction.

That assumption breaks the moment a reasoning model needs to use the knowledge base as a tool - not to answer a question, but to do work. Search for related pages. Read a contradicted entry. Write a resolution. Commit a lifecycle transition. Then search again based on what it found.

Now the model is the caller. The knowledge layer is the called. Same storage. Opposite direction.

This is the coordination problem that most knowledge tools never have to solve, because they pick one mode and stay there. Building a system that handles both cleanly turns out to be the interesting design challenge.

The Four-Interface Pattern

The clearest way to see the tension is through the four surfaces a well-designed knowledge system needs to support:

CLI : a human (or script) calls the system, which calls an LLM, which returns an answer. Streaming for humans, structured output for pipes. The model is a service.

IDE/App Plugin : same direction, richer context. The plugin renders wikilinks as navigation, exposes a lifecycle dashboard for human decisions. Still: human calls system calls LLM.

Web Chat UI : conversational, multi-turn, natural language operations. "Run lint." "Show me stale pages." The system interprets these and dispatches them. Still: human calls system calls LLM.

MCP : a reasoning model connects to the knowledge layer as a set of tools. It decides what to search, what to read, when to write, which lifecycle state to commit. The system is now memory. The model is now the brain.

The insight is that the first three interfaces share a common structure. The fourth inverts it entirely.

Why the Memory Layer Needs Opinions

Here is where the design gets non-obvious.

When a reasoning model can freely read, write, and transition pages, you need the storage layer to enforce constraints the model might not. A lifecycle transition graph that blocks invalid state changes. An immutable audit trail that records every edit and its stated reason, regardless of whether a human or an agent made it. Provenance tracking that ties every page to its source material.

These aren't features for the human users. They're the constraints that make it safe to connect a reasoning model to the same base a human is actively maintaining.

A model can call write_page then lifecycle_transition. What it cannot do is move a page from stale to contradicted - that transition isn't in the allowed graph. The memory layer enforces this the same way it enforces it for a CLI command. It doesn't care which surface issued the instruction.

The audit trail records triggered_by: mcp and the model's stated reason. Run the lifecycle log the next morning and the full chain is there, whether it was a human clicking in a plugin or an agent running at 2 AM.

The Hint Engine Doesn't Disappear

One question that comes up: if a reasoning model generates its own next steps dynamically, what happens to the deterministic hint engine that suggests follow-up queries?

It serves the human-facing interfaces. No LLM cost, instant suggestions based on current wiki health, contextual to what the user just asked. A reasoning model doesn't need that, it generates its own chains based on what it finds. Two different users, two different workflows, same storage layer.

The coexistence is the point. The four-interface pattern isn't about choosing between human-driven and agent-driven workflows. It's about a knowledge layer that supports both without leaking the distinction into the storage model.

Reference Implementation

The pattern described here is implemented in Synthadoc, an open-source AI knowledge base tool. The video above walks through all four interfaces against a live demo wiki - CLI queries, Obsidian plugin lifecycle management, web chat multi-turn conversation, and connecting Claude Code as a reasoning brain over MCP.

If you're building something similar, the part worth paying attention to isn't the interface count, it's the audit trail and transition graph. Those are what make the memory layer safe to connect to a model that can write, not just read.

Google's OKF Is Proposing a Standard for How AI Agents Consume Domain Knowledge - Here's How Synthadoc Already Fits

Paul Chen — Mon, 15 Jun 2026 14:03:53 +0000

On June 12, 2026, Google Cloud published Open Knowledge Format (OKF) v0.1 - a vendor-neutral specification for how AI agents should consume structured knowledge. If you've been thinking about how to give your agents reliable, domain-specific memory beyond RAG over raw documents, this is worth your attention.

We built Synthadoc to solve exactly this problem, and the OKF spec reads like a formal description of what we built. Independent convergence on the same design is usually a sign that both teams found something real.

This post breaks down what OKF is, where Synthadoc already aligns with it, how we're converging on the remaining gaps, and why we think Synthadoc is one of the most important implementations the OKF ecosystem needs.

What Problem OKF Is Solving

Organizations today fragment their knowledge across metadata catalogs, wikis, Notion pages, code comments, and internal documents. When you want an AI agent to reason over that knowledge, you face a choice: either build a custom pipeline for each source, or dump everything into a vector store and hope similarity search finds the right facts.

OKF proposes a third path: a portable, file-based format for curated knowledge that any AI agent can consume without vendor-specific APIs or custom parsers. The format is deliberately minimal - markdown files with YAML frontmatter, organized in directories whose paths represent concept identity. Standard markdown links between files create a traversable knowledge graph.

The only required field is type. Everything else - title, description, tags, timestamp, resource - is optional. The philosophy is producer/consumer independence: the same format works whether you generated it from a data catalog, a wiki, or a code annotation pipeline.

This is a strong design philosophy. It means OKF bundles are readable by humans in any text editor, versionable in git, and consumable by any agent that can read files.

Google's reference implementation

Alongside the spec, Google shipped three things: an Enrichment Agent that ingests BigQuery metadata and emits OKF bundles (built on Google's Agent Development Kit with Gemini as the backend), a Visualizer that renders any OKF bundle as a self-contained interactive HTML file using Cytoscape.js, and three pre-built sample bundles for GA4 e-commerce, Stack Overflow, and Bitcoin datasets.

The Enrichment Agent is worth studying. It is a working OKF producer, a concrete proof that the format is implementable, not just specifiable. But its source material is structured: BigQuery tables, schemas, foreign-key relationships, data catalog metadata. This is one important slice of the OKF producer space.

The other slice - the harder one for most organisations - is unstructured source material: internal PDFs, research papers, spreadsheets, meeting notes, product documentation. That is the gap Synthadoc fills. Google's Enrichment Agent and Synthadoc are complementary producers in the same ecosystem, covering different source types with the same output format.

	Google Enrichment Agent	Synthadoc
Source material	BigQuery datasets, data catalogs	PDFs, XLSX, PNG, plain text
OKF role	Producer	Producer
Lifecycle management	None	5-state (draft → archived)
Conflict resolution	None	Contradicted state + auto-resolve
Human editorial gate	None	draft → active transition
LLM backend	Gemini only	Gemini, Anthropic, OpenAI, Groq, Qwen, Ollama, Deepseek, Minimax

Synthadoc's Design Was Already There

Synthadoc compiles raw documents (PDFs, spreadsheets, images, Youtube video, plain text) into a structured wiki - a collection of markdown files in a directory, each with YAML frontmatter, linked together by wikilinks. That wiki is then queryable by both humans (through Obsidian or a web UI) and AI agents (through the API).

Here's what a Synthadoc wiki page looks like:

---
title: Alan Turing
tags: [biography, theory, cryptography]
status: active
confidence: high
created: 2026-04-08
sources:
  - file: turing-biography.pdf
    hash: sha256:9f3a...
    ingested: 2026-04-08
---

# Alan Turing

Alan Mathison Turing (1912–1954) was an English mathematician and computer scientist...

The [[von-neumann-architecture]] that underlies modern computers shares conceptual roots
with Turing's stored-program ideas. Turing also influenced [[programming-languages-overview]]
by formalising what it means for a problem to be computable.

When we mapped this against the OKF specification, the alignment was immediate:

OKF requirement	Synthadoc
Markdown + YAML frontmatter	Native format for all wiki pages
`index.md` reserved for navigation	`wiki/index.md` in every wiki — auto-generated and maintained
Hierarchical concept organization	`ROUTING.md` groups pages into topic branches (`People`, `Hardware`, etc.) - logical hierarchy via manifest rather than physical directory structure
Markdown links as knowledge graph edges	`[[wikilinks]]` compile to a traversable knowledge graph
`tags` field	First-class frontmatter field on every page
`timestamp` (ISO 8601)	`created` and lifecycle timestamps throughout
No proprietary SDK required	Standard files, readable by any tool
Git-compatible	Every wiki is a git-managed directory

The core insight OKF encodes that knowledge for agents should live in plain files with structured metadata, not locked in a database, is the same insight Synthadoc was built on.

Converging with OKF

Synthadoc was designed independently before OKF was published, so a few intentional design choices don't yet match the spec exactly. Here's what they are and how we're closing each gap.

Wikilinks → standard markdown links.
OKF uses [Alan Turing](people/alan-turing.md); Synthadoc uses [[alan-turing]]. The graph structure is identical - only the syntax differs. We're adding an OKF export mode that rewrites wikilinks to standard markdown links at export time, so any Synthadoc wiki can be published as a fully spec-compliant OKF bundle.

Adding the type field.
OKF's only required field is type , a classifier for what kind of knowledge the page contains (concept, person, technology, event). Synthadoc's frontmatter uses status for lifecycle state and confidence for epistemic weight - complementary metadata, not a replacement. We're adding type to the schema in the next release. With that one field, every Synthadoc wiki page will be OKF-compliant out of the box.

sources → resource alignment.
OKF's resource field is a single URL pointing to an external source. Synthadoc's sources array goes deeper: each entry includes the file path, SHA-256 hash, file size, and ingestion date, and individual claims carry line-level citations. This is a superset of what OKF specifies, not a conflict. The OKF export will surface the primary source URL in the resource field while preserving Synthadoc's full provenance in the body.

These three changes give Synthadoc full OKF compliance while keeping the richer metadata that makes the wiki useful for humans and auditors, not just agents.

Where Synthadoc Goes Further

OKF v0.1 explicitly lists several open design questions - problems the specification hasn't solved yet. These are interesting to us because they're exactly what Synthadoc was designed to handle.

Contradiction resolution. When two source documents say conflicting things about the same topic, OKF has no answer yet. Synthadoc flags the affected page as contradicted and can automatically propose a reconciled view using LLM-assisted lint. The resolution is recorded in the page's lifecycle audit trail.

Stale information management. OKF acknowledges this as an open problem. Synthadoc has a stale lifecycle state triggered when a source document changes after initial ingest. Scheduled lint runs detect drift and surface pages that need review.

Adversarial review. OKF doesn't address how to audit knowledge for unsupported claims or overconfident assertions. Synthadoc's lint pass runs an adversarial review on each page, flagging claims that lack source support and writing findings directly into lint_warnings frontmatter.

These aren't listed as criticisms of OKF v0.1 is explicitly a "starting point for community feedback." They're listed because they represent the class of problems any serious knowledge management system needs to solve as it matures.

Synthadoc as an OKF Producer: Bridging Agents to Domain Knowledge

Here's the framing we find most compelling.

OKF describes a consumption format , a way for AI agents to read structured knowledge. What it doesn't describe is how that knowledge gets created, validated, kept current, and made queryable. That's the production problem.

Synthadoc solves the production problem. It ingests raw documents from your organization, compiles them into a structured wiki, validates each page through adversarial lint, manages the page lifecycle as sources change over time, and serves the result through a query API.

The output of Synthadoc, a directory of markdown files with YAML frontmatter, linked by wikilinks, with an index.md, is naturally aligned with what OKF describes as agent-consumable knowledge.

This means you can think of Synthadoc wikis as OKF-compatible knowledge bundles that cover the full lifecycle:

If you're building an agent that needs domain expertise, a customer support agent that needs product knowledge, a research agent that needs a scientific literature summary - a coding assistant that needs your internal architecture decisions - Synthadoc turns your existing documents into a structured, validated, self-maintaining knowledge base the agent can trust.

Our Perspective: What the Standard Still Needs

OKF v0.1 is a strong starting point, and we think the direction is right. But having built in this space, we have a few opinions worth putting on record, not as criticism, but as a contribution to where v0.2 should go.

A format standard without a quality standard is garbage-in, garbage-out.
If any tool can produce an OKF bundle, agents will inevitably consume contradictory, stale, or unsupported content formatted as valid OKF. The spec defines how knowledge is structured, but not how it is validated. The ecosystem needs production tooling that enforces quality - adversarial review, conflict detection, lifecycle management - before knowledge reaches an agent. A format alone doesn't make a knowledge base trustworthy.

Agent memory needs trust signals, not just content.
OKF sepc currently gives an agent no way to know how confident to be in what it's reading. A page written two years ago from a single unverified source should carry different weight than one reviewed last week, cross-referenced against multiple documents, and cleared by an adversarial lint pass. confidence, status, and claim-level provenance need to become first-class concepts in the spec, not optional extensions, if agents are to make consequential decisions from OKF knowledge.

Domain-specific knowledge is where this matters most.
General facts are already baked into LLM weights. What agents actually need is proprietary, time-sensitive, domain-specific knowledge that cannot be in training data - internal architecture decisions, evolving product specifications, specialist research. This is precisely where OKF is most valuable, and also where freshness and accuracy matter most. A standard that doesn't address knowledge currency will struggle in exactly the use case it's most suited for.

The human editorial layer is non-negotiable for consequential decisions.
Fully automated pipelines - ingest everything, trust everything - break down when agents act on the results. Knowledge intended for agent consumption needs a defined editorial state: has a human reviewed this page and cleared it for use? Synthadoc's draft → active lifecycle transition is exactly this gate. We think OKF should define a concept of editorial provenance so consumers can distinguish machine-generated drafts from human-validated knowledge.

Synthadoc v0.8.0

The OKF alignment discussed in this post ships alongside Synthadoc v0.8.0, released June 12, the same day as OKF v0.1. The release focuses on three areas:

Multi-turn conversation : the web UI maintains full session history across turns. Follow-up questions resolve correctly through context injection and automatic phrase rewriting. Long sessions compress older turns into summaries rather than discarding them. Sessions persist server-side and survive page refreshes; the sidebar shows history as a collapsible tree with turn-count badges.
Qwen DashScope + Claude Opus 4.8 : Qwen cloud models (qwen-plus, qwen-max, qwq-32b) now work via DashScope's OpenAI-compatible endpoint. New DashScope accounts receive 1 million free tokens valid for 90 days. Claude Opus 4.8 is supported for maximum answer quality.
Conversational job operations : the Action Agent handles live job queries directly in chat. Ask for job status and receive a real-time table; job IDs appear as clickable chips for drill-down. Multi-status filters ("show failed and skipped jobs") work across turns.

If you find Synthadoc useful, a ⭐ on GitHub helps the project reach more people: https://github.com/axoviq-ai/synthadoc.

References

OKF v0.1 Specification : GoogleCloudPlatform/knowledge-catalog
Synthadoc : github.com/axoviq-ai/synthadoc

Synthadoc: A Self-Invalidating Query Cache

Paul Chen — Thu, 11 Jun 2026 02:12:28 +0000

Most LLM queries feel expensive the second time. You've already asked your wiki "What is Moore's Law?" this morning. The answer hasn't changed - nothing in your wiki has changed - but if you ask again this afternoon, Synthadoc will hit the retrieval pipeline, build a prompt, call the LLM, and wait 2–10 seconds for an answer you already have.

v0.7.0 eliminates that. A query cache stores the result the first time, and serves it instantly on every repeat - with one rule: the cache invalidates itself automatically whenever your wiki changes. No manual flush, no TTL to configure. You never serve a stale answer, and you never pay twice for the same question.

This post covers the cache design and the performance benchmarks we ran against it. The streaming query pipeline and web chat UI that shipped alongside it in v0.7.0 are covered in the companion post: Synthadoc: Streaming Queries and a Local Web Chat UI.

Query Caching: When Not to Call the LLM

The cache design started from a specific observation: most queries against a domain wiki are repeated. A team maintaining a knowledge base about their software architecture will ask the same questions dozens of times - during onboarding, during incident reviews, during planning sessions. Every one of those calls hits the LLM and incurs both latency and cost.

The cache eliminates that. But the tricky part is knowing when to invalidate it.

Cache Key Design

key = SHA-256(normalized_question + "|" + wiki_epoch + "|" + provider_model)

Three components:

Normalized question - lowercased, whitespace-collapsed. "What is Moore's Law?" and "what is moore's law?" hit the same cache entry.

Wiki epoch - an integer counter on the server instance. It starts at 0 on startup and increments on every ingest job completion and every lifecycle state transition. When the epoch changes, the cache key for every question changes. Prior entries don't get deleted immediately, they just become unreachable. Old entries are cleaned up in a background sweep (entries more than 5 epochs behind current, or older than 7 days).

Provider/model - "openai/gpt-4o-mini" or "anthropic/claude-sonnet-4-6". Switching models invalidates the cache. A cached answer from a smaller model shouldn't surface when you've upgraded to a better one.

The epoch approach is what makes invalidation automatic. You don't call "invalidate cache" after an ingest, the epoch bump does it implicitly. Any query after a wiki change computes a new key that has never been seen, misses the cache, and calls the LLM fresh. The previous answer doesn't need to be deleted; it simply ceases to be looked up.

Diagram: Cache Lookup, Hit, Miss, and Epoch Invalidation

Measured Results

Rather than leaving the latency claims as estimates, we wrote a full performance test suite against the cache layer. All numbers below come from running pytest tests/performance/test_query_cache_perf.py locally on a Windows development machine with an SSD. Linux bare-metal numbers are consistently 30–40% better.

Chart 1 - Cache read latency distribution (500 reads, 200 cached entries)

P50 = 0.26ms, P95 = 0.34ms, P99 = 0.41ms against a 10ms SLO. The distribution is extremely tight - the persistent connection eliminates the per-call connection-open overhead that was the main source of outliers. Every percentile sits well inside the budget with headroom to spare.

Chart 2 - Cache hit vs miss latency at varying LLM speeds

The left panel uses a log scale because the gap is so large it can't be shown linearly. Cache hit P50 stays flat at ~0.25ms regardless of LLM speed - one shared persistent connection makes the hit path a pure queue-and-execute SQLite read with no file-open cost. The miss path scales directly with LLM latency. The right panel shows the resulting speedup factor:

Simulated LLM speed	Cache miss P50	Cache hit P50	Speedup
50ms (fast provider)	95ms	0.29ms	~330×
200ms (mid provider)	235ms	0.32ms	~730×
500ms (slow provider)	544ms	0.24ms	~2270×
2000ms (reasoning model)	2055ms	0.26ms	~7900×

The cache hit time is so small relative to any real LLM that the ratio is dominated entirely by provider latency. At reasoning models (o3-mini, MiniMax M2) a single saved round-trip reclaims 15–30 seconds of wall time.

Chart 3 - Concurrent readers: persistent connection scaling curve

Single reader: 0.5ms P95. Ten readers: 2.0ms. Twenty-five: 3.8ms. Fifty: 7.8ms. One hundred: 14.9ms. The curve is smooth and monotonically increasing - no Windows spikes, no non-monotonic jitter. All concurrent reads queue through one shared aiosqlite background thread; the connection-open overhead that caused the old instability is simply not there. For a local single-user tool the realistic ceiling is n=5–10 concurrent reads, where P95 is under 2ms. Even at n=100 the tail is well inside a 50ms budget.

Chart 4 - Cache vs no-cache throughput (queries/second)

The throughput advantage starts at 75.7× at n=1 and compresses to 4.1× at n=100. The compression is expected: asyncio.gather() parallelises the simulated LLM calls so the no-cache path scales nearly linearly with concurrency. The cache path, sharing one connection, serializes through the aiosqlite queue and grows sublinearly. But critically, the cache always wins by a wide margin - 4.1× at n=100 is far better than the 1.3× seen before the persistent connection fix. At realistic single-user concurrency (n=1–5), the advantage is 33–76×.

Estimated Latency Gains

A typical Synthadoc query against a mid-size wiki has two latency components:

Phase 1 (BM25 retrieval): 100–200ms. This runs regardless of cache.
Phase 2 (LLM synthesis): 2–10 seconds depending on provider, model, and answer length.

A cache hit skips Phase 2 entirely. The server reads the cached result_json from SQLite (~0.26ms P50 on SSD via a persistent connection), then emits a synthetic SSE burst at full network speed. The client receives what looks like a live streamed response, but the entire burst completes in under 100ms instead of waiting 2–10 seconds for the LLM. With a reasoning model provider, that gap widens to 15–30 seconds per query, the cache makes those queries feel instant.

The Cache Is Shared Across All Three Surfaces

CLI, Obsidian plugin, and Web Chat UI all share the same cache.db. If you ran synthadoc query "..." from the CLI this morning and the wiki hasn't changed, opening the Obsidian modal and asking the same question will hit the cache. The key is identical - same normalized question, same epoch, same model.

# Drop the entire cache - both LLM response cache and query cache
synthadoc cache clear
Cache cleared: 47 entries removed.

What Makes This Architecturally Different

The caching architecture differs from the typical approach of setting an explicit TTL (cache for 24 hours, or cache for one week). TTL-based caches are almost always wrong at the edges: they're either too short (you evict answers that are still valid) or too long (you serve stale content after a wiki update). Epoch-based invalidation is event-driven — the cache is valid until something in the wiki changes, exactly. You don't think about expiry. You ingest new content, and the next query automatically goes to the LLM. Every query after that hits the cache again until the next change.

Quick Demo

Query caching is covered in the quick-start guide against the history-of-computing demo wiki:

Query caching: Step 23 - Query caching

The full thing runs locally in about ten minutes:

git clone https://github.com/axoviq-ai/synthadoc.git
pip install -e ".[dev]"
synthadoc install history-of-computing --target ~/wikis --demo
synthadoc plugin install history-of-computing
synthadoc web   # opens browser

If you find Synthadoc useful, a ⭐ on GitHub helps the project reach more people: https://github.com/axoviq-ai/synthadoc.

[Boost]

Paul Chen — Wed, 10 Jun 2026 02:56:51 +0000

Paul Chen

Jun 8

Synthadoc: Streaming Queries and a Local Web Chat UI

#ai #llm #architecture #cli

9 min read

Synthadoc: Streaming Queries and a Local Web Chat UI

Paul Chen — Mon, 08 Jun 2026 21:02:24 +0000

There's a moment every Synthadoc user hits eventually. You've got forty or fifty compiled pages, a nightly ingest schedule running, lint keeping everything healthy. And then you open a terminal, type synthadoc query "...", and wait. The BM25 retrieval is instant. But then the cursor blinks. The LLM is thinking. You wait four seconds, six seconds, eight seconds. The answer eventually appears, all at once, like a curtain dropping.

That wait is fine the first time. It gets annoying on the tenth query when you're in a research session and you already know the answer is coming - you just want to read it as it forms, not stare at a blinking cursor.

v0.7.0 improves that. Streaming query output across all three query surfaces, and a local web chat UI that understands the health of your wiki. The architecture behind each of these turned out to be more interesting than I expected when we started building them. (A companion post covers the third feature in this release: a self-invalidating query cache.)

Diagram 1: What Changed in v0.7.0: The Architecture at a Glance

The diagram below maps the full Synthadoc architecture as it stands after v0.7.0. Items marked [NEW] are additions in this release; everything else was already present. The two features in this post touch two separate layers: the access layer gains a new client, and the engine gains new agents.

Two additions are the focus of this post:

Query Web UI (Access Layer): the new synthadoc web browser client using HTTP + SSE
Query (stream) · Action · Hint Engine (Agents): streaming query pipeline, live command execution, deterministic hint generation

Everything flows through the same server process per wiki. The CLI, Obsidian plugin, and Web Chat UI are all thin clients talking to the same HTTP + SSE endpoint. There's no separate service for the web UI. The MCP Server (shown as optional with a dashed border) is a fourth access path for AI tools like Claude Desktop or Cursor — it exposes the same wiki operations over the Model Context Protocol and requires opt-in setup.

Three Ways to Query Your Wiki and When to Use Each

Before getting into the streaming mechanics, it's worth laying out the three query surfaces Synthadoc now supports. All three can answer the same question, the difference is workflow fit.

CLI: when the query is part of a larger workflow

The CLI is where you go when a query isn't just a question, it's a step in something automated. The obvious case is CI/CD: a post-ingest job that queries the wiki to verify a newly compiled page before promoting it to active. Less obvious is using it as part of an agent integration, where an external orchestrator issues queries and parses the structured JSON output.

# Stream to terminal - tokens appear as the LLM generates them
synthadoc query "What were the main causes of the 2008 financial crisis?"

# Script mode - waits for full response, stdout is clean for piping
synthadoc query "Summarize page: moore's-law" --no-stream | jq .

# Force LLM call even if cache has a result - useful when wiki just changed
synthadoc query "What changed in the latest ingest?" --no-cache

The --no-stream flag is specifically for automation. Streaming output is beautiful on a terminal and disruptive in a pipeline. A script that parses stdout doesn't want token-by-token delivery, it wants a complete JSON blob when the query is done. --no-stream gives it that.

Obsidian Plugin: when you're in a research session

The Obsidian plugin exists for a different moment: you're writing a note, you need to check a claim against your wiki, and you don't want to leave Obsidian. The query modal (Ctrl/Cmd+P → Synthadoc: Query: ask the wiki...) is the right tool here. It renders [[wikilinks]] as clickable links, which means an answer that references related pages becomes navigable instantly.

The streaming behaviour in the Obsidian plugin mirrors the CLI, tokens appear as they arrive, citations follow at the end. The bypass cache checkbox is visible in the modal, unchecked by default. For researchers doing active ingest sessions, checking it once gets you fresh output without reaching for the terminal.

Web Chat UI: when you want a session, not a one-shot query

synthadoc web is the new entry. It opens a local chat interface in your browser, nothing leaves your machine, no cloud service, no authentication. It's designed for the kind of session that's too exploratory for the CLI and too long for the Obsidian modal.

Each turn is an independent query - the same cache applies here as in the CLI and Obsidian plugin. The chat history is displayed in the browser, but prior messages are not yet injected into the LLM prompt; multi-turn context injection is planned for a future release.

What the web UI adds over the other surfaces: operational commands. You can type "run lint", "show wiki status", "what pages are orphan pages?" or "schedule ingest every night at 9 PM" directly in the chat, and the Action Agent parses those and executes them live against your wiki, with results shown inline.

The screenshot above shows a live session against the history-of-computing demo wiki. The response to "What changed in the wiki this week?" includes a date-indexed ingest table, current lifecycle counts (Active: 80, Draft/Stale/Contradicted/Archived: all zero), and three action chips - "Activate a draft page", "Archive a stale page", "Restore an archived page to draft" - rendered inline as clickable buttons. The left panel shows prior session queries, allowing you to jump back into an earlier thread.

Streaming: The Architecture Behind a Two-Phase Response

Every Synthadoc query goes through two phases. Phase 1 is retrieval: BM25 search, routing, sub-question decomposition if needed. This is synchronous and fast, typically 100–200ms. Phase 2 is synthesis: the LLM generates an answer from the retrieved pages. This is where the latency lives.

The decision to stream only Phase 2 was deliberate. Phase 1 finishes before the first LLM token could possibly arrive, there's no partial retrieval state worth exposing. So the SSE protocol is clean:

The status events let the UI give immediate feedback. The user knows within 150ms whether the wiki found relevant pages or not before any LLM latency has accumulated. "sources: 3" in the synthesizing event tells them the answer is backed by three pages before they've read a single word of it.

The gap event fires only when the wiki doesn't have enough to answer confidently. Instead of a vague "I don't know," it returns suggested_searches - concrete ingest strings the user can use to fill the gap. These are generated by a secondary LLM call that decomposes the original question into targeted search queries - the same decomposition that drives sub-question retrieval, reused here to produce actionable ingest suggestions.

Provider Streaming Behavior

Not all providers stream in the same sense. API-based providers - OpenAI, Anthropic, Gemini, Ollama - emit tokens as they are generated, so the CLI and web UI render them character-by-character in real time. The latency shown in the SSE sequence above (one token every ~20ms) is what these providers deliver.

CLI subprocess providers - Claude Code (claude-code) and Opencode (opencode) - work differently. They run as child processes and write their output only when the process exits, so there is no per-token stream to intercept. Synthadoc runs the subprocess to completion, then emits the result word-by-word through the same SSE pipe. The words arrive in a rapid burst rather than a gradual flow - the total wait is the same, but the perceived streaming effect is a short pause followed by the full answer appearing almost at once.

If you are using a CLI subprocess provider and queries are timing out, increase the default timeout:

synthadoc query "..." --timeout 180

The default is 60 seconds, which is sufficient for API providers but may be short for subprocess providers on complex queries.

Session Management

Sessions live server-side in audit.db, in two tables: chat_sessions and chat_messages. The React UI stores only the session_id in memory - it's React state, not localStorage. This means sessions don't survive page reload, and every new browser tab starts fresh. This is a deliberate design choice: a session is tied to one exploratory thread, not your entire browsing history.

Chat messages are stored to audit.db after each turn, but prior messages are not yet injected into the LLM prompt, each query is answered independently. The session record is used for mode persistence and hint rotation, not for conversational context. Multi-turn prompt injection is planned for a future release.

Diagram 2: Web Query Flow: Client to Server, Session to Stream

The diagram below traces a complete web UI query round-trip, from the user typing a question to the hint chips updating after the response. The left column is the browser; the right column is the server.

A few things worth highlighting in this flow. The session_id lives only in React state - close the tab and it's gone. The mode determined at POST /sessions (step 1) persists for the lifetime of that tab and shapes hint generation at every done event (step 3 and 4). The HintEngine never calls the LLM - it reads the answer content and the session mode and applies deterministic rules to generate the three chips.

Adaptive Hints: No LLM Required

The hint chips - three clickable suggestions rendered below the chat input - update after every response. They're generated by a deterministic HintEngine, not an LLM. No API call, no extra cost.

The engine first classifies the wiki's health state when the session is created:

Mode	Condition	Initial hints
`NEW_WIKI`	Fewer than 5 pages	Guide user toward first ingest
`EXPLORER`	First session, healthy wiki	Offer tour queries
`HEALTH_CHECK`	Stale or contradicted pages exist	Surface lint and lifecycle actions
`POWER_USER`	Returning user, healthy wiki	Context-sensitive topic suggestions

After each assistant response, the done SSE event carries a next_hints array, three suggestions computed from the answer content and session mode. If the answer mentioned a specific page, the hints might suggest a follow-up on a related page. If the answer triggered a knowledge gap, the hints offer the suggested_searches as clickable options.

The design principle here is that hints should reflect where you are in the conversation, not where you were when you opened the browser. A user on a HEALTH_CHECK session who just asked about contradicted pages shouldn't see generic "try querying about X" chips, they should see "run lint", "list orphan pages", "archive contradicted page". The mode carries through the session, shaping every hint update.

What Makes This Architecturally Different

Most streaming chat interfaces work the same way: user sends a message, server calls the LLM, tokens stream back. There's no retrieval, no structured knowledge, no notion of whether the answer is backed by reviewed sources.

Synthadoc's streaming pipeline is a two-phase system where the first phase is a structured knowledge retrieval against a compiled, lifecycle-tracked wiki. The tokens you receive as they arrive are not hallucinated filler - they're synthesized from pages that passed lint, have known provenance, and carry a lifecycle state that tells you when they were last reviewed. The sources: N in the status event isn't decorative. It tells you before the first word of the answer how much of your wiki was relevant.

The session mode detection adds something I haven't seen elsewhere: the server classifies your wiki's health state when you open a session and uses that classification to shape every hint update for the rest of the session. A HEALTH_CHECK session doesn't give you generic "explore your wiki" prompts, it gives you "these pages need attention." The hints aren't cosmetic. They're a live triage system for wiki health.

Quick Demo

The CLI streaming and web UI are covered in the quick-start guide against the history-of-computing demo wiki:

CLI streaming: Step 5 - Query the pre-built wiki
Web Chat UI: Step 22 - Use the web chat UI

The full thing runs locally in about ten minutes:

git clone https://github.com/axoviq-ai/synthadoc.git
pip install -e ".[dev]"
synthadoc install history-of-computing --target ~/wikis --demo
synthadoc plugin install history-of-computing
synthadoc web   # opens browser

If you find Synthadoc useful, a ⭐ on GitHub helps the project reach more people: https://github.com/axoviq-ai/synthadoc.

Synthadoc: Staleness Detection, Full Audit Trails, and Four Export Formats - No Extra LLM Calls

Paul Chen — Mon, 01 Jun 2026 20:53:18 +0000

There's a category of problem that only shows up after you've been running an automated knowledge system for a while. The first month feels like magic - pages compile themselves, citations appear, everything is fresh. Three months later, you open a page about a library that shipped three breaking versions since the source was last ingested. The page looks perfectly healthy. The confidence is "high." The lint passed. And yet, everything in it is quietly wrong.

Static knowledge bases have no vocabulary for "this was true." Synthadoc v0.6.0 gives your wiki one.

Synthadoc release v0.6.0 ships two features that change how a wiki ages: a five-state page lifecycle machine that tracks content freshness with a permanent audit trail, and a wiki export system that serializes not just content but provenance, history, and cost, in four machine-readable formats, with zero additional LLM calls.

The 5-State Page Lifecycle

The core idea is simple: every page has a status that reflects what the system knows about it right now, not just what it says. That status moves through five states based on signals from ingest, lint, and the source files themselves.

Automatic transitions (system-triggered):

Transition	Trigger	Who
`→ draft`	New page created via ingest	IngestAgent
`draft → active`	Lint passes all structural and consistency checks	LintAgent
`active → stale`	SHA-256 hash of source file has changed since last ingest	LintAgent
`stale → draft`	Source re-ingested with`--force`; page updated	IngestAgent
`draft / active / stale → contradicted`	New source conflicts with this page; status set directly, bypasses transition API	IngestAgent

Manual transitions (user CLI commands):

Command	Transition	Description
`lifecycle activate <slug>`	`draft → active`	Promote without waiting for the next lint run
`lifecycle archive <slug>`	`draft / active / contradicted / stale → archived`	Retire the page; it's kept for reference
`lifecycle restore <slug>`	`archived → draft`	Re-admit the page; re-enters the lint queue

Note: active → stale and stale → draft have no user-facing CLI command, they are exclusively system-triggered by lint and re-ingest respectively. The only path out of contradicted is archiving it; you cannot promote a contradicted page directly to active.

Every single transition - automatic or manual - is permanently written to the audit database with a timestamp, the triggering agent or user, and a reason string. That's the part that actually matters. The state tells you where a page is. The log tells you how it got there and when someone last looked at it.

# Check the full history of a page
synthadoc lifecycle log alan-turing

Slug                      From           To             By           Timestamp              Reason
----------------------------------------------------------------------------------------------------
alan-turing               null           draft          ingest       2026-04-12T09:14:22    initial ingest
alan-turing               draft          active         lint         2026-04-12T09:31:07    all checks passed
alan-turing               active         stale          lint         2026-05-03T02:00:11    source hash mismatch
alan-turing               stale          draft          ingest       2026-05-03T08:22:55    re-ingest of stale page
alan-turing               draft          active         lint         2026-05-03T08:45:02    all checks passed

If you prefer a visual view, the same full cross-wiki audit trail is available in Obsidian under Synthadoc: Manage Page Lifecycle → Audit Log. Every transition shows colour-coded From/To state badges, the triggering agent or user, the timestamp, and the reason string - searchable by slug, filterable by state, paginated:

For a fleet-level view, synthadoc status gives a live summary across all five states, including pages sitting in candidates, along with an action hint for anything that needs attention:

synthadoc status

Wiki:         history-of-computing
Pages:        42
Jobs pending: 0
Jobs total:   187

Page lifecycle:
  active         38
  draft           2  <- run `synthadoc lint run` to promote
  draft (staged)  1  <- promote from candidates/ first, then lint
  stale           1  <- re-ingest needed
  contradicted    0
  archived        1

Two things worth knowing about what these numbers mean. First, Pages: 42 at the top counts only pages that have been admitted into wiki/ - pages still quarantined in wiki/candidates/ are excluded from that total. Second, draft and draft (staged) are distinct rows: draft is pages already inside wiki/ waiting for their first lint pass; draft (staged) is pages physically quarantined in wiki/candidates/ , and they haven't been promoted yet, have no lifecycle state, and are invisible to every part of the system until a human explicitly promotes them. The lifecycle section only shows draft (staged) when the count is greater than zero, so on a wiki with staging turned off you'll never see that row. The action hints tell you exactly what to do next for each group: run lint for drafts, re-ingest for stale pages, review and archive for contradictions.

If you prefer to manage lifecycle states visually, the Obsidian plugin surfaces the same data in Synthadoc: Manage Page Lifecycle → Current States. The table is sortable and filterable by state, shows the last transition timestamp and who triggered it, and gives you a one-click archive button per page. The contradicted chip makes it easy to find the pages that need attention first:

Candidates Staging: A Quality Gate Before Lifecycle Begins

The lifecycle machine handles what happens after a page enters the wiki. Candidates staging handles whether it enters at all.

Where a new page lands depends entirely on the staging policy configured for that wiki. There are three options:

off (default): every new page goes straight into wiki/ as draft. Staging is not involved.
all: every new page goes to wiki/candidates/ regardless of confidence. Nothing is admitted automatically - you review and promote everything.
threshold: IngestAgent checks the page's confidence rating against your configured minimum. Pages that meet or exceed it go directly into wiki/; pages that fall below it go to wiki/candidates/ for review.

wiki/candidates/ is a holding area excluded from search, context packs, and export. No downstream consumer sees a candidate. The page exists on disk, but it hasn't been admitted into the lifecycle yet, it has no audit log entry and doesn't appear in synthadoc status counts.

Here's how the three paths look end-to-end under the threshold policy:

The key design decision here: staging and lifecycle are orthogonal systems that compose cleanly. Staging decides admission. Lifecycle decides state after admission. A page in wiki/candidates/ has no lifecycle state yet, it's not in the audit log, it doesn't count in synthadoc status, and it doesn't appear in any export. The moment you promote it, it enters the lifecycle as draft and the lint queue picks it up on the next run.

This matters for teams that need a human gate on automated ingestion. Nightly ingest jobs run at 2AM, pull new sources, compile pages. They all land in candidates. A person reviews the list in the morning, promotes what looks right, discards what doesn't. The wiki only grows with reviewed content.

# Enable threshold staging: auto-promote high-confidence, hold everything else
synthadoc staging policy threshold --min-confidence high

# Morning review
synthadoc candidates list

Candidates (3):
  machine-learning-fundamentals    confidence: medium   ingested: 2026-05-31
  attention-mechanism              confidence: low      ingested: 2026-05-31
  transformer-architecture         confidence: medium   ingested: 2026-05-31

synthadoc candidates promote transformer-architecture
synthadoc candidates discard attention-mechanism

Wiki Export: Four Formats, Zero LLM Calls

Export was designed around one constraint: once your wiki is compiled, you shouldn't need to spend more API budget to serialize it. All four formats are computed entirely from the stored wiki state - no prompts, no completions, no waiting.

The --status flag is what makes export practically useful. When you're feeding a downstream LLM, you probably only want active pages — the ones that passed lint and haven't gone stale:

synthadoc export --format llms.txt --status active
synthadoc export --format json --status active --output exports/wiki.json

The --status contradicted flag is genuinely useful for forensics — you can export just the pages with conflicts and analyse them without touching the rest of the wiki.

The JSON format is the one worth drawing attention to specifically. Most wiki exports give you a flat document dump. This one gives you provenance at the sentence level (claims[] maps each paragraph to the exact source file and line range that generated it), the complete state transition history (lifecycle_history[]), and the per-page API cost to compile it. If you're building downstream tooling or reporting on knowledge quality, these three fields eliminate an entire layer of instrumentation you'd otherwise have to build yourself.

{
  "slug": "alan-turing",
  "status": "active",
  "ingest_cost_usd": 0.0012,
  "claims": [
    {
      "text": "Turing proposed the imitation game in 1950...",
      "source": "raw_sources/turing-biography.md",
      "lines": [42, 48]
    }
  ],
  "lifecycle_history": [
    { "from": null, "to": "draft", "by": "ingest", "ts": "2026-04-12T09:14:22" },
    { "from": "draft", "to": "active", "by": "lint", "ts": "2026-04-12T09:31:07" }
  ]
}

What Makes This Different

Most LLM wiki tools treat knowledge as append-only. You ingest, you query. There's no concept of a page going stale, no audit trail of who reviewed what and when, and no way to know that the page you're reading was compiled from a source that changed three months ago. They're effectively write-once databases with a chat interface on top.

Synthadoc's lifecycle machine makes the wiki temporally aware. A SHA-256 hash is stored for every source at ingest time. When lint runs (nightly, typically), it compares current hashes against stored ones. A changed source triggers an automatic active → stale transition with a timestamp. You know exactly which pages need attention and when they last didn't.

The other thing that separates Synthadoc architecturally is that it's not a retrieval pipeline with a generation step, it's a compilation pipeline. Every page is a synthesized artifact, not a retrieved chunk. That's why the JSON export can include ingest_cost_usd per page: because each page has a discrete compilation history, not a query-time cost that varies every time someone asks a question.

The combination of lifecycle tracking and export also enables something practical for teams: you can run synthadoc export --format llms.txt --status active as the input to a downstream agent, and you know exactly what you're giving it. No stale content. No contradicted pages. Just the subset of the wiki that the system has marked as reviewed and consistent.

Quick Demo

The quickest way to see the lifecycle machine in action is step 8 of the quick-start guide: Step 8 — Manage Page Lifecycle.

Export is step 21: Step 21 — Export Your Wiki.

Both steps work with the history-of-computing demo wiki, so you can run the full thing locally in about ten minutes against your selected LLM provider:

git clone https://github.com/axoviq-ai/synthadoc.git
pip3 install -e ".[dev]"
synthadoc install history-of-computing --target ~/wikis --demo
synthadoc plugin install history-of-computing

If you find Synthadoc useful, a star ⭐ on GitHub goes a long way toward keeping this project visible: https://github.com/axoviq-ai/synthadoc.

Synthadoc: Built an AI Judge for Our LLM Wiki Compiler - Here's What We Learned

Paul Chen — Sat, 23 May 2026 17:19:15 +0000

There's a particular kind of anxiety that comes from reading an LLM-compiled document you wrote six months ago. The prose is clean. The structure is coherent. And then you spot a sentence like "this approach became the industry standard by the late 1990s" - and you have no idea whether that came from your source material, or whether the model just... said it.

That's the problem we've been chipping away at with Synthadoc, an open-source LLM knowledge compiler. You feed it raw source files - PDFs, text docs, YouTube transcripts, web pages - and it synthesises them into a structured, queryable wiki. We shipped structural lint early on: contradiction detection, orphan page checks, broken link validation. But structural checks don't tell you whether the content is trustworthy. They tell you the wiring is correct, not whether the building should fall down.

In v0.5.0 we tackled this directly with two features. Here's how they work and why we think they matter.

The Adversarial Judge

The first instinct when you want to validate LLM output is to ask the same model to review itself. This doesn't work well. A model that just synthesised a page with a particular framing will tend to confirm that framing when asked to check it. It's not hallucinating - it's just consistent in the way humans are consistent with their own reasoning.

The fix is simple in principle: use a different model to review the output. A judge with different training data, different inductive biases, different tendencies to hedge or assert. In practice, that means configuring a second model - ideally from a different provider entirely - to act as a sceptical editor after each lint run.

# config.toml
[agents]
lint        = { provider = "minimax",   model = "MiniMax-M2.5" }
adversarial = { provider = "anthropic", model = "claude-sonnet-4-6" }

The adversarial pass runs after structural checks complete. Each page is sent to the judge with a single brief: find claims that are overstated, unsupported, or contradicted elsewhere in the source material. Results come back as {claim, concern} pairs, capped at a configurable limit per page (default: 2 - enough signal without drowning the author in noise).

Warnings are written directly into each page's YAML frontmatter:

lint_warnings:
  - claim: "Saved over fourteen million lives."
    concern: "This figure lacks scholarly consensus — historians dispute both
              the precision and the causal attribution to Turing's cryptanalysis alone."

The field is absent when no warnings exist, and cleared automatically if you run --no-adversarial , so stale warnings never persist past the last lint run that produced them.

When we ran this on the history-of-computing demo wiki for the first time, the judge flagged a paragraph about the second AI winter that described symbolic approaches as having been "largely abandoned." The compilation model had written that confidently. The judge noted it was contested - hybrid approaches continued in some research communities throughout the period. That's exactly the kind of subtle overstatement that's invisible to a format checker, and invisible to the model that wrote it.

Why this matters for teams

The adversarial pass produces documented evidence that a second, independent reviewer assessed each page. For enterprise knowledge bases - compliance documentation, research synthesis, internal policy wikis that audit trail has real value. "The LLM wrote it and we ran a second LLM over it with a different model" is a meaningfully stronger claim than "the LLM wrote it." Not a guarantee of accuracy. A documented process.

Claim-Level Provenance

The adversarial pass tells you a claim might be wrong. Provenance tells you exactly where it came from so you can check yourself.

The core mechanism is a citation annotation pass that runs during ingest, immediately after each page section is written. The model receives the numbered source text alongside the compiled paragraph and returns the paragraph with a ^[filename:L–L] marker appended:

Alan Turing proposed the Turing Test in 1950.^[turing-biography.txt:12-24]

The marker encodes the source filename and the exact line range in the raw document that supports the claim. These markers are stored in the page body, recorded in the audit database, validated by lint, and - in Obsidian - rendered as interactive chips in Reading View.

Click a chip and you get the Source Viewer: the referenced lines, highlighted, with surrounding context. For PDF sources, a pagemap sidecar resolves line numbers to the correct PDF page so the viewer offers an "Open PDF at page N →" button that navigates directly to the passage.

The annotation pass has a fallback: if the LLM fails, returns unparseable output, or references line numbers that don't exist, the original un-annotated section is used and the failure is recorded as an audit event. Ingest always completes. Results are also cached by section SHA-256, so re-ingesting an unchanged file doesn't incur an extra LLM call just to re-annotate the same paragraphs.

The full citation record

Every citation is stored in claim_citations in the audit database - page slug, source file, line range, and a 100-character excerpt of the annotated paragraph. You can query it directly:

synthadoc audit citations -w history-of-computing --page alan-turing
synthadoc audit citations -w history-of-computing --broken

Or browse the whole wiki via Synthadoc: View Page Provenance in the Obsidian command palette - a sortable, paginated table where every row opens the Source Viewer for that citation's exact line range.

Why this matters for teams

Traceability is one of the hardest blocks to enterprise AI adoption. "The model synthesised this from your documents" is not a citation. A recorded ^[turing-biography.txt:12-24] that links back to the primary source and surfaces that passage on click, that's closer. It doesn't eliminate the need for human review, but it makes human review fast enough to actually happen.

Two Layers of Trust

These features are complementary and designed to be used together. Provenance tells you where a claim came from. Adversarial review tells you whether you should trust it. Neither is sufficient alone.

A page that has full citation coverage and a second-model lint review is not a guaranteed-accurate page. But it's a page where you know where every claim came from, and where an independent model has registered any concerns it had. That feels like the right foundation for knowledge systems that get used in high-stakes contexts, as opposed to knowledge systems that sit in a folder and get ignored because nobody trusts them.

A Few Honest Notes

The annotation pass is itself an LLM call, which means it can make mistakes: misidentifying the supporting line range, over-annotating trivial sentences, or under-annotating genuinely sourced claims. We've found it to be reasonably accurate in practice, and the fallback-to-unannotated behaviour means a bad annotation result doesn't corrupt the page. But it's not ground truth. Treat it as strong evidence, not proof.

Similarly, the adversarial judge is only as good as the model you point it at. A judge that's too agreeable produces noise. A judge that's too aggressive produces fatigue. The adversarial_max_per_page cap (configurable, default 2) helps, but choosing the right model for the right domain still takes some experimentation.

If any of this is interesting or you're thinking through similar problems in your own knowledge tooling, feedback is genuinely welcome. The project is at https://github.com/axoviq-ai/synthadoc , and stars all appreciated.

Synthadoc: From Raw Documents to Domain Intelligence (Youtube)

Paul Chen — Sun, 17 May 2026 13:33:15 +0000

Synthadoc is an open-source AI knowledge engine that turns your raw documents - PDFs, research papers, web pages, spreadsheets - into a living, queryable wiki your team can actually trust.

In this video, we walk through how Synthadoc ingests sources, detects contradictions between them, keeps your knowledge base clean with lint and routing tools, and lets you query it with cited answers grounded in your own domain content. We also cover the Obsidian plugin, context packs for LLM grounding, staging for quality control, and the built-in audit trail for enterprise accountability.

Whether you're a researcher, a team lead managing a growing knowledge base, or a developer building AI-powered workflows. Synthadoc gives you the infrastructure to go from scattered documents to reliable domain intelligence.

🔗 GitHub: https://github.com/axoviq-ai/synthadoc
⭐ Star the project if you find it useful!

Synthadoc: From Raw Documents to Domain Intelligence (Youtube) https://youtu.be/rIGO6zi9XQE?si=nuhdnvdl9pcZ4NV_

Paul Chen — Sun, 17 May 2026 04:22:14 +0000

youtu.be

Synthadoc: Routing at Scale, Quality Gates, and the Knowledge Backend Pattern

Paul Chen — Mon, 11 May 2026 14:21:30 +0000

When we shipped v0.1.0, Synthadoc did one thing well: it turned raw sources into a structured wiki that got smarter with every ingest. v0.2.0 made that wiki searchable with hybrid BM25 + vector retrieval. v0.3.0 opened up the source types - YouTube transcripts, web search fan-out, CLI provider integration so your existing Claude Code or Opencode subscription could power the whole thing.

But a pattern kept surfacing in user feedback. Once a wiki crossed a few hundred pages, three problems appeared in a cluster: queries got slower, low-confidence pages polluted search results, and there was no clean way to pipe the wiki's structured knowledge into an agent prompt without getting back a synthesised answer when you wanted the raw evidence.

v0.4.0 addresses all three. This post walks through the design decisions behind each feature, the benchmark numbers, and why we think the third one, context packs, points at something larger than a feature.

The Scale Problem

BM25 is fast. On a 100-page wiki, a query completes in single-digit milliseconds. The problem is that BM25 is also undiscriminating: it scores every page against every query, regardless of how obviously irrelevant most of those pages are.

That's fine at 100 pages. At 1,000 pages with diverse topics - a personal research wiki covering ML, distributed systems, organisational theory, and management literature - you're running a full-corpus scan for every query. And because query decomposition splits one question into 3–5 sub-questions, you multiply that cost by 3–5 on every call.

We benchmarked the unrouted baseline across corpus sizes. The numbers aren't alarming until they are:

Corpus size	Full-corpus P95 latency	Routed P95 latency (2 of 10 branches)
100 pages	7 ms	7 ms
500 pages	38 ms	12 ms
1,000 pages	74 ms	18 ms
5,000 pages	112 ms	21 ms
10,000 pages	191 ms	24 ms

Routed search stays near-flat as the corpus grows, because the per-branch page count doesn't change even as the total wiki does. Full-corpus search grows with the wiki - not catastrophically, but noticeably, and that growth compounds across decomposed sub-queries.

The other problem at scale is more subtle: a query about treatment protocols for hypertension shouldn't touch the pages about distributed consensus algorithms. Not just for performance reasons - also because irrelevant pages can drift into synthesis and dilute the answer.

Feature 1: Routing Layer

Why It Matters for a Self-Growing Wiki

The scale problem above gets worse in a specific way that's easy to underestimate: a well-configured Synthadoc wiki doesn't stay at its initial size. With nightly scheduled ingests pulling from web searches, PDFs, YouTube transcripts, and curated source lists, a wiki can double from 200 to 400 pages within a few weeks, then reach 1,000 within a few months without the user doing anything manually. That's exactly the point.

But without routing, query quality degrades silently as that growth happens. Every new page increases the BM25 corpus, and the search engine has no way to know that "What are the treatment protocols for hypertension?" should not touch the 300 pages about software architecture. More pages means more irrelevant candidates competing to drift into synthesis, more false positives, and more latency - and none of it is visible until queries start returning noticeably diluted answers.

Routing is what makes autonomous growth sustainable. It's the mechanism that keeps query scope bounded to what's actually relevant, regardless of how large the total wiki becomes. The branch taxonomy is defined once; IngestAgent maintains it automatically from that point forward - every new page created by ingest is auto-placed into the most relevant branch, so ROUTING.md stays accurate as the wiki grows without manual intervention.

The routing layer introduces a file called ROUTING.md at the wiki root. Its format is intentionally simple - the same ## H2 → [[slug]] structure already used in index.md:

## People
- [[alan-turing]]
- [[grace-hopper]]
- [[ada-lovelace]]

## Hardware
- [[von-neumann-architecture]]
- [[eniac-computer]]
- [[transistor-and-microchip]]

## Networks
- [[internet-origins]]
- [[arpanet-history]]

User-owned. Scaffold creates it once from the current index structure and never rewrites it. The user defines the branch taxonomy; the system maintains it.

How Routing Works at Query Time

At query time, QueryAgent reads ROUTING.md (cached per session, invalidated on write), passes the branch headings and the user's query to the LLM, and receives back a short JSON array of the 1–2 most relevant branch names. BM25 then runs only over the slugs listed under those branches.

If ROUTING.md is absent, or if no branch scores above threshold, the system falls back to full-corpus search transparently - no error, no degraded output.

IngestAgent also uses routing: when a new page is created, it's slotted into the most relevant branch automatically. ROUTING.md stays consistent without manual maintenance.

Aliases

The other half of the routing feature is alias resolution. Anyone who maintains a personal knowledge base has personal terminology that diverges from canonical names. You might always call a concept by a shorthand, an acronym, or a translation - and BM25 will miss the connection because the strings don't match.

Aliases live in each page's YAML frontmatter:

---
title: Alan Turing
aliases: [turing, the turing paper, incomputability guy]
---

At query time, before BM25 runs, QueryAgent expands any alias matches in the query to their canonical slug. The user's internal vocabulary resolves to the wiki's vocabulary without re-learning what the LLM decided to call things.

ScaffoldAgent suggests initial aliases when generating a page. Users refine them in Obsidian's Properties panel.

Protected Scaffold Zone

One problem that appeared as wikis grew: users would hand-edit index.md to add an introduction, a personal note, a custom link - and the next scaffold run would erase it. Scaffold owned the whole file.

v0.4.0 introduces a marker line:

# My Research Wiki

My custom introduction and notes here.
Scaffold never touches anything above this line.

<!-- synthadoc:scaffold -->

## People
- [[alan-turing]] — Theoretical foundations of computation

Scaffold only regenerates content below . Everything above is preserved verbatim across every scaffold run. The marker is inserted automatically on the first scaffold run if absent.

Routing CLI

synthadoc routing init           # generate ROUTING.md from current index (one-time)
synthadoc routing validate       # report dangling slugs — dry run, no changes
synthadoc routing clean          # auto-remove dangling entries

routing validate is worth running after bulk ingests or manual page deletions:

Dangling slugs in ROUTING.md (3):
  [Hardware]  [[eniac-computer]]
  [People]    [[konrad-zuse]]
  [Networks]  [[arpanet-history]]

Scheduling for a Self-Growing Wiki

There are two distinct scheduling patterns worth setting up: nightly growth and weekly housekeeping.

Nightly growth — ingest pulls in new sources automatically:

synthadoc schedule add --op "ingest --batch raw_sources/" --cron "0 2 * * *" -w my-wiki

As each ingest job completes, IngestAgent writes the new page to wiki/ and appends its slug to ROUTING.md under the most relevant branch. No manual step needed — the routing index grows alongside the wiki.

Weekly housekeeping - three operations, run in sequence:

synthadoc schedule add --op "lint run"      --cron "0 3 * * 0" -w my-wiki   # Sunday 3 AM
synthadoc schedule add --op "scaffold"      --cron "0 4 * * 0" -w my-wiki   # Sunday 4 AM
synthadoc schedule add --op "routing clean" --cron "0 5 * * 0" -w my-wiki   # Sunday 5 AM

The order matters. Lint runs first and removes dead wikilinks left behind by deleted pages. Scaffold runs next and regenerates index.md to reflect the current page set - new categories get added, empty ones get removed. Routing clean runs last and prunes any dangling slug entries from ROUTING.md that no longer have a corresponding wiki page. After all three, the index and routing table are consistent with the actual state of the wiki.

One thing to be clear about: routing init is a one-time setup command, not something to schedule. Running it again would overwrite ROUTING.md and erase any branch customisations you've made since the initial setup. routing clean is the recurring maintenance command - it only removes entries for missing pages and never touches branch structure.

If you prefer to declare the schedule in config rather than via the CLI, add a [schedule] block to .synthadoc/config.toml and register everything in one step:

[schedule]
jobs = [
  { op = "ingest --batch raw_sources/", cron = "0 2 * * *" },
  { op = "lint run",                    cron = "0 3 * * 0" },
  { op = "scaffold",                    cron = "0 4 * * 0" },
  { op = "routing clean",               cron = "0 5 * * 0" },
]

synthadoc schedule apply -w my-wiki

With nightly ingests and a weekly maintenance trio in place, the wiki grows, stays accurate, and self-corrects - without requiring manual intervention.

Feature 2: Candidates Staging

The second feature addresses a different scale problem: quality at the write path.

Before v0.4.0, IngestAgent wrote new pages directly to wiki/ with no review step. A high-confidence page about a well-structured source landed right next to a speculative page inferred from a thin web article. Both entered BM25, both appeared in orphan detection and contradiction checks, and both showed up in synthesis - with no signal to distinguish them.

The Staging Concept

Candidates staging introduces a fork in the write path. Pages go to wiki/candidates/ instead of wiki/ when they don't meet a configurable confidence threshold. They're excluded from BM25, orphan detection, and contradiction checks until explicitly promoted.

The policy is configured in .synthadoc/config.toml:

[ingest]
staging_policy = "threshold"      # "off" | "all" | "threshold"
staging_confidence_min = "high"   # "high" | "medium" | "low"

Three policies:

Policy	Behaviour
`"off"`	All new pages go directly to`wiki/` - current behaviour, default
`"threshold"`	Pages meeting`staging_confidence_min` auto-promote; lower confidence → `wiki/candidates/`
`"all"`	Every new page requires explicit promotion, regardless of confidence

The config is hot-reloaded - a policy change takes effect on the next ingest job with no server restart.

The Staging Workflow

Reviewing Candidates

synthadoc candidates list

Candidates (3):
  eniac-computer    confidence: low     ingested: 2026-05-05
  konrad-zuse       confidence: medium  ingested: 2026-05-05
  arpanet-history   confidence: high    ingested: 2026-05-04

synthadoc candidates promote arpanet-history
synthadoc candidates discard eniac-computer
synthadoc candidates promote --all

Promotion does four things atomically: moves the file from wiki/candidates/ to wiki/, appends the slug to index.md under the best-matching category, appends it to ROUTING.md under the best-matching branch, and records the promotion in the audit trail. Discard deletes the file and records the reason.

Scheduling and Staging

Unlike routing and lint, candidates staging doesn't have a useful scheduled action. The two available commands — candidates promote --all and candidates discard --all - are too blunt to schedule safely. Scheduling promote --all on a timer would auto-approve exactly the pages the quality gate held back. Scheduling discard --all would silently delete pages you haven't reviewed yet. Neither command has a --older-than or --confidence filter that would make scheduled execution sensible.

The right integration is manual: run candidates list after a large batch ingest, or once a week if the wiki is growing quickly. It takes seconds. The review step is the intentional human checkpoint in an otherwise automated pipeline, it's where you decide what enters the wiki's searchable knowledge base.

If candidates are accumulating faster than you can review them, the right response is to adjust the policy rather than schedule a cleanup:

# Widen the auto-promote threshold - fewer pages need review
synthadoc staging policy threshold --min-confidence medium

# Or turn staging off entirely - if you trust all your sources
synthadoc staging policy off

The staging policy is the dial; the CLI review commands are for the remainder that needs a human decision.

Why This Matters

The practical effect of staging_policy = "threshold" on a high-volume wiki is significant. After ingesting 30 web articles from varied sources, typical results look like: 22 high-confidence pages auto-promote and enter the main wiki immediately; 8 lower-confidence pages wait in candidates. Those 8 include speculative pages, ambiguous slug assignments, and pages where the source was thin enough that the LLM wasn't confident what it was describing.

In the full-corpus approach, those 8 pages would be silently diluting every query that touched their topic. In the staged approach, they're visible and actionable.

Feature 3: Context Packs

The third feature came from a different direction. Users weren't asking "how do I get a better answer?" They were asking "how do I get the raw evidence so I can do something with it myself?"

QueryAgent synthesises. That's its job. But synthesis isn't always what you want. Sometimes you want the actual page excerpts - cited, bounded, ranked by relevance - to paste into an agent prompt, to attach to a report, to review before a meeting, or to feed into an automated pipeline.

Context packs are the answer.

How Context Packs Work

The ContextAgent reuses QueryAgent.decompose() and HybridSearch. It doesn't synthesise - it packs. Each page excerpt is included verbatim (up to its per-page limit), with attribution: slug, relevance score, confidence level, tags, source path.

Output Format

# Context Pack: history of early computing pioneers
Generated: 2026-05-09T14:22:01
Token budget: 4000 | Used: 3847 | Omitted: 2 pages (budget exceeded)

---

## [[alan-turing]] - relevance: 0.92
> Alan Turing developed the theoretical foundations of computation with his 1936
> paper "On Computable Numbers." The paper introduced the abstract Turing machine
> and proved the existence of undecidable problems...
Source: `wiki/alan-turing.md` | Confidence: high | Tags: people, mathematics

## [[grace-hopper]] - relevance: 0.87
> Grace Hopper pioneered compiler development and made programming accessible to
> humans. She developed the first compiler (A-0) in 1952 and later led the team
> that created COBOL...
Source: `wiki/grace-hopper.md` | Confidence: high | Tags: people, software

---

## Omitted — token budget exceeded
- [[von-neumann-architecture]] — ~820 tokens
- [[eniac-computer]] — ~650 tokens

CLI Usage

synthadoc context build "microservices patterns"
synthadoc context build "microservices patterns" --tokens 8000
synthadoc context build "microservices patterns" --output context.md

Token Budget Control

This is where context packs differ from simply querying the wiki. An external agent doesn't want an unbounded blob of text - it has its own context window to manage, its own prompt already taking up tokens, and its own cost constraints. The token_budget parameter gives the caller precise control over how much of the wiki's knowledge gets included.

Synthadoc packs pages greedily by relevance score until the budget is exhausted, then lists everything that didn't fit with estimated token counts. The calling agent knows exactly what it got and what it didn't - and can decide whether to request a larger budget, run a second more focused query, or proceed with what's there. There are no surprises about how much context the call will consume.

This predictability is what makes context packs suitable for production agentic pipelines. An agent orchestrator can reserve a fixed token slice for domain knowledge, call context/build with that exact budget, and get back a response that fits - every time, regardless of how large the underlying wiki has grown.

The REST API - Synthadoc as a Knowledge Backend

Context packs expose a POST /context/build endpoint that returns the same data as structured JSON. This is where the pattern becomes interesting.

An agent that needs grounding context before reasoning can call Synthadoc's REST API directly, get back a bounded, cited, ranked set of page excerpts, and inject them into its own prompt - without going through a synthesis step. Synthadoc becomes the knowledge layer, the agent provides the reasoning.

POST /context/build
{ "goal": "microservices patterns for high-throughput event processing", "token_budget": 4000 }

200 OK
{
  "goal": "...",
  "token_budget": 4000,
  "tokens_used": 3847,
  "pages": [
    {
      "slug": "event-driven-architecture",
      "relevance": 0.94,
      "excerpt": "Event-driven architecture decouples producers from consumers...",
      "source": "wiki/event-driven-architecture.md",
      "confidence": "high",
      "tags": ["architecture", "distributed-systems"]
    }
  ],
  "omitted": [
    { "slug": "kafka-internals", "estimated_tokens": 980 }
  ]
}

The existing MCP server exposes this as a native tool call. An agent running in any MCP-compatible host can call context/build before it reasons, get structured evidence back, and proceed with grounding it wouldn't otherwise have.

This is what we mean by the "knowledge backend pattern": Synthadoc manages accumulation, deduplication, contradiction detection, and retrieval. The calling agent manages reasoning and action. The division of labour is clean, the token envelope is caller-controlled, and the knowledge layer is persistent across agent sessions.

Other v0.4.0 Changes

Plugin Install CLI

synthadoc plugin install history-of-computing

In earlier versions, installing the Obsidian plugin required locating the plugins directory manually, copying files, and restarting Obsidian. v0.4.0 adds a single CLI command that installs the plugin directly into the active Obsidian vault. That's the entire CLI step - the rest is done in Obsidian.

AI Research Demo: Contradiction Detection End-to-End

The demos/ai-research/ demo now includes a PDF source (llm-benchmarks-q1-2026.pdf) that explicitly disputes a claim in the existing wiki. Running the demo shows the full contradiction detection and flagging lifecycle: source ingested, existing page status updated to contradicted, audit event recorded. The demo now covers all five IngestAgent decision paths - create, update, skip, flag, and contradiction.

Decision Cache Prompt-Awareness

A quiet but consequential fix: the LLM decision cache key previously included only the content hash and existing slugs. This meant that changes to purpose.md - the file that scopes what belongs in the wiki - were invisible to the cache. An ingest run after a purpose change would serve stale decisions for any source whose content hadn't changed. v0.4.0 includes the decision prompt itself in the cache key, so any change to purpose.md automatically busts the cache for all affected sources.

v0.1 to v0.4: What Changed

It's worth stepping back to see what the four versions have built:

Version	Core addition
v0.1.0	Ingest-time synthesis: sources become a structured wiki, not raw chunks
v0.2.0	Hybrid BM25 + vector search; query decomposition; knowledge gap detection; full audit trail
v0.3.0	YouTube and web search ingestion; CLI provider integration (Claude Code & Opencode support); CJK support; contradiction detection improvements
v0.4.0	Routing at scale; candidates staging for quality control; context packs as a knowledge backend API

The first two versions established the ingest-then-query model and made it trustworthy - every action audited, every contradiction surfaced. The third version expanded what you could ingest and who could afford to run it. The fourth version addresses what happens when the wiki grows to a size where the original flat model starts to show cracks.

The routing benchmarks are honest about where we are: 191ms for a full-corpus query across 10,000 pages is fine for interactive use, but it compounds across sub-questions and becomes a real cost in high-throughput agentic pipelines. The routed 24ms figure at the same corpus size is where we want the system to be. The benchmark-gated release process we introduced in v0.4.0 is the mechanism that ensures it stays there as the codebase evolves.

Try Synthadoc v0.4.0

Synthadoc v0.4.0 is available now on GitHub under the AGPL-3.0 licence. The quickest path:

git clone https://github.com/axoviq-ai/synthadoc.git
cd synthadoc
pip3 install -e ".[dev]"
synthadoc install history-of-computing --target ~/wikis --demo
synthadoc plugin install history-of-computing
synthadoc use history-of-computing
synthadoc serve

Then open Obsidian, open ~/wikis/history-of-computing as a vault, install the Dataview community plugin, and enable the Synthadoc plugin. The demo wiki runs against Gemini Flash 2.0, which is free-tier eligible — no cost to run a full ingest-query-lint cycle.

GitHub: https://github.com/axoviq-ai/synthadoc
Quick-start guide: https://github.com/axoviq-ai/synthadoc/blob/main/docs/user-quick-start-guide.md
Design document: https://github.com/axoviq-ai/synthadoc/blob/main/docs/design.md
Release notes: https://github.com/axoviq-ai/synthadoc/releases/tag/v0.4.0

Feedbacks are welcome. The routing taxonomy and context pack output format are both early - if your use case pushes against their current shape, we want to know.

Synthadoc: Your Coding Tool Is Now Your Wiki Brain

Paul Chen — Tue, 05 May 2026 16:02:35 +0000

If you use Claude Code or Opencode, you are already paying for an LLM subscription.

Before v0.3.0, running Synthadoc also required a separate API key - Anthropic, OpenAI, Gemini, or one of the others.

v0.3.0 removes that requirement. Set provider = "claude-code" in one config file and your coding tool subscription becomes the brain of your personal wiki. No additional API key. No additional cost.

Why This Matters Beyond Convenience

The obvious win is billing consolidation. But there is a more interesting point underneath it.

Coding tools like Claude Code and Opencode started as pair programmers. They answer questions about your codebase. They write functions, fix bugs, explain unfamiliar code. That is what the marketing says.

What they actually are is a general-purpose LLM with a subscription model - capable of anything the underlying model can do, not just code. The coding framing is a UI convention, not a capability limit.

Synthadoc v0.3.0 makes that explicit. The same Claude subscription you use to navigate a TypeScript monorepo can now synthesize your research documents, detect contradictions in your notes, and answer structured questions about your domain knowledge. The subscription becomes infrastructure, not a point tool.

This is the same pattern that made cloud storage interesting once it stopped being "backup for photos" and became "filesystem for any application." The value is in the generality.

How It Works

Synthadoc's LLM providers are abstracted behind a single LLMProvider interface. Every agent - IngestAgent, QueryAgent, LintAgent - calls provider.complete() and receives a structured response. The provider implementation handles everything underneath: API calls, authentication, response parsing, error handling, quota detection.

For cloud APIs (Anthropic, OpenAI, Gemini), the provider sends an HTTP request. For Claude Code and Opencode, a new CodingToolCLIProvider class takes a different route: it launches the CLI tool as a subprocess, passes the prompt via stdin, reads the response from stdout, and parses the output.

# .synthadoc/config.toml
[agents]
default = { provider = "claude-code" }
lint    = { provider = "claude-code" }

That is the entire configuration change. No API keys to set. No environment variables.

The server also accepts a runtime override:

synthadoc serve -w my-wiki --provider claude-code

This overrides config.toml for the lifetime of the server process - useful for quickly switching providers without editing files.

Figure 1 - All agents share one provider interface. CLI providers slot in without changing anything else in the stack.

What Works, What Does Not

The CLI provider route is not a perfect substitute for a direct API connection. Two limitations matter:

No vector embeddings. Synthadoc's optional semantic re-ranking (hybrid BM25 + vector search) requires an embed() call, which the CLI tools do not expose. When using a CLI provider, search falls back to BM25-only. For most wikis up to a few hundred pages, BM25 is fast and accurate - this only matters if you have enabled vector search and are running a large corpus.

Quota is shared. Your CLI provider subscription has a daily or session quota. Heavy ingest operations - batch-ingesting 50 documents, for example - consume from the same budget as your coding work. The engine detects quota exhaustion, permanently fails the job with a clear message, and does not retry. You resume after the quota resets.

Both are manageable trade-offs for a personal wiki. If you need vector search or are running a high-volume ingest, a direct API key is the better choice — pick from any of the eight providers based on your quality, vision, and budget requirements. If you want zero additional configuration and are comfortable with BM25 search, the CLI provider path is clean.

The Provider Landscape in v0.3.0

One of the quieter design decisions in Synthadoc is that the LLM provider is not baked into the product. You pick the one that fits your constraints:

Provider	API Key Required	Free Tier	Vision	Notes
Gemini Flash	Yes	1M tokens/day, no credit card	Yes	Default; best free option
Groq	Yes	Rate-limited	No	Fast, good for text-only
Ollama	No	Fully local	Model-dependent	Runs on your machine
DeepSeek	Yes	No (cheap)	No	Very low cost per token
Anthropic	Yes	No	Yes
OpenAI	Yes	No	Yes
MiniMax	Yes	No	Yes
Claude Code	No	Included with subscription	No	New in v0.3.0
Opencode	No	Included with subscription	No	New in v0.3.0

Figure 2 — Per-agent model routing from a single subscription. Ingest and query use Opus for synthesis quality; lint uses Haiku for speed and lower quota consumption. Two config lines; one bill.

DeepSeek is also new in v0.3.0 - routes through the OpenAI-compatible endpoint, very low cost per token for text-heavy ingest workloads, and the R1 reasoning model <think> tags are stripped automatically before the response reaches Synthadoc.

The point of this table is not that any one provider is the best. It is that you should not need to change your workflow to use the tool. If you are already set up with Anthropic for Claude Code, switching your wiki to the same provider takes one config line.

Quota Exhaustion and Reliability

One detail the implementation gets right: quota exhaustion is a permanent failure, not a retryable one.

Most LLM API errors are transient - a timeout, a 5xx from an overloaded endpoint. Synthadoc retries those with exponential backoff. Quota exhaustion is different. Retrying a quota-exhausted call wastes the next request, then the one after that, burning whatever small remaining budget exists.

When a CodingToolCLIProvider detects that the subprocess output indicates quota exhaustion, it raises CodingToolQuotaExhaustedException. The orchestrator catches this and calls fail_permanent() on the job - no retries, no backoff. The job sits in failed state with a clear message: "Claude Code quota exhausted. Resume after quota resets." You do not return to find a queue of 50 failed retries.

Small detail. Meaningful when it happens.

Getting Started with Claude Code or Opencode Provider

Prerequisite: Claude Code or Opencode must already be installed and logged in on your machine. If not, follow the Claude Code setup guide or the Opencode setup guide first.

# 1. Clone and install
git clone https://github.com/axoviq-ai/synthadoc.git
cd synthadoc
pip3 install -e ".[dev]"

# 2. Install the demo wiki
# Linux / macOS
synthadoc install history-of-computing --target ~/wikis --demo

# Windows
synthadoc install history-of-computing --target %USERPROFILE%\wikis --demo

# 3. Set as the active wiki (no -w needed from here on)
synthadoc use history-of-computing

4. Configure the CLI provider - edit ~/wikis/history-of-computing/.synthadoc/config.toml:

[agents]
default = { provider = "claude-code" }

Or skip editing config.toml entirely and pass --provider at startup:

# 5. Start the engine - no API key needed
synthadoc serve --provider claude-code

6. Install the Obsidian plugin - from the cloned synthadoc/ repo directory:

# Linux / macOS
cd ~/synthadoc   # or wherever you cloned it
vault=~/wikis/history-of-computing
mkdir -p "$vault/.obsidian/plugins/synthadoc"
cp obsidian-plugin/main.js obsidian-plugin/manifest.json "$vault/.obsidian/plugins/synthadoc/"

# Windows (cmd.exe)
cd %USERPROFILE%\synthadoc
mkdir "%USERPROFILE%\wikis\history-of-computing\.obsidian\plugins\synthadoc"
copy obsidian-plugin\main.js "%USERPROFILE%\wikis\history-of-computing\.obsidian\plugins\synthadoc\"
copy obsidian-plugin\manifest.json "%USERPROFILE%\wikis\history-of-computing\.obsidian\plugins\synthadoc\"

Then in Obsidian: fully quit and reopen, then:

Settings → Community plugins → Synthadoc → Enable, set Server URL to http://127.0.0.1:7070
Settings → Community plugins → Browse → search "Dataview" → Install → Enable (required for the live dashboard)

The full configuration reference is in docs/design.md - Coding tool CLI providers.

What Else Is in v0.3.0

The CLI provider work is one piece of a larger release. v0.3.0 also ships:

YouTube video ingestion - paste a URL, get a structured wiki page with executive summary and [MM:SS] timestamped transcript
Web search fan-out - one search query decomposes into sub-questions, ingests multiple sources, builds cross-references automatically
CJK multilingual support - Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports
Knowledge gap detection hardening - signal 5 redesigned for deterministic multi-aspect query scoring
DeepSeek provider - eighth provider, OpenAI-compatible, very low cost

Full release notes: github.com/axoviq-ai/synthadoc/releases/tag/v0.3.0.

Synthadoc is open source under AGPL-3.0 at github.com/axoviq-ai/synthadoc.

If you are already using Claude Code or Opencode daily, the marginal cost of running a structured personal wiki on top of the same subscription is zero. The marginal benefit compounds over time. That is a trade worth making.

Synthadoc: From YouTube to Wiki: How v0.3.0 Turns Any Content into Structured Knowledge

Paul Chen — Mon, 04 May 2026 19:10:00 +0000

You have a YouTube playlist of 40 conference talks. You have bookmarked 200 web articles. You have a folder of PDFs you keep meaning to read.

None of that is knowledge yet. It is a queue.

Synthadoc v0.3.0 drains the queue.

The Problem with "Saving" Things

Most people have a system for collecting information. Bookmarks, Notion pages, Pocket, starred emails. The collection grows. The retrieval never quite works. You remember you saved something about transformer attention mechanisms six months ago but cannot find it. You watch a 45-minute conference talk, absorb maybe 30% of it, and have no structured record of the rest.

The issue is not storage. It is synthesis. Saving a link preserves a pointer. It does not extract the claim, connect it to what you already know, or surface the contradiction with something you read last week.

Synthadoc v0.1.0 solved this for documents - PDFs, Word files, spreadsheets, images. v0.2.0 added hybrid BM25 + vector search so retrieval stayed sharp as the wiki grew. v0.3.0 extends the ingest surface to the two sources where most knowledge actually lives in 2026: video and the live web.

Ingesting a YouTube Video

The workflow is a single command:

synthadoc ingest "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

Or from Obsidian: open the Ingest: from URL... modal, paste the YouTube link, press Ingest.

What happens next:

Synthadoc fetches the video's caption track - no audio download, no third-party transcription API, no API key required.
The transcript is chunked with embedded [MM:SS] timestamps preserved so every claim is traceable to a specific moment in the video.
The LLM generates an executive summary: what the video is about, the main topics covered, and the key takeaway - in three to five sentences.
The full timestamped transcript follows the summary in the wiki page.
Cross-references to existing wiki pages are built automatically during ingest. If your wiki already has a page on "attention mechanisms" and the video mentions it, a [[attention-mechanisms]] wikilink appears in the new page.

Figure 1 — The YouTube ingest pipeline. One URL in; a structured, cross-referenced wiki page out.

The result is a wiki page that looks like this:

---
title: "The Illustrated Transformer"
status: active
confidence: medium
created: 2026-05-03T14:22:01
sources: [youtube.com/watch?v=...]
---

**Executive summary:** Jay Alammar's visual walkthrough of the Transformer
architecture. Covers self-attention, multi-head attention, positional
encoding, and the encoder-decoder structure using animated diagrams.
Key takeaway: the attention mechanism allows each token to "look at" every
other token in the sequence simultaneously, which is what enables parallelism
over RNNs.

---

[00:42] The problem with sequence models is that they process tokens
one at a time, making parallelisation during training difficult...

[03:15] Self-attention computes a weighted sum of all values in the
sequence. The weights come from a compatibility function between
a query and all keys...

No manual work. The video is now part of your wiki.

Web Search Fan-Out

The YouTube capability sits alongside a web search feature that works differently from a standard web search.

synthadoc ingest "search for: transformer attention mechanisms 2025"

Synthadoc does not return ten blue links. It:

Decomposes the query into 3–5 sub-questions covering different facets of the topic.
Searches the web for each sub-question independently.
Ingests the top results for each, synthesizing each source into a wiki page.
Builds cross-references across all the newly created pages.

A single web search command can add eight to fifteen structured pages to your wiki in one operation. The result is not a reading list - it is synthesized knowledge, cross-referenced and ready to query.

Why Timestamps Matter

One detail worth pausing on: the [MM:SS] timestamps in the transcript are not decoration.

When you later ask:

synthadoc query "what did the transformer paper say about positional encoding?"

The answer includes the source citation. Because the timestamp is embedded in the page body, the citation points not just to the video but to the moment in the video. You can verify the claim in thirty seconds by jumping to that timestamp.

This is the same principle that makes citations in academic papers useful. The claim is not just "somewhere in this source." It is traceable.

The Ingest Surface in v0.3.0

With v0.3.0, Synthadoc can ingest from the following source types in a single unified pipeline:

Source	How
PDF, Word, XLSX, CSV, TXT	Direct file extraction
Images (PNG, JPG, WEBP, etc.)	Vision LLM extracts text and structure
Web pages and articles	URL fetch + synthesis
YouTube videos	Caption extraction + executive summary
Web search results	Multi-query fan-out + synthesis
PowerPoint / presentations	Slide text extraction

Every source type produces the same output: a structured Markdown wiki page with frontmatter, wikilinks to related pages, and a traceable source reference.

Figure 2 — Every source type feeds the same pipeline. The wiki grows; the query quality compounds.

What the Wiki Looks Like After 30 Days

The compounding effect is the real story. After 30 days of normal usage — ingesting the things you would have saved anyway - a Synthadoc wiki typically contains:

50–150 pages covering your domain
Automatic cross-references linking related concepts
Contradiction flags where two sources disagree (Synthadoc surfaces these, you resolve them)
Orphan detection for pages no other page links to yet
Full audit trail of what was ingested, when, and at what cost

The 50th query to this wiki is dramatically smarter than the first, because every previous ingest has built the structure the query runs against.

That is the core idea from v0.1.0, still true - but now the inputs include everything.

Getting Started

# 1. Clone and install
git clone https://github.com/axoviq-ai/synthadoc.git
cd synthadoc
pip3 install -e ".[dev]"

2. Set an API key — Gemini Flash is the default (free tier, 1M tokens/day, no credit card).
Get a key at aistudio.google.com/app/apikey, then:

# macOS / Linux
export GEMINI_API_KEY=AIza…

# Windows
set GEMINI_API_KEY=AIza…

No API key? If you already have Claude Code or Opencode, set provider = "claude-code" in your wiki's config.toml instead - see docs/design.md.

# 3. Install the demo wiki
# Linux / macOS
synthadoc install history-of-computing --target ~/wikis --demo

# Windows
synthadoc install history-of-computing --target %USERPROFILE%\wikis --demo

# 4. Set as the active wiki (no -w needed from here on)
synthadoc use history-of-computing

# 5. Start the engine
synthadoc serve

6. Install the Obsidian plugin — from the cloned synthadoc/ repo directory, copy the pre-built plugin into your vault:

# Linux / macOS (run from the synthadoc/ repo root)
cd ~/synthadoc   # or wherever you cloned it
vault=~/wikis/history-of-computing
mkdir -p "$vault/.obsidian/plugins/synthadoc"
cp obsidian-plugin/main.js obsidian-plugin/manifest.json "$vault/.obsidian/plugins/synthadoc/"

# Windows (cmd.exe — run from the synthadoc\ repo root)
cd %USERPROFILE%\synthadoc
mkdir "%USERPROFILE%\wikis\history-of-computing\.obsidian\plugins\synthadoc"
copy obsidian-plugin\main.js "%USERPROFILE%\wikis\history-of-computing\.obsidian\plugins\synthadoc\"
copy obsidian-plugin\manifest.json "%USERPROFILE%\wikis\history-of-computing\.obsidian\plugins\synthadoc\"

Then in Obsidian: fully quit and reopen Obsidian, then:

Settings → Community plugins → Synthadoc → Enable, set Server URL to http://127.0.0.1:7070
Settings → Community plugins → Browse → search "Dataview" → Install → Enable (required for the live dashboard)

# 7. Ingest a YouTube video
synthadoc ingest "https://www.youtube.com/watch?v=YOUR_VIDEO_ID"

Synthadoc is open source under AGPL-3.0. The full quick-start guide, architecture docs, and demo wiki are at github.com/axoviq-ai/synthadoc.

👉 README: https://github.com/axoviq-ai/synthadoc#readme
👉 Quick-start guide: https://github.com/axoviq-ai/synthadoc/blob/main/docs/user-quick-start-guide.md

*Synthadoc v0.3.0 also ships CJK multilingual query support, knowledge gap detection hardening, a DeepSeek provider, and coding tool CLI providers (Claude Code, Opencode) - no separate API key needed if you already have a coding tool subscription. Full release notes in docs/design.md.