DEV Community: Rost

GFM vs CommonMark vs Pandoc Markdown: Syntax Compared

Rost — Sat, 08 Aug 2026 10:05:11 +0000

Markdown looks like one language until the same file renders differently on GitHub, Hugo, Obsidian, or Pandoc. And the problem is not that Markdown is unreliable.

It is that "Markdown" describes a family of related syntaxes, parsers, and platform features rather than a single universal document format. CommonMark defines a precise portable core, GitHub Flavored Markdown adds features useful for software collaboration, and Pandoc Markdown expands the language into a serious document-authoring format.

Choosing between them depends on where the document must render. A README file, a Hugo blog post, and an academic paper each have different requirements. This comparison is part of the broader documentation tooling picture and covers the formal dialects, platform-specific extensions, and practical portability rules so you can pick the right syntax for your target environment. For a quick syntax reference, the Markdown cheatsheet covers the essential formatting elements.

Markdown Is Not One Language

The original Markdown syntax was intentionally small and loosely specified. That made it easy to read and implement, but different parsers began interpreting ambiguous input differently.

CommonMark was created to define consistent parsing rules for the fundamental Markdown structures. GitHub Flavored Markdown, usually called GFM, builds on that foundation with several widely used extensions.

Pandoc Markdown takes a different approach. Instead of remaining a small web-oriented syntax, it adds document features such as citations, metadata, footnotes, definition lists, attributes, and mathematical notation.

A simplified relationship looks like this:

flowchart TD
    M[Markdown family] --> C[CommonMark core]
    C --> G[GitHub Flavored Markdown]
    C --> X[Other CommonMark-based renderers]
    M --> P[Pandoc Markdown]
    G --> GH[GitHub platform features]
    X --> H[Hugo with Goldmark]
    X --> GL[GitLab Flavored Markdown]
    P --> PDF[PDF and academic workflows]
    P --> DOCX[DOCX and publishing workflows]

This hierarchy is useful, but it is not exact inheritance in every implementation. Each renderer can enable, disable, or add syntax independently.

The Short Answer

Use CommonMark-compatible syntax when portability matters most.

Use GFM when writing README files, pull requests, issue templates, and technical documentation intended primarily for GitHub-compatible platforms.

Use Pandoc Markdown when the source document must become PDF, DOCX, EPUB, LaTeX, slides, or an academic paper with citations and metadata.

For a Hugo technical blog, use the CommonMark core plus the Goldmark extensions that your site explicitly enables. Do not assume every feature visible on GitHub will work merely because Hugo is described as GFM-compatible.

Opinionated take: if you only remember one rule for a Hugo technical blog, treat CommonMark plus GFM-style tables and task lists as the default, and treat everything else — footnotes, math, callouts, header attributes — as an explicit, tested extension rather than an assumed default. That single habit prevents most of the portability failures described below.

CommonMark: The Portable Core

CommonMark is a formal specification for the basic Markdown language. Its main contribution is not a large collection of features, but consistent parsing.

It defines how parsers should interpret:

Paragraphs
ATX and Setext headings
Block quotes
Ordered and unordered lists
Fenced and indented code blocks
Emphasis and strong emphasis
Links and images
Reference-style links
Inline code
Thematic breaks
Raw HTML blocks
Hard and soft line breaks

A CommonMark document can still behave differently at the presentation layer. CSS, syntax highlighting, heading anchors, HTML sanitization, and link policies are outside the core parsing rules.

CommonMark should therefore be treated as a reliable structural baseline, not a promise that every renderer will produce an identical page.

A Portable CommonMark Example

# Service Deployment

The service exposes a small HTTP API.

## Requirements

- Linux
- Docker
- 8 GB of memory

## Start the service

```bash
docker compose up -d
```

See the [configuration guide](configuration.md) for details.

This type of document works across almost every modern Markdown environment. It uses headings, paragraphs, lists, fenced code, and ordinary links without relying on dialect-specific extensions.

GitHub Flavored Markdown: CommonMark for Software Projects

GitHub Flavored Markdown is a formal dialect based on CommonMark. It preserves the CommonMark parsing model and adds features commonly needed in repository documentation and collaboration.

The formal GFM specification adds:

Pipe tables
Task list items
Strikethrough
Extended autolinks
Restrictions around some raw HTML tags

These extensions are now so common that many users think they are part of standard Markdown. They are not part of the CommonMark core.

GFM Tables

| Backend | Best use |
|---|---|
| Ollama | Local experiments |
| vLLM | Shared inference |
| SGLang | Structured workloads |

A strict CommonMark parser is allowed to treat this as ordinary paragraph text. A GFM-compatible parser recognizes it as a table. For a deeper look at table syntax and alignment options, see Tables in Markdown.

GFM Task Lists

- [x] Install Docker
- [x] Download the model
- [ ] Add monitoring

Task list syntax is useful in issues, pull requests, and project documentation. Outside a supporting renderer, it may appear as an ordinary list containing literal square brackets.

GFM Strikethrough

Use the ~~old endpoint~~ new endpoint.

Strikethrough is widely supported, but it is still an extension rather than portable CommonMark syntax.

GFM Autolinks

GFM recognizes more URL-like and email-like text without requiring angle brackets or explicit link syntax.

Visit https://example.com/docs for details.

In strict CommonMark, explicit autolinks use angle brackets:

<https://example.com/docs>

The explicit form is safer when a document must travel through unknown Markdown processors.

GitHub.com Supports More Than Formal GFM

A frequent source of confusion is the assumption that every Markdown feature visible on GitHub belongs to the GFM specification.

It does not.

GitHub.com adds platform-level processing and features around the GFM parser. Depending on the context, GitHub can support:

Mathematical expressions
Mermaid diagrams
Alerts
Issue and pull request references
User and team mentions
Commit references
Emoji shortcodes
Collapsible HTML sections
Color previews
Repository-relative links
Automatic heading anchors

Some of these features are syntax extensions. Others are post-processing behavior or integrations with GitHub data.

This distinction matters because another renderer may accurately claim GFM compatibility without implementing GitHub's math renderer, Mermaid integration, issue references, or alert styling.

GitHub Mermaid Diagrams

GitHub renders a fenced code block marked mermaid as a diagram:

```mermaid
flowchart LR
    A[Markdown] --> B[Rendered diagram]
```

A generic GFM renderer may display the same block as highlighted source code. The Markdown remains valid, but the enhanced rendering is platform-specific. For a practical introduction to Mermaid syntax, see the Mermaid Diagrams Quickstart.

GitHub Mathematical Expressions

GitHub supports inline and block mathematical expressions using dollar delimiters and additional escaping forms.

The cache size is approximately $2nlhd$ bytes.

$$
C = 2nlhd
$$

Math is not part of formal GFM. Moving this content to another renderer requires a compatible math extension such as KaTeX, MathJax, or Pandoc math support.

GitHub Alerts

GitHub supports alert-style block quotes such as:

> [!WARNING]
> Changing this setting clears the cache.

On GitHub, this can appear as a styled warning. On a plain CommonMark renderer, it usually appears as an ordinary block quote containing [!WARNING].

That fallback is readable, which makes GitHub alerts less dangerous than extensions that disappear completely. They are still not portable presentation elements.

Pandoc Markdown: Markdown as a Document Language

Pandoc Markdown is designed for document conversion rather than one particular website. It uses Markdown as the source syntax for producing HTML, PDF, DOCX, EPUB, LaTeX, presentations, and other formats.

Its default Markdown reader includes a large extension set. Important capabilities include:

YAML metadata blocks
Footnotes
Citations
Multiple table formats
Definition lists
Mathematical notation
Header identifiers and attributes
Code block attributes
Fenced divisions
Bracketed spans
Superscript and subscript
Strikeout
Line blocks
Numbered example lists
Raw LaTeX
Raw HTML
Automatic section numbering
Bibliography processing

Pandoc Markdown is much more expressive than CommonMark or formal GFM. That expressiveness makes it powerful for publishing, but less safe as an interchange format.

Pandoc Footnotes

Markdown has several incompatible dialects.[^dialects]

[^dialects]: CommonMark, GFM, and Pandoc Markdown are three
    important examples.

Footnote syntax is supported by many modern tools, but it is not part of CommonMark or formal GFM.

GitHub currently renders footnotes in several content contexts, but that is a GitHub platform feature rather than a formal GFM guarantee. A renderer claiming only CommonMark or GFM compatibility may not support them.

Pandoc Citations

PagedAttention improves KV cache memory management
[@kwon2023pagedattention].

With a bibliography file and citation style, Pandoc can resolve this into a formatted academic citation and bibliography.

pandoc article.md \
  --citeproc \
  --bibliography references.bib \
  --csl ieee.csl \
  -o article.pdf

The citation syntax remains readable in an unsupported renderer, but it will not become a formatted reference without Pandoc or another compatible citation processor. Pandoc's reader-side flexibility also underpins conversion workflows in the other direction — see converting Word documents to Markdown for a practical example of using Pandoc's extended dialect as an intermediate format.

Pandoc Definition Lists

CommonMark
: A precise specification for core Markdown.

GFM
: A CommonMark-based dialect with software-oriented extensions.

Pandoc Markdown
: An extended authoring format for document conversion.

Definition lists are useful in manuals, glossaries, and technical books. They normally degrade poorly in renderers that do not support them because the colon lines remain visible as plain text.

Pandoc Header Attributes

## Cache Configuration {#cache-config .deployment}

Pandoc interprets the braces as an explicit identifier and class list. Many other Markdown renderers show the attribute text directly in the heading.

This is one of the clearest examples of useful syntax that should not be placed in a document expected to render everywhere.

Pandoc Fenced Divisions

::: warning
Changing this option restarts the server.

Pandoc converts this into a structural division with a class. Templates, CSS, filters, or output writers can decide how that structure should appear.

Most CommonMark and GFM renderers do not recognize the fence. They display the colons and content as ordinary text.

CommonMark vs GFM vs Pandoc Markdown

The following matrix describes the formal dialects, not every feature added by GitHub.com, Hugo, Obsidian, GitLab, or another platform.

Feature	CommonMark	Formal GFM	Pandoc Markdown
Headings	Yes	Yes	Yes
Emphasis	Yes	Yes	Yes
Links and images	Yes	Yes	Yes
Block quotes	Yes	Yes	Yes
Ordered and unordered lists	Yes	Yes	Yes
Fenced code blocks	Yes	Yes	Yes
Raw HTML syntax	Yes	Restricted in some contexts	Yes
Pipe tables	No	Yes	Yes
Task lists	No	Yes	Yes
Strikethrough	No	Yes	Yes
Extended autolinks	No	Yes	Configurable
Footnotes	No	No	Yes
Citations	No	No	Yes
YAML metadata	No	No	Yes
Definition lists	No	No	Yes
Mathematical notation	No	No	Yes
Header attributes	No	No	Yes
Fenced divisions	No	No	Yes
Raw LaTeX	No	No	Yes
Bibliography processing	No	No	Yes

The word "No" does not mean a platform can never support the feature. It means the feature is not guaranteed by that dialect's formal specification.

Which Syntax Works on GitHub?

For README files, issues, pull requests, discussions, and wikis, GFM is the natural baseline.

You can generally use:

CommonMark syntax
Tables
Task lists
Strikethrough
Extended autolinks
Syntax-highlighted code fences
GitHub-specific references
GitHub-supported math
GitHub-supported diagrams
GitHub alerts
Footnotes where supported by the content surface

The portability risk begins when GitHub performs additional rendering beyond formal GFM. Mermaid diagrams, mathematical notation, issue references, and alert presentation may not survive outside GitHub.

For repository files that are also published elsewhere, test the source in the second renderer rather than treating GitHub preview as authoritative.

Which Syntax Works in Hugo?

Hugo uses Goldmark as its default Markdown renderer. Goldmark conforms to CommonMark and provides extensions compatible with important parts of GFM.

In a typical Hugo configuration, the following work well:

CommonMark structure
Fenced code blocks
Pipe tables
Strikethrough
Task lists
Automatic heading IDs
Syntax highlighting
Footnotes when the extension is enabled
Definition lists when enabled
Typographic substitutions when enabled

Hugo also adds features outside Markdown through:

Front matter
Shortcodes
Render hooks
Page resources
Internal reference functions
Template processing
Site configuration

These Hugo features do not travel with the Markdown file. For a practical example of Hugo deployment, see Deploy Hugo to AWS S3.

Hugo Front Matter Is Not Markdown Content

A Hugo page commonly starts with YAML, TOML, or JSON metadata:

---
title: "Markdown Compatibility"
description: "Compare Markdown dialects and renderers."
date: 2026-07-31
tags:
  - Markdown
  - documentation
---

Pandoc can also recognize YAML metadata blocks, but it interprets fields according to its own templates and writers. GitHub normally displays the block as a YAML-like section or treats it as repository metadata only in specific systems.

The same syntax can therefore be recognized in more than one tool without having the same semantics.

Raw HTML in Hugo

Goldmark does not render potentially unsafe raw HTML by default in a standard Hugo configuration.

A block such as:

<div class="notice">
  Restart the service after changing this value.
</div>

may be omitted unless raw HTML rendering is enabled or the content is implemented through a shortcode or render hook.

For a controlled technical blog, enabling raw HTML can be reasonable. It still makes the source less portable and should be a deliberate site-level decision.

Mermaid in Hugo

A fenced mermaid block is still just a code block unless the Hugo theme, render hook, shortcode, or JavaScript pipeline transforms it into a diagram.

GitHub and Hugo may therefore accept identical Mermaid source while using completely different rendering mechanisms.

Which Syntax Works in Pandoc?

Pandoc can read several Markdown dialects explicitly:

pandoc --from=markdown input.md
pandoc --from=commonmark input.md
pandoc --from=gfm input.md
pandoc --from=commonmark_x input.md

This is one of Pandoc's most useful portability features. The operator can tell Pandoc which dialect the source claims to use instead of relying on a vague .md file extension.

Pandoc also lets you enable or disable individual extensions:

pandoc \
  --from=markdown-footnotes-pipe_tables \
  input.md \
  -o output.html

Or start from a narrower format and add one feature:

pandoc \
  --from=commonmark+footnotes \
  input.md \
  -o output.html

You can inspect available extensions with:

pandoc --list-extensions=markdown
pandoc --list-extensions=commonmark
pandoc --list-extensions=gfm

This extension model is powerful, but it means "Pandoc Markdown" is not always one fixed configuration. Build commands and defaults files are part of the document specification.

Which Syntax Works in Obsidian?

Obsidian stores notes as Markdown files, but its authoring model includes several application-specific features.

Common examples include:

Wiki links
Embedded notes
Embedded files
Callouts
Block references
Tags
Properties
Highlighting
Comments
Dataview queries from plugins
Application-specific URI links

A wiki link such as:

[[Markdown Compatibility]]

is meaningful inside an Obsidian vault. GitHub, CommonMark, and a default Pandoc reader normally display it as literal bracketed text.

An embed is even more application-specific:

![[compatibility-table]]

The referenced content is not present in the file itself. Exporting or publishing the note therefore requires an expansion step that resolves the embed.

Obsidian is a good example of why storage in .md files does not guarantee Markdown portability. For a practical look at Obsidian as a knowledge management tool, see Obsidian for Personal Knowledge Management.

Which Syntax Works in GitLab?

GitLab Flavored Markdown uses CommonMark as its core and includes GFM features such as tables and task lists. It then adds GitLab-specific behavior including cross-references, mathematical notation, diagrams, and other collaboration features.

A README written in conservative GFM usually moves between GitHub and GitLab without major damage.

Platform integrations do not travel as reliably. Issue references, user mentions, diagrams, math handling, and special block syntax can behave differently even when the basic Markdown remains readable.

Platform Support Matrix

This matrix describes common default behavior. Themes, plugins, extensions, and configuration can change individual cells.

Feature	GitHub	Hugo Goldmark	Pandoc	Obsidian	GitLab
CommonMark core	Yes	Yes	Yes	Mostly	Yes
Pipe tables	Yes	Yes	Yes	Yes	Yes
Task lists	Yes	Yes	Yes	Yes	Yes
Strikethrough	Yes	Yes	Yes	Yes	Yes
Footnotes	Yes	Configurable	Yes	Yes	Yes
YAML metadata	Context-dependent	Front matter	Yes	Properties	Context-dependent
Math	Yes	Requires setup	Yes	Yes	Yes
Mermaid	Yes	Requires setup	Output-dependent	Yes	Yes
Citations	No native bibliography	Requires tooling	Yes	Plugin-dependent	No native bibliography
Definition lists	No	Configurable	Yes	Limited	Limited
Header attributes	Limited	Renderer-dependent	Yes	Limited	Limited
Wiki links	No	No by default	No by default	Yes	Wiki-dependent
Callouts or alerts	GitHub syntax	Theme or shortcode	Template-dependent	Obsidian syntax	GitLab syntax
Raw HTML	Sanitized or restricted	Disabled by default	Yes	Context-dependent	Sanitized or restricted

"Yes" still does not guarantee identical HTML or visual presentation. It means the environment recognizes the general feature.

Syntax That Is Usually Safe Everywhere

The safest portable subset includes:

ATX headings using #
Ordinary paragraphs
Blank lines between blocks
- for unordered lists
1. for ordered lists
Fenced code blocks using backticks
Inline code using backticks
Emphasis using *text*
Strong emphasis using **text**
Ordinary links
Ordinary images
Block quotes
Thematic breaks
Explicit angle-bracket autolinks

An intentionally conservative document might look like this:

# Deployment Guide

This guide explains how to deploy the service.

## Requirements

- Docker
- Linux
- A supported GPU

## Configuration

Create a file named `compose.yaml`.

```yaml
services:
  application:
    image: example/application:1.0
```

For more information, see the [configuration reference](config.md).

> Back up existing data before upgrading.

This syntax travels well because it does not depend on tables, footnotes, attributes, callouts, or platform processing.

Syntax That Commonly Breaks

Portability problems tend to cluster around a small number of features.

Pipe Tables

Pipe tables are well supported by GFM-oriented tools, but not by strict CommonMark.

A table can degrade into unreadable text when passed through a parser that does not recognize it. For highly portable documents, consider short lists or semantic HTML generated during a build step.

Footnotes

Footnote syntax has become common, but it remains an extension.

Different tools may:

Support only one footnote format
Place footnotes differently
Generate different identifiers
Reject multi-paragraph footnotes
Render the source literally

Use footnotes when the publishing pipeline is known. Avoid depending on them in README files that must render across arbitrary systems.

Heading IDs and Attributes

This Pandoc syntax is not portable:

## Installation {#installation .procedure}

Use an ordinary heading and let the renderer generate its own anchor when portability matters.

Also avoid hard-coding links to auto-generated heading IDs unless every target uses the same slugification rules.

Callouts and Alerts

GitHub, Obsidian, GitLab, MkDocs, Docusaurus, and Hugo themes can all support callout-like blocks, but they often use different syntax.

A portable fallback is an ordinary block quote:

> Warning: Back up the database before upgrading.

It is less visually impressive, but it preserves meaning everywhere.

Wiki Links

Wiki links are concise inside knowledge-management tools:

[[KV Cache]]

They are poor interchange syntax because the target path, file name, heading rules, and resolution behavior belong to the application.

Use standard Markdown links in content intended for publication:

[KV cache](kv-cache.md)

Raw HTML

Raw HTML is the usual escape hatch when Markdown cannot express a layout. It is also a common portability and security failure.

A renderer may:

Remove the HTML
Escape it
Sanitize selected elements
Allow blocks but not inline elements
Refuse Markdown parsing inside HTML
Pass it unchanged only in trusted mode

Use raw HTML only when the publishing target is controlled.

Mathematical Notation

Dollar-delimited math is popular but not universally interpreted.

The source:

The complexity is $O(n^2)$.

may become:

Rendered mathematics
Ordinary text with dollar signs
Incorrect emphasis
Input to a different math parser

Choose one math pipeline and test it in every target environment.

Mermaid and Other Diagram Blocks

A Mermaid code fence is syntactically safe because unsupported renderers normally display it as code.

The semantic result is still different. Readers may see a rendered architecture diagram on GitHub and raw Mermaid source in another environment.

This is graceful degradation, not true compatibility.

The Three Layers of Markdown Compatibility

It helps to separate compatibility into three layers.

Layer 1: Parsing Compatibility

Does the parser recognize the structure?

Examples include headings, tables, footnotes, and fenced divisions.

Layer 2: Transformation Compatibility

Does the platform apply additional processing?

Examples include:

Rendering Mermaid
Resolving citations
Expanding wiki links
Linking issue numbers
Processing shortcodes
Generating a table of contents

Layer 3: Presentation Compatibility

Does the result look and behave appropriately?

Examples include:

Table styling
Syntax highlighting
Alert colors
Heading anchors
Responsive images
Footnote placement
Math fonts

Two platforms can parse identical syntax while producing substantially different presentation.

A Better Portability Model

Instead of asking whether a file is "valid Markdown," ask four narrower questions:

Which dialect is the source written in?
Which parser reads it?
Which extensions are enabled?
Which platform transformations run afterward?

For example:

Dialect: CommonMark plus GFM tables
Parser: Goldmark
Extensions: tables, strikethrough, task lists, footnotes
Platform: Hugo
Additional processing: render hooks and Mermaid JavaScript

That description is much more useful than saying "the site uses Markdown."

Choosing a Dialect by Use Case

README Files

Use GFM.

README files benefit from:

Tables
Task lists
Fenced code
Autolinks
Strikethrough
GitHub references

Avoid excessive dependence on GitHub-only features when the repository is mirrored to GitLab, rendered on a package registry, or included in generated documentation.

Hugo Technical Articles

Use CommonMark-compatible Markdown with a documented Goldmark extension set.

Tables, code fences, footnotes, and Mermaid can be reasonable because you control the build pipeline. Prefer Hugo shortcodes or render hooks over embedding large amounts of raw HTML.

Keep Hugo-specific syntax isolated and easy to find.

Academic Documents

Use Pandoc Markdown.

Citations, bibliography processing, footnotes, metadata, mathematical notation, cross-references, and conversion to PDF or DOCX justify the reduced portability.

Store the Pandoc command, defaults file, filters, bibliography, and templates beside the source. The source file alone does not fully describe the build.

Books and Long-Form Documentation

Pandoc Markdown is usually the strongest of the three options when multiple output formats matter.

Definition lists, citations, attributes, metadata, and structured transformations become more important as document complexity grows.

For web-only documentation hosted in a Git repository, GFM or a CommonMark-based documentation generator may remain simpler.

Notes and Personal Knowledge Bases

Use the native syntax of the selected notes application when application features provide real value.

Obsidian wiki links, embeds, and callouts are useful inside a vault. Treat export as a compilation process rather than assuming the raw files are already portable publications.

Shared Documentation Across Unknown Systems

Use a conservative CommonMark subset.

Avoid:

Wiki links
Platform alerts
Header attributes
Citations
Raw HTML
Custom containers
Application embeds
Shortcodes

Portability usually requires giving up convenience features.

Practical Rules for Portable Markdown

Start with CommonMark Structure

Use CommonMark for the document skeleton:

Headings
Paragraphs
Lists
Links
Images
Block quotes
Code blocks

This ensures that the main meaning survives even when optional extensions fail.

Add GFM Features Deliberately

Tables and task lists are reasonable when all important targets support them.

Do not assume "most tools support GFM" without testing the exact target. Some claim GFM compatibility while enabling only selected extensions.

Isolate Platform Extensions

Keep platform-specific syntax in clearly identifiable blocks.

For example, centralize Hugo shortcodes, Pandoc citations, or Obsidian embeds rather than scattering them through every paragraph.

Isolation makes later conversion easier.

Prefer Graceful Degradation

A Mermaid block degrades into readable source code. A GitHub alert degrades into a block quote.

A wiki embed may degrade into an unexplained file name, while a Pandoc fenced division may expose punctuation around the content.

Choose extensions whose fallback remains understandable.

Do Not Depend on Auto-Generated Heading IDs

Heading anchor algorithms differ between GitHub, Hugo, Pandoc, and documentation generators.

For cross-document links, use renderer-supported explicit IDs only when the target pipeline is controlled. Otherwise, link to the document rather than a generated fragment.

Keep Build Configuration with the Content

Pandoc extensions, Hugo settings, plugins, filters, and JavaScript integrations determine how Markdown behaves.

Commit relevant configuration files with the source:

content/
  article.md
pandoc.yaml
references.bib
config/
  _default/
    markup.yaml
layouts/
  _default/
    _markup/

A .md extension alone does not capture the publishing environment. For a structured approach to documenting these decisions, see Decision Records for AI-Driven Development.

Test Markdown Against Every Important Target

Visual preview in one editor is not enough. The editor may support a richer dialect than the production renderer.

For Pandoc, test explicit input formats:

pandoc --from=commonmark article.md -o commonmark.html
pandoc --from=gfm article.md -o gfm.html
pandoc --from=markdown article.md -o pandoc.html

Warnings and visible source punctuation reveal which features are dialect-specific.

For Hugo, build the production site:

hugo --gc --minify

Then inspect the generated HTML rather than relying only on an editor preview.

For repositories, view the committed file on the actual hosting platform. Local Markdown extensions in VS Code may not match GitHub or GitLab.

Troubleshooting Common Rendering Mismatches

When a file that worked on one platform breaks on another, the failure usually falls into one of a handful of repeatable patterns. The table below lists the symptom as you would actually see it, the most likely cause, and a concrete command or check to confirm and fix it.

Symptom	Likely cause	Confirm and fix
A pipe table renders as one long paragraph with visible `\	` characters	Renderer is strict CommonMark without a tables extension
`[^note]` stays inline as literal text instead of becoming a superscript footnote marker	The footnote Goldmark extension is not enabled	In Hugo, check for `footnote` under `markup.goldmark.extensions` in `hugo.yaml`, rebuild with `hugo --gc --minify`, and look for `<sup>` in the generated HTML
A ``mermaid` fence shows as plain grey source code instead of a diagram	The platform performs no post-processing on the fenced block	GitHub renders it natively; Hugo needs a render hook, shortcode, or JS pipeline — check the built HTML for `<pre><code class="language-mermaid">` versus an `<svg>`
`## Heading {#id}` shows the literal curly braces in the rendered heading text	Header attribute syntax is Pandoc-specific, not CommonMark or GFM	Remove the attribute syntax for portable output, or pre-convert with `pandoc --from=markdown --to=gfm file.md -o out.md`
`[[Note Name]]` displays as literal double square brackets	Wiki link syntax is application-specific to tools like Obsidian	Replace with a standard Markdown link, `[Note Name](note-name.md)`, before exporting outside the vault
`[@kwon2023pagedattention]` stays as plain bracketed text instead of a formatted citation	No bibliography or citeproc pass was applied	Re-run with `pandoc --citeproc --bibliography=refs.bib input.md -o output.pdf` and confirm the CSL style is specified
`> [!WARNING]` renders as an ordinary quoted paragraph instead of a styled alert	Alert styling is a GitHub.com platform feature, not part of formal GFM	Expected outside GitHub; keep the wording readable as a plain block quote rather than depending on the color styling

This is the fastest first pass before assuming a Markdown "bug" — most of these mismatches are a missing extension or a platform-only feature, not broken syntax. For code-fence-specific issues such as missing syntax highlighting or unsupported language identifiers, see the dedicated guide on Markdown code blocks.

Lint the Portable Subset

A Markdown linter cannot guarantee renderer compatibility, but it can remove avoidable ambiguity.

Useful rules include:

Use consistent heading styles
Add blank lines around lists and code blocks
Use fenced rather than indented code
Specify code fence languages
Avoid skipped heading levels
Use consistent list markers
Avoid ambiguous emphasis around punctuation
Keep line endings consistent
Validate links and images

For multi-target publishing, add a build test for each important renderer rather than relying only on syntax linting.

Converting Between Dialects with Pandoc

Pandoc can normalize documents from one dialect to another:

`bash pandoc \ --from=markdown \ --to=gfm \ article.md \ -o article-gfm.md `

Or convert GFM into Pandoc Markdown:

`bash pandoc \ --from=gfm \ --to=markdown \ README.md \ -o document.md `

This is useful, but conversion is not guaranteed to preserve every feature.

Potential losses include:

Platform-specific references
Callout styling
Complex tables
Embedded application objects
Custom attributes
Raw HTML behavior
Plugin syntax
Diagram rendering
Exact whitespace and formatting

Pandoc preserves document structure better than original source formatting. Treat conversion as a build step, not a reversible text formatter.

Recommended Strategy for Hugo Sites

For a Hugo technical blog, the most practical policy is:

Use CommonMark for core prose and structure.
Enable a small documented set of Goldmark extensions.
Use GFM-style tables and task lists where they improve readability.
Implement Mermaid through one consistent render hook or shortcode.
Handle math through one documented KaTeX or MathJax pipeline.
Use Hugo front matter only at the start of content files.
Prefer render hooks and shortcodes over raw HTML.
Keep source links as standard Markdown links where possible.
Test migrated or externally sourced documents through Hugo.
Document any syntax that will not render correctly on GitHub.

This approach accepts that Hugo content is not universally portable while keeping the portability boundary visible.

The worst approach is accidental dialect mixing: GitHub alerts, Obsidian embeds, Pandoc attributes, and Hugo shortcodes placed in the same document without a defined build pipeline.

Decision Table

Use case	Recommended syntax	Reason
Portable plain-text document	CommonMark	Smallest reliable baseline
GitHub README	GFM	Tables, tasks, and repository workflows
GitHub issue template	GFM plus GitHub features	Platform is the intended target
Hugo blog post	CommonMark plus configured Goldmark extensions	Controlled publishing pipeline
Academic paper	Pandoc Markdown	Citations, math, metadata, PDF output
Multi-format book	Pandoc Markdown	Structured conversion to many outputs
Obsidian vault	Obsidian Markdown	Backlinks, embeds, and knowledge workflows
GitHub and GitLab mirror	Conservative GFM	Strong shared feature set
Unknown renderer	CommonMark subset	Lowest compatibility risk

Conclusion

CommonMark, GitHub Flavored Markdown, and Pandoc Markdown are not competing versions of the same product. They solve different problems.

CommonMark provides a dependable parsing foundation. GFM adds practical features for software collaboration, while Pandoc Markdown turns Markdown into a rich source language for publishing and conversion.

The safest rule is simple: write the smallest dialect that satisfies the real destination. Use CommonMark when content must travel, GFM when GitHub-style collaboration is the target, and Pandoc Markdown when document structure and output formats matter more than universal rendering.

Markdown portability is not achieved by avoiding every extension. It is achieved by knowing which extensions are part of the source contract and testing them in every renderer that matters.

References

Data Gravity: The Real Cost of API-First AI

Rost — Fri, 07 Aug 2026 09:54:30 +0000

Every API call feels like a simple transaction - until enough of them accumulate that your fine-tuning data, evaluation harnesses, and tool schemas are all shaped around one vendor, and switching stops being a routing change.

That's data gravity: the same force that made pulling data out of AWS S3 expensive long before AI existed, now operating one layer up the LLM hosting stack. It doesn't require a bad contract or a malicious vendor. It's compounding integration debt - every fine-tuned checkpoint, cached embedding, and evaluation harness tuned to one provider's output format makes the next one cheaper to add and the whole pile more expensive to move.

The mechanism has four stages, and none of them announce themselves. Most teams don't decide to become dependent - they drift from Exploration into Integration, then Optimization, until Dependency feels less like a choice and more like the ground truth of their architecture. Recognizing which stage you're in, and what it costs to reverse, is the point of this article.

The Mechanism: Four Stages of Lock-In

graph LR
    A[Exploration<br/>swap a base URL] --> B[Integration<br/>workflows assume<br/>the API's shape]
    B --> C[Optimization<br/>fine-tunes, caches,<br/>vector stores]
    C --> D[Dependency<br/>product quality =<br/>vendor's model]
    style A fill:#e8f4fd
    style B fill:#cfe8fb
    style C fill:#a8d4f5
    style D fill:#6fb3ea

Exploration. You call an API, prototype, iterate. Switching cost is low - changing a base URL and a key covers most of it.

Integration. You build workflows around the API's shape. Error handling assumes its rate-limit headers. Retry logic matches its backoff curves. Your evaluation harness is tuned to its output format. Switching now means refactoring, not routing.

Optimization. You fine-tune. You cache. You build vector stores and custom pipelines that depend on that provider's embedding space, tokenization, or tool-calling schema. Your data is embedded in their ecosystem. Switching means rebuilding, not refactoring.

Dependency. Your product's performance depends on that provider's model quality. Downgrading to a self-hosted alternative means accepting lower capability. The trade-off stops being architectural and becomes product-level.

Each stage compounds the previous one. The transition from Exploration to Dependency rarely feels like a decision - it feels like progress, right up until the moment a vendor changes the terms.

Why It Matters Right Now

Three forces are making data gravity urgent instead of theoretical.

Open-weight models are closing the capability gap. Moonshot AI's Kimi K3, a 2.8-trillion-parameter sparse mixture-of-experts model released in July 2026, scored 57 on the Artificial Analysis Intelligence Index - third overall, comparable to Claude Opus 4.8 and GPT-5.5, and still behind Claude Fable 5 and GPT-5.6 Sol, but close enough that the gap is now a deliberate trade-off rather than a forced compromise. Qwen and DeepSeek ship under permissive licenses with native support across vLLM and SGLang. For coding and infrastructure tasks specifically, open-weight models routinely land within 5-15% of frontier API quality - close enough that the lock-in cost, not the capability gap, becomes the deciding factor.

Geopolitics is fragmenting data flows. In July 2026, security researchers found that Claude Code had shipped hidden detection code since version 2.1.91 (April 2, 2026) that checked a user's system timezone against Asia/Shanghai and Asia/Urumqi and scanned proxy hostnames against a list of Chinese corporate and AI-lab domains - including Alibaba, Baidu, ByteDance, and Moonshot AI - encoding the match invisibly into the tool's own system prompt. Anthropic called it an anti-distillation experiment; Alibaba responded by banning Claude Code for its employees effective July 10, 2026, and ordering Claude models deleted from company infrastructure. Whatever the intent, the episode is a preview of a world where cross-border AI data flows carry protocol-level risk, not just contractual risk. If your data and your tool's behavior live in someone else's runtime, you're subject to decisions you can't audit - which is the same conclusion LLM Self-Hosting and AI Sovereignty reaches from the policy and jurisdiction side rather than the switching-cost side covered here.

Memory economics are tightening. SK Hynix CEO Kwak Noh-jung told Reuters in July 2026 that 2027 will be the memory industry's worst-ever supply shortage, with customer demand expected to outstrip production capacity "even beyond 2030." SambaNova closed the first tranche of a $1B Series F at an $11B valuation the same month, explicitly to scale inference hardware manufacturing. The narrative that API costs fall indefinitely was already shaky; a multi-year hardware shortage makes owning your inference stack a strategic hedge rather than a hobbyist preference.

The Real Cost Isn't Tokens

The pricing comparison is the wrong frame. It's not "$0.01 per 1K input tokens vs. $0.002 self-hosted" - it's architectural dependency, and the clearest recent proof is OpenClaw's collapse.

OpenClaw grew to roughly 247,000 GitHub stars on the strength of running Claude through flat-rate Pro and Max subscriptions rather than metered API billing. On April 4, 2026, Anthropic revoked the ability to use those subscription OAuth tokens in third-party tools. Users who wanted to keep running OpenClaw with Claude had to switch to pay-as-you-go billing at 10 to 50 times the effective cost of their old plan. That is Dependency, stage four, made visible almost overnight: a huge community had optimized its entire workflow around one provider's specific pricing mechanism, and when that mechanism disappeared, the workflow's economics didn't degrade gracefully - they broke. The OpenClaw vs. Hermes usage data shows a meaningful share of that traffic migrating toward self-hosted and open-weight alternatives in the months after.

The same pattern shows up quietly inside individual companies. A team that builds a code-review agent against one provider's API accumulates fine-tuning data in that provider's format, an evaluation harness tuned to that provider's output shape, and tool-calling integrations built around that provider's schema. None of that is measured in tokens. It's measured in engineering weeks the day you try to leave - the same architectural-dependency problem that the LLM Architecture cluster covers at the routing, cost, and guardrail layer above hosting.

How to Score Your Lock-In

Count how many of these your team has accumulated for a given provider:

Dependency	Do you have this?
Fine-tuning datasets stored in a provider-specific format
Cached embeddings tied to a provider's embedding space
Evaluation harnesses tuned to a provider's output shape
Custom tool schemas built around a provider's tool-calling API
Team knowledge specific to a provider's failure modes and workarounds
Product features that assume a specific model's capability ceiling
Billing or usage patterns tied to a provider-specific plan (subscription vs. metered)

0-2 checked: Exploration - switching cost is still close to zero. 3-5: Integration - expect a real refactor. 6+: Optimization or Dependency - you are not choosing your AI provider anymore; you are renting your architecture from them. The count itself is the warning signal, and it costs nothing to calculate.

What Self-Hosting Actually Costs - and Doesn't

The economics are real but secondary to the lock-in question. Cost Optimization for LLM Systems works through the hardware break-even math in detail - at roughly an hour or more of daily local usage, a consumer GPU like an RTX 4090 typically pays for itself against equivalent API spend within 4-8 months. That analysis is the right place for the $/token comparison; the point worth repeating here is that the break-even calculation only matters once you've decided portability is worth optimizing for. Teams deep in the Dependency stage often find the migration cost dwarfs any hardware savings, which is exactly the trap this article is about.

The Antidote: Portability as a Strategy

The goal isn't to avoid APIs. It's to keep your data layer portable long enough to make deliberate choices instead of drifting into a stage you didn't choose.

Start local, go remote deliberately. Prototype with self-hosted models - llama.cpp, GGUF quantization, or a full comparison of local hosting tools to pick a stack. When a task genuinely needs frontier capability, use the API for that task specifically - but keep the data layer decoupled from which model answered it.

Prefer open-weight to closed-API when the quality gap is close. When a model ships as open weights - Kimi K3, Qwen, Gemma, DeepSeek - you can run it, fine-tune it, quantize it, and own the relationship end to end. The capability gap is a known, shrinking, task-dependent trade-off. The lock-in gap is a slow-moving trap that doesn't announce its size until you try to leave.

Build abstraction where it actually matters. Not "wrap everything behind an interface" - that's a rule that delays the problem without solving it. Build the abstraction around data formats, evaluation logic, and tool schemas specifically: fine-tuning datasets in framework-agnostic formats (JSONL, parquet), evaluation harnesses that score model output rather than a specific API's response shape, and tool-calling logic that translates to and from provider-specific schemas rather than being written against one.

Route deliberately instead of committing to one provider. Model routing strategies - capability-based, cost-aware, latency-aware - let you send routine traffic to a local model and edge cases to a frontier API, which keeps you inside the Integration stage indefinitely instead of drifting into Optimization around a single vendor.

Quantify your lock-in on a schedule. Re-run the scoring table above quarterly per provider. When the count climbs, that's data gravity doing its work whether or not anyone made an explicit decision to let it.

What This Looks Like in Practice

A practical stack that resists data gravity by design:

Inference: llama.cpp for local, single-machine serving; vLLM or SGLang for production-grade self-hosted throughput. All three expose OpenAI-compatible APIs, so application code doesn't need to know which one is behind it.
Fine-tuning: datasets stored in standard formats - JSONL, parquet - never a provider's proprietary fine-tuning-job format.
Evaluation: framework-agnostic harnesses that score outputs, not API response envelopes, so the same eval suite runs whether the model is local or remote.
Tool calling: a provider-agnostic JSON schema translated to and from each vendor's tool-calling format, rather than application logic written directly against one vendor's shape.
Vector stores: local-first options such as Qdrant, Milvus, or Chroma, with embeddings computed through a portable library rather than tied to one provider's embedding endpoint - see chunking strategies in RAG for how this fits the retrieval layer.

This isn't a self-hosting manifesto. Plenty of workloads belong on a frontier API, permanently. It's a recognition that the engineers who can measure and manage data gravity - rather than discover it the day a vendor changes its pricing - end up with more options, not fewer.

The Bottom Line

Data gravity is why open-weight capability matters more than a single benchmark score. A model that runs locally at 85-95% of a frontier model's quality is often the better architectural choice, because you keep the data relationship. The frontier race between GPT-5.6 Sol, Fable 5, Kimi K3, and Qwen is genuinely interesting, but the infrastructure layer underneath it - who holds the fine-tuning data, whose schema the tools speak, whose pricing model the workflow assumes - is what actually determines which teams have options in three years and which ones are renting their architecture from someone else.

Score your lock-in before a vendor's pricing page forces the question for you.

Sources

Claude Code Subagents: Setup, Config, and When to Use Them

Rost — Wed, 05 Aug 2026 09:07:21 +0000

Most Claude Code sessions get slow and cluttered for the same reason: every exploratory grep, every log dump, and every "let me check one more file" stays in the main conversation forever.

Subagents exist to fix exactly that problem. They are one of the agent primitives built into Claude Code for handling noisy, parallelizable work — a way to push the mess into an isolated window and bring back only the summary that matters.

A subagent is not a smarter Claude, and it is not the same thing as a Skill. It is a separate reasoning agent with its own context window, its own tool allowlist, and no memory of your current conversation unless you explicitly fork it. Understanding that distinction is the difference between a subagent setup that quietly saves you context budget and one that just adds latency for no benefit.

Subagents vs Skills vs MCP

Claude Code gives you three extension points that solve different problems, and they get conflated constantly because all three can technically "help with a task."

Layer	What it is	When to reach for it
Skill	Instructions loaded into the main agent's context on demand	Reusable procedures, checklists, playbooks — see Claude Skills for developers
Subagent	A separate agent with its own context window, dispatched for delegated work	Noisy exploration, parallelizable research, anything you want kept out of the main session
MCP server	An external tool/data connector exposed over a protocol	Reaching systems outside the local session — APIs, databases, remote services

A useful rule of thumb: a hook enforces a hard constraint deterministically, a Skill gives the main agent a capability inline, and a subagent is for work you want to delegate and keep out of the main context entirely. If a Skill's job is to orchestrate a tool that doesn't exist yet, that is usually a sign you need an MCP server, not a subagent. Claude Code isn't alone in this shape — OpenCode's ecosystem has a comparable idea in its specialised agents, which split planning, research, and review across dedicated roles in a similar way.

What a subagent actually is

Three properties define a Claude Code subagent, and all three matter for how you use it:

Isolated context. A subagent starts with a fresh window. It does not see your conversation history unless you explicitly fork it, which keeps its output from being polluted by whatever you discussed three turns ago.
A restricted tool allowlist. Subagents can only use a subset of what the parent session already has — they cannot grant themselves new capabilities, and a well-designed subagent should get only the tools its job requires (read-only tools for a research agent, for example).
No cross-subagent visibility. Subagents cannot see each other's work in progress. If task B genuinely needs task A's output, that is a sequential dependency, not something you can parallelize across two subagents.

The trigger for reaching for one is not "this task is hard." It is "this task is noisy" — the kind of work that generates a lot of intermediate output (dozens of file reads, a long log, an exploratory grep across the whole repo) where none of that intermediate material needs to survive into your next conversation turn.

When to use a subagent (and when not to)

Good fits: codebase exploration before a big change, automated test runs where you only care about pass/fail and failure summaries, security or style reviews, and any multi-step research task whose raw output would otherwise flood your main session.

Bad fits: two-second lookups ("what does this function return"), anything requiring tight back-and-forth refinement, and dependent tasks you're tempted to "parallelize" even though the second one needs the first one's answer. Using a subagent for a trivial lookup just adds the overhead of spinning up a fresh context window for no real isolation benefit.

Measuring the payoff: context and cost math

The pitch for subagents is abstract until you put numbers on a real task. Take a common one: grep a ~500-file service for every place a deprecated config key is still read, then report the exact file:line matches.

Approach	Main-session context consumed	What survives into your next turn
Direct exploration, no subagent	~35-45K tokens — every grep hit, every file you opened to double-check, every dead end	All of it, including the wrong turns
Delegated to an Explore subagent	~1.5-3K tokens — one summarized report	Only the findings that mattered

That's roughly a 15-20x reduction in what your main session has to carry for that step, which is the actual mechanism behind "subagents keep sessions faster" — it is not magic, it is context that never gets loaded in the first place.

The cost side compounds the same way. Using the pricing from the Claude Code pricing breakdown, running that same exploration pass on Opus (\$5/MTok input, \$25/MTok output) costs roughly \$0.20-0.25 for the ~40K input tokens alone. Routing it to Haiku (\$1/MTok input, \$5/MTok output) drops that to \$0.04-0.05 — and the main session's Opus budget is never touched by the exploration tokens at all, since it only ever sees the ~2K-token summary.

Defining a custom subagent

Custom subagents live as Markdown files with YAML frontmatter, either project-scoped in .claude/agents/ (committed to the repo, shared by the whole team) or user-scoped in ~/.claude/agents/ (personal tools you bring to every project).

---
name: code-reviewer
description: >
  Reviews staged changes for bugs, security issues, and style violations
  before commit. Use when the user asks to review, audit, or check
  changes prior to committing or opening a PR.
tools: Read, Grep, Glob
model: sonnet
skills:
  - security-checklist
---
You are a careful code reviewer. Read the staged diff, flag concrete
issues with file:line references, and end with a short pass/fail summary.
Do not modify any files.

The description field is the most important line in the file. It is what the parent session's routing logic reads to decide whether this subagent fits the current task. Write it like a job posting — name the trigger condition explicitly, not a vague "helps with code." Vague descriptions get skipped or misapplied by automatic dispatch.

The tools field is your isolation boundary. Give a research subagent Read, Grep, and Glob and nothing else; giving it every tool available defeats the entire point of running it in a restricted sandbox. The optional skills field preloads the full content of named Skills into the subagent's startup context — useful when a subagent needs domain knowledge without spending a turn discovering and loading it mid-task.

Model routing: cheap models for grunt work

Subagents are also where cost control gets real. Route file discovery, log scanning, and other cheap-to-verify work to Haiku, and reserve Sonnet or Opus for the reasoning-heavy steps — architecture decisions, ambiguous debugging, anything where getting it wrong is expensive. Haiku is roughly 15x cheaper per token than Opus, and on the kind of noisy exploration subagents are built for, that gap adds up fast across a real working session.

The Explore, Plan, Execute pattern

For complex, multi-step work, the pattern that holds up in practice is Explore, Plan, Execute — using cheap subagents for the parts that generate noise, and keeping the human review gate in the one place it actually matters.

sequenceDiagram
    participant You
    participant Main as Main session
    participant Explore as Explore subagent (Haiku)
    participant Execute as Execute agent (Sonnet/Opus)

    You->>Main: Describe the task
    Main->>Explore: Delegate codebase exploration
    Explore-->>Main: Return summarized findings
    Main->>Main: Enter Plan mode, propose approach
    Main->>You: Show plan for review
    You->>Main: Approve or adjust
    Main->>Execute: Hand off approved plan
    Execute-->>Main: Apply changes, run tests
    Main-->>You: Report results

The key detail people get backwards is where the review gate belongs. Exploration is cheap, so let a subagent read freely without asking permission first. Planning is analytical, so let the agent design the approach on its own. But before any agent modifies files, you want to see the plan and approve it — that is what Claude Code's plan mode (permissionMode: plan) is for, and it is the same principle discussed in the broader vibe coding best practices around reviewing every diff before it lands.

Common mistakes

A handful of mistakes show up repeatedly once teams start writing custom subagents:

Vague descriptions. "Helps with code" will never route correctly. Name the exact trigger condition.
Over-broad tool access. Giving a read-only research subagent write and bash access removes the isolation guarantee that made it worth creating in the first place.
Parallelizing dependent tasks. If task B needs task A's finished output, run them sequentially — subagents cannot coordinate mid-task the way a shared orchestrator can. For workflows that genuinely need agents talking to each other mid-task, that's a different shape of problem; see multi-agent orchestration patterns if you're building a production system rather than a single-repo workflow.
Using a subagent for trivial work. "Format this JSON" or "run this one command" doesn't need a fresh context window; just do it directly.

Worked example: a code-review subagent end to end

Say you want every non-trivial commit reviewed before it lands. Drop the code-reviewer definition shown earlier into .claude/agents/code-reviewer.md, commit it so the whole team shares the same reviewer, and invoke it with a natural request like "review my staged changes before I commit." Claude Code matches your request against the subagent's description, spins it up with only Read, Grep, and Glob access, and it comes back with file:line-referenced findings and a pass/fail summary — none of the file-by-file noise from getting there ever touches your main session.

What that looks like in the main transcript, annotated:

You:  review my staged changes before I commit

Main: [dispatches code-reviewer subagent — 6 files read, 1 grep pass,
       zero of it shown here]

Main: code-reviewer findings:
      - auth/session.go:142 — token refresh path doesn't handle expired
        refresh token; falls through to nil dereference
      - auth/session.go:203 — style: error wrapped without %w
      PASS/FAIL: FAIL (1 blocking issue)

Six file reads and a grep pass happened, and your main session paid for exactly four lines of it. That gap — everything the subagent did versus the three-line summary you actually see — is the entire value proposition in one transcript.

If your team also uses Spec-Driven Development scaffolds, a review subagent slots naturally into the validation step; see GitHub Spec Kit vs Kiro vs Claude Code SDD Workflows for how that review gate compares across portable and IDE-integrated SDD setups.

Is it worth setting up custom subagents?

Not on day one. The built-in general-purpose subagent already covers most exploration and research delegation without you writing a single YAML file, and a single Explore-Plan-Execute pass is enough for most day-to-day work. Write a custom .claude/agents/*.md file only once you've delegated the same task by hand three times — a code reviewer, a test-runner triager, a docs-lookup agent for one specific internal library. Teams that write five subagents in their first week usually end up with five stale description fields nobody updates when the actual trigger condition drifts, which quietly breaks automatic routing months later. Start with zero custom subagents, add one at a time, and only when repetition — not theoretical usefulness — demands it.

Known limitations

A few rough edges are worth knowing before you build around subagents:

No recursive delegation. A subagent cannot spawn its own subagents. If a task genuinely needs a second layer of delegation, that is a sign you want a different orchestration shape — see multi-agent orchestration patterns for what that looks like outside a single Claude Code session.
No memory across invocations. Every dispatch starts from zero, even if you called the same subagent five minutes ago on a related task. There is no built-in mechanism for a subagent to remember its last run.
Isolation is a tool allowlist, not a sandbox. A subagent with Bash access can still touch the filesystem and network like any other tool call. Restricting tools reduces blast radius; it does not create a hard security boundary.

Troubleshooting

Subagent never triggers. The description is almost always the problem. Rewrite it around the specific trigger condition instead of a general capability statement, and double-check the file lives in .claude/agents/ (project) or ~/.claude/agents/ (personal) with the right extension.

Subagent burns too much context anyway. Check the tools allowlist — an overly broad toolset invites overly broad exploration. Also check whether the task should have been split into two subagents instead of one doing everything.

A listed skill doesn't load inside the subagent. Claude Code skips a missing or disabled skill named in the skills field rather than failing the run, and logs a line to that effect in the debug output (/debug from the main session, then reproduce the dispatch) — something like skill "security-checklist" not found, skipping. Run /doctor afterward to confirm the rest of your setup is healthy.

Results feel inconsistent between runs. This is often a model-routing issue, not a subagent-design issue — reasoning-heavy work assigned to a cheap model will vary more. Move it to Sonnet or Opus and keep Haiku for the deterministic, low-ambiguity steps.

Subagents are one piece of a much larger toolbox; if you're comparing Claude Code against the rest of the AI developer tools ecosystem before committing to this workflow, that overview is a good next stop.

Useful links

Ollama to vLLM: When to Migrate Your Local LLM Server

Rost — Sun, 02 Aug 2026 13:59:56 +0000

Ollama is one of the easiest ways to run a local language model, but convenience can conceal the moment when a local experiment becomes a shared inference service that needs better scheduling and observability.

That is where vLLM becomes relevant. Migrating from Ollama to vLLM is not an automatic upgrade, however. It is a trade: you exchange some of Ollama's simplicity for greater control over batching, memory management, concurrency, distributed inference, and production operations.

This guide covers the practical signals that indicate migration is warranted, the risks of moving too early, and a staged approach that keeps both servers running side by side during validation. The goal is to help you decide based on measurements rather than feature lists. For the wider landscape of local, self-hosted, and cloud options beyond just these two runtimes, see LLM Hosting in 2026: Local, Self-Hosted & Cloud Infrastructure Compared.

Ollama and vLLM Solve Different Problems

Ollama is primarily optimized for convenient model consumption. It gives developers a concise command-line interface, a local API, a model library, Modelfiles, and straightforward support for common desktop and workstation configurations.

vLLM is an inference engine and serving platform. Its central concerns are high-throughput request scheduling, efficient KV cache management, continuous batching, model parallelism, and compatibility with applications built for OpenAI-style APIs.

The distinction matters because the two servers can look similar from the outside. Both can expose a chat API, stream tokens, run quantized models, and serve local applications. Their operating models become visibly different only when the server is placed under sustained or concurrent load.

A useful summary:

Requirement	Ollama	vLLM
Fast local setup	Excellent	More involved
Curated model downloads	Excellent	Usually Hugging Face based
GGUF workflow	First-class	Supported, not main strength
Single-user chat	Excellent	Often unnecessary
Concurrent API traffic	Limited but configurable	Core use case
Continuous batching	Not the primary model	Core feature
Prefix cache reuse	Limited operational control	Built-in optimization
Multi-GPU model serving	Limited compared with vLLM	Tensor and pipeline parallelism
Production metrics	Basic response timing data	Prometheus metrics endpoint
Deployment tuning	Minimal	Extensive

The question is not which server is universally better. It is whether your workload still matches the operating model that makes Ollama attractive. If you want the fuller picture across more than these two runtimes, our comparison of Ollama, vLLM, LocalAI, Jan, LM Studio and other local LLM tools covers the broader field.

Signs You Have Outgrown Ollama

A slow response does not, by itself, justify a migration. Generation speed is often constrained by model size, quantization, memory bandwidth, prompt length, or GPU capability rather than the serving engine, and the stronger migration signals only appear once workload shape itself starts to matter.

Multiple Users Cause Unstable Latency

A local LLM server can feel fast during an isolated test and then degrade sharply when several clients connect. Requests begin waiting behind long generations, time to first token becomes inconsistent, and a single large prompt can affect everyone sharing the model.

Ollama can process parallel requests, and OLLAMA_NUM_PARALLEL controls how many requests a loaded model may handle concurrently — see how Ollama handles parallel requests for the queuing and memory mechanics behind that setting. That parallelism is not free: memory requirements grow with both the configured parallel request count and context length.

This is often the first practical warning. A configuration that works for one 8K conversation may become impossible when four clients each reserve a much larger context.

vLLM is designed to combine work from active requests through continuous batching. Instead of treating each request as an isolated inference job, it continuously updates the batch as sequences arrive, generate tokens, and finish — a scheduling model that generally becomes more valuable as concurrency increases.

GPU Utilization Is Low While Requests Are Queued

A queue does not necessarily mean that the GPU is fully used. In a simple serving arrangement, work may be serialized even though additional requests could have contributed useful computation to the current decode step.

vLLM's scheduler is designed to keep more useful work in flight. PagedAttention manages KV cache memory in blocks, while continuous batching allows active sequences to enter and leave the execution batch dynamically.

The result is not guaranteed to be lower latency for every individual request. Under load, however, it can produce substantially better aggregate throughput and more predictable resource utilization.

Long Prompts Dominate Time to First Token

Long-context coding assistants, RAG pipelines, and agent sessions can repeatedly send large system prompts or shared document prefixes. Processing those input tokens is the prefill stage, and it can dominate time to first token.

vLLM supports chunked prefill and automatic prefix caching. Prefix caching allows later requests to reuse KV cache blocks when their initial token sequence matches an already processed prefix.

This is particularly useful when requests share:

A long system prompt
The same tool definitions
A stable repository summary
Repeated few-shot examples
A common RAG document prefix
A shared conversation history

Prefix caching does not make output generation faster. It reduces repeated prompt computation, so its benefit depends on whether requests actually contain identical reusable prefixes.

You Need More Than One GPU

A model that does not fit on one GPU is a strong reason to consider vLLM. It supports tensor parallelism across GPUs and pipeline parallelism across multiple nodes or devices.

This does not make multi-GPU inference effortless. GPU interconnect bandwidth, PCIe topology, model architecture, container shared memory, and communication overhead still affect performance.

Nevertheless, vLLM provides a deliberate path for distributed inference. Ollama is usually a better match for a single desktop or workstation where the chosen model already fits comfortably.

You Need Production-Level Observability

Ollama API responses expose useful timing fields such as model load duration, prompt evaluation duration, generated token count, and generation duration. These values are enough for local benchmarking and application-level logging.

vLLM exposes Prometheus-compatible metrics through its /metrics endpoint. That makes it easier to track request volume, queueing, time to first token, inter-token latency, cache usage, preemptions, throughput, and request outcomes over time.

Once users depend on the service, observability stops being optional. Without queue, cache, and latency metrics, it is difficult to distinguish an undersized GPU from an oversized context limit, poor scheduling, cold model loading, or simply too many simultaneous requests.

Where vLLM Actually Wins

vLLM's most important advantage is not that it can produce one response faster than Ollama on every machine. The meaningful advantage is that it gives the operator more mechanisms for using expensive accelerator memory and compute efficiently across many requests.

Continuous Batching

Traditional static batching works best when requests have similar input and output lengths. Interactive LLM traffic rarely behaves that way: one user asks for a short classification, another submits a 20K-token prompt, and a third generates several thousand tokens of code.

Continuous batching changes the active batch as requests progress. Completed sequences leave, new sequences enter, and the engine attempts to avoid wasting batch capacity on requests that have already finished.

This improves throughput when traffic is concurrent and uneven. It provides little benefit when a single user sends one request at a time.

Paged KV Cache Management

During generation, the server stores attention keys and values for previously processed tokens. This KV cache can consume a large amount of GPU memory, especially with long contexts and multiple active sequences.

vLLM manages this cache in blocks instead of requiring each sequence to reserve one large contiguous allocation. The approach reduces memory fragmentation and allows available cache capacity to be used more flexibly.

The practical value is higher concurrency within the same memory budget. It does not remove the underlying cost of long context, but it reduces avoidable waste around that cost.

Prefix Caching

Many production requests share a substantial beginning. Tool-enabled agents may send identical function schemas, support bots may use the same policy documents, and coding assistants may repeatedly include the same repository instructions.

Automatic prefix caching can reuse the computed cache for matching prefixes. It is especially useful when a stable, large prefix is followed by a relatively small request-specific suffix.

It is less useful when templates, timestamps, document ordering, or dynamically generated metadata change near the beginning of every prompt. Small differences in tokenization can prevent the prefix from matching.

Parallel and Distributed Inference

vLLM supports several forms of parallelism, including tensor, pipeline, data, expert, and context parallelism. Not every deployment needs these modes, but their availability matters when a service grows beyond one GPU.

For a workstation with two suitable GPUs, tensor parallelism may allow a larger model to run across both devices. For a replicated service, data parallelism can create multiple engine replicas for additional throughput.

These features introduce operational complexity. They should be adopted because measurements demonstrate a capacity problem, not because distributed inference appears more sophisticated.

Broader Production Controls

vLLM exposes controls for GPU memory utilization, maximum model length, maximum active sequences, quantization, cache data types, speculative decoding, tool calling, structured output, model aliases, authentication keys, and distributed execution.

That flexibility makes the server easier to tune for a particular workload, but it also creates more opportunities for an invalid or inefficient configuration. Migrating to vLLM means taking responsibility for those decisions.

Where Ollama Still Wins

A migration guide should not treat Ollama as an inferior preliminary tool. For many local deployments, it remains the better server.

Personal Workstations

For one developer using a chat interface, code assistant, or occasional local API, the operational advantages of vLLM may never compensate for its additional setup.

Ollama installs quickly, downloads models through a simple registry, and hides many model-specific details. It is well suited to experimentation and private desktop use.

GGUF Model Collections

Ollama has a natural workflow around GGUF models and Modelfiles. Existing users may have curated quantizations, adapters, templates, system prompts, and parameters that work reliably with their hardware.

vLLM supports GGUF, but its strongest path is generally through supported Hugging Face model repositories and quantization formats such as AWQ, GPTQ, BitsAndBytes, FP8, or vendor-specific formats. Moving an existing GGUF deployment to vLLM without evaluating a more native checkpoint format can preserve the inconvenience of migration while missing some of the performance advantages.

Mixed CPU and GPU Offloading

Desktop inference sometimes relies on partial GPU offloading because the entire model does not fit in VRAM. This can be practical for occasional use, particularly when latency is not critical.

vLLM is generally most compelling when the model and required KV cache capacity can be served effectively by the available accelerator configuration. A workload that depends heavily on system RAM and CPU offloading may be better suited to Ollama or llama.cpp.

Rapid Model Switching

Ollama makes it easy to pull, run, stop, and switch among many local models. That is useful for evaluation, writing, coding, embeddings, vision, and ad hoc experimentation.

A vLLM deployment is more commonly built around a deliberately selected model that remains loaded as a service. Multi-model deployment is possible, but it requires more explicit resource planning.

Minimal Administration

Ollama is intentionally opinionated. That can be a limitation under load, but it is an advantage when nobody wants to maintain an inference platform, and if the local server has one user, acceptable latency, and no meaningful queue, migration is likely to create work rather than remove it.

Do Not Migrate Based on Tokens per Second Alone

Single-request token generation speed is an incomplete benchmark. Two servers may produce similar decode throughput for one sequence while behaving very differently with eight concurrent clients.

A useful evaluation should measure at least:

Time to first token
Inter-token latency
End-to-end request latency
Prompt processing throughput
Output token throughput
Requests completed per minute
Queue wait time
GPU memory consumption
GPU utilization
Failure and timeout rate

Run the same model family, precision, context length, prompt set, output limit, and concurrency level on both servers. Otherwise, the test is more likely to compare model packaging and configuration than serving engines.

The most useful comparison is a small load test that represents your real traffic. For a shared coding assistant, that might include long system prompts, repeated prefixes, streaming responses, and two to eight simultaneous sessions.

Plan the Model Migration First

Ollama model names do not automatically map to equivalent vLLM model identifiers. An Ollama package may contain a particular GGUF quantization, prompt template, stop-token configuration, and default parameters.

Before changing the server, identify:

The original model family and version
Whether it is a base or instruction-tuned model
The current quantization and effective precision
The prompt or chat template
The configured context length
Stop tokens and generation defaults
Tool-calling or structured-output requirements
Any LoRA adapters or custom system prompts

Then choose a vLLM-supported checkpoint that matches the intended behavior. Do not assume that an AWQ or FP8 checkpoint will behave identically to the GGUF build previously used in Ollama — the model migration is often more significant than the API migration.

Check VRAM Before Starting vLLM

A model fitting into GPU memory does not mean that it can serve the required workload. VRAM must cover more than model weights.

The practical memory budget includes:

model weights
+ KV cache
+ CUDA graphs and runtime allocations
+ temporary workspace
+ multimodal processor caches, if used
+ safety margin

Long contexts and concurrent sequences primarily expand the KV cache requirement. Increasing the maximum context length therefore reduces the number of simultaneous requests that can fit, even if most requests never use the full limit.

Start with a realistic --max-model-len rather than the largest value advertised by the model, and avoid setting GPU memory utilization so aggressively that minor workload variation causes out-of-memory failures. A stable service with slightly less theoretical capacity is more useful than one that fails at its first traffic spike.

A Minimal vLLM Docker Compose Deployment

The following example starts an OpenAI-compatible vLLM server on port 8000:

services:
  vllm:
    image: vllm/vllm-openai:latest
    container_name: vllm
    restart: unless-stopped
    ports:
      - "8000:8000"
    ipc: host
    gpus: all
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
    environment:
      HF_TOKEN: ${HF_TOKEN:-}
    command:
      - --model
      - Qwen/Qwen3-8B
      - --served-model-name
      - local-model
      - --max-model-len
      - "16384"
      - --gpu-memory-utilization
      - "0.90"
      - --api-key
      - ${VLLM_API_KEY:-change-me}

Create an environment file:

cat > .env <<'EOF'
HF_TOKEN=
VLLM_API_KEY=replace-with-a-long-random-value
EOF

Start the server:

docker compose up -d

Check the logs:

docker compose logs -f vllm

Test the models endpoint:

curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer replace-with-a-long-random-value"

Send a chat request:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer replace-with-a-long-random-value" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [
      {
        "role": "user",
        "content": "Explain continuous batching in two paragraphs."
      }
    ],
    "temperature": 0.2,
    "max_tokens": 300,
    "stream": false
  }'

For a maintained deployment, pin the image to a tested vLLM release instead of leaving it on latest. Review release notes before upgrading because command-line options, model implementations, metrics, and engine behavior can evolve. This Compose file is intentionally minimal; for the fuller setup guide — OpenAI API compatibility, PagedAttention tuning, and a deeper vLLM-vs-Ollama comparison — see the vLLM Quickstart.

OpenAI API Compatibility Is Not Complete Interchangeability

Both Ollama and vLLM provide OpenAI-compatible endpoints, which can make the application migration relatively small. In many clients, changing the base URL, API key, and model name is enough to establish a connection.

For example:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="replace-with-a-long-random-value",
)

response = client.chat.completions.create(
    model="local-model",
    messages=[
        {
            "role": "user",
            "content": "What should I monitor on an LLM server?",
        }
    ],
    temperature=0.2,
)

print(response.choices[0].message.content)

Compatibility should still be tested at the feature level. Examine:

Streaming event behavior
Supported request parameters
Chat template selection
Tool-call parsing
Reasoning output handling
JSON or schema-constrained output
Embeddings endpoints
Multimodal inputs
Token usage reporting
Error response formats
Model name discovery
Context-length enforcement

A client that only sends ordinary chat completions will usually be easier to migrate than an agent framework that depends on a particular tool-call parser or nonstandard extension.

Chat Templates Are a Common Migration Failure

Instruction-tuned models expect conversations to be serialized using a specific chat template. The template inserts role markers, separators, control tokens, and generation prompts in the format used during training.

Ollama packages much of this behavior inside its model definition. With vLLM, the template is normally obtained from the model tokenizer configuration, although an operator can provide one explicitly.

A server may start successfully even when the selected template is wrong. The symptoms appear in model behavior:

The model repeats role labels
Responses contain special tokens
System instructions are ignored
Tool calls are malformed
The model continues the user message
Output quality is much worse than expected

Before blaming the inference engine, compare the fully rendered prompt used by each deployment.

Use a Staged Migration

Replacing a working local server in one step creates unnecessary risk. Ollama and vLLM can run side by side on different ports while you validate the new deployment.

Stage 1: Reproduce One Model

Choose the model responsible for most API traffic and match its instruction tuning, context requirement, generation parameters, and chat behavior as closely as possible. Do not begin by moving every experimental model.

Stage 2: Validate API Behavior

Run existing integration tests against the vLLM endpoint, including streaming, cancellation, timeouts, tool calls, malformed requests, context overflow, and concurrent access. Record behavioral differences rather than hiding them behind client retries.

Stage 3: Establish a Baseline

Measure one-request performance first. This confirms that the model is loaded correctly and provides a reference for later tests.

Record prompt tokens per second, output tokens per second, time to first token, total latency, and GPU memory usage.

Stage 4: Add Realistic Concurrency

Test the number of simultaneous requests expected in normal operation and during a plausible peak, using representative prompt and output lengths rather than identical synthetic requests. Watch queueing, cache use, preemptions, time to first token, and tail latency.

Stage 5: Move One Client

Route a noncritical application or a small percentage of traffic to vLLM. Keep Ollama available as a fallback until the new server has operated reliably under real use.

Stage 6: Tune From Measurements

Adjust model length, memory utilization, maximum active sequences, prefix caching, parallelism, and quantization only after identifying a measured constraint. Changing several parameters at once makes performance regressions difficult to explain.

A Practical Migration Checklist

Before switching clients, verify the following:

[ ] The target model is supported by vLLM
[ ] The selected checkpoint and quantization fit in VRAM
[ ] Enough VRAM remains for the required KV cache
[ ] The maximum context length reflects real usage
[ ] The correct chat template is available
[ ] Stop tokens and generation defaults are tested
[ ] Streaming works with existing clients
[ ] Tool calls and structured output are validated
[ ] The public model alias remains stable
[ ] Authentication is enabled
[ ] The server is not exposed directly to the internet
[ ] Prometheus metrics are collected
[ ] GPU metrics are collected separately
[ ] Load tests include realistic concurrency
[ ] Timeouts and cancellations are handled
[ ] A rollback path to Ollama exists

This list is deliberately operational. Installing vLLM is usually easier than proving that it behaves correctly for an existing application.

Security and Network Exposure

Neither a local Ollama endpoint nor a vLLM endpoint should be casually exposed to the public internet. An unauthenticated inference server can consume expensive GPU capacity, reveal model behavior, and become a route for denial-of-service attacks through very long prompts or outputs.

vLLM can require an API key for its OpenAI-compatible endpoints, but an API key is not a complete security boundary. For shared or remote access, place the service behind a reverse proxy or API gateway that provides TLS, network restrictions, request-size limits, rate limits, access logging, and appropriate authentication — the same pattern covered in Ollama behind a reverse proxy with Caddy or Nginx applies just as well in front of vLLM.

Also consider model-specific risks. Multimodal URL loading, custom model code, remote files, and unrestricted tool execution can expand the attack surface beyond ordinary text generation.

When Not to Migrate

Stay with Ollama when:

One or two users access the server
Requests are mostly sequential
The model already delivers acceptable latency
Easy GGUF management is important
CPU or partial GPU offloading is required
Models are changed frequently
Nobody wants to operate additional infrastructure
There is no measured concurrency or throughput problem

A move to vLLM should solve a concrete limitation. "Production" is not a magic threshold that invalidates Ollama, especially for an internal service with modest traffic.

Conversely, do not preserve Ollama merely because it was easier to install. If users regularly wait in a queue, repeated prefixes consume significant prefill time, or a larger model must be distributed across GPUs, the simpler server may have become the more expensive choice operationally.

Keep Ollama for Development and Add vLLM for Shared Serving

The most practical architecture is often not a complete replacement. Developers can keep Ollama running in Docker Compose on their workstations for model exploration, GGUF testing, and private interactive use while a shared vLLM instance serves a stable model to applications and teams. That split also matters for AI sovereignty — keeping both runtimes self-hosted means prompts, weights, and inference logs stay under your control regardless of which server handles a given request.

This separates two different workflows:

Ollama:
experimentation -> model switching -> personal tools -> local chat

vLLM:
selected model -> shared endpoint -> concurrent traffic -> monitoring

The arrangement also lowers migration risk. Models can be tested locally before a suitable checkpoint is promoted to the shared vLLM deployment.

Migration Decision Flow

The following diagram summarizes the key decision points:

flowchart TD
    A[Ollama serving LLM] --> B{Multiple users<br>with unstable latency?}
    B -->|No| C[Stay with Ollama]
    B -->|Yes| D{Long shared<br>prefixes?}
    D -->|Yes| E[Strong vLLM signal]
    D -->|No| F{Need multi-GPU<br>or observability?}
    F -->|Yes| E
    F -->|No| G{Measured concurrency<br>problem?}
    G -->|No| C
    G -->|Yes| E
    E --> H[Plan staged migration]
    H --> I[Validate side by side]
    I --> J[Switch clients gradually]

Conclusion

Ollama is difficult to beat as a local model runner. It removes enough packaging and configuration work that developers can concentrate on the model and application rather than the inference stack.

vLLM becomes the stronger choice when the server itself is the problem to be engineered. Concurrent traffic, queueing, repeated long prefixes, multi-GPU models, capacity planning, and production observability are the migration signals that matter.

Do not migrate because vLLM has a longer feature list. Migrate when measurements show that Ollama's simpler operating model no longer matches the workload. Until that point, simplicity is not a technical weakness; it is an optimization.

Keeping Specs, Tests, And Code In Sync In AI Development

Rost — Sat, 01 Aug 2026 10:43:20 +0000

AI coding agents ship features fast, but specs, tests, and code quietly drift apart. This guide covers a traceability model, spec-to-test and spec-to-code mapping, and the CI checks that catch drift before a merge.

A spec that nobody re-checks against the running system is worse than no spec at all, because it creates false confidence. Reviewers trust the document instead of the diff, and an AI agent asked to "follow the existing pattern" will happily follow whatever the code actually does, even when that contradicts the requirement it was supposed to satisfy.

The fix is not more documentation. It is a small, enforceable link between four things that already exist in most repositories: the requirement, the design decision behind it, the tests that prove it, and the commits or pull requests that changed it.

Once that link exists as data rather than as a shared understanding, you can query it. You can ask which requirements have no test coverage, which tests no longer map to any requirement, and which files changed in a pull request without a matching requirement ID. That query is the actual deliverable of this article, and the rest of the post walks through how to build it with tools you likely already run.

The Drift Problem: Why Specs, Tests, And Code Fall Out Of Sync

Drift shows up in four recognisable shapes, and AI-assisted teams tend to hit all four faster than teams that write every line by hand.

Spec changes, code does not. A requirement gets clarified in a follow-up conversation or a comment thread, but nobody regenerates or edits the implementation to match.
Code changes, spec does not. An agent or a developer fixes a bug or refactors a module, and the spec keeps describing the old behavior as if it were still current.
Tests cover implementation, not intent. Unit tests assert what the code currently does, which is circular: they pass by construction even when the code satisfies the wrong requirement.
Pull requests do not reference requirements. Reviewers approve a diff on the strength of "looks reasonable" because there is no explicit claim to check it against.

Recent process research on AI development frameworks identifies specification drift as a recurring risk precisely because agents regenerate code quickly and repeatedly, and each regeneration is a fresh opportunity for spec and implementation to diverge a little further. The Spec-Driven Development vs Vibe Coding debate is really an argument about this same failure mode: a spec that nobody enforces degenerates into the same drift you get without one, just with extra ceremony.

Modern spec-kit-style workflows increasingly frame this as specification rot: the spec keeps looking authoritative while quietly losing its connection to what the system actually does. The core definition of spec-driven development treats the spec as the source of truth, but a source of truth only stays true if something keeps checking it against reality.

A Traceability Model For AI-Assisted Development

A workable traceability model needs six identifiers that connect a business requirement all the way down to the lines of code and the pull request that implemented it. Most teams already have three or four of these; the missing ones are usually the design decision ID and the explicit link back from tests and commits.

Identifier	Lives in	Example
Requirement ID	`requirements.md` or spec tool	`REQ-014`
Design decision ID	ADR / decision record	`ADR-0032`
Task ID	task breakdown or issue tracker	`TASK-014-3`
Test ID	test file or test name	`test_req_014_password_reset`
Commit / PR link	Git history	`PR #482`
Changed files	Git diff	`auth/reset.go`, `auth/reset_test.go`

The relationships between these identifiers form a graph rather than a straight line, because one requirement can spawn several tasks, and one pull request can touch several requirements at once.

graph TD
    REQ["Requirement<br/>REQ-014"] --> ADR["Design Decision<br/>ADR-0032"]
    ADR --> TASK["Task<br/>TASK-014-3"]
    TASK --> CODE["Code Change<br/>auth/reset.go"]
    TASK --> TEST["Test<br/>test_req_014_password_reset"]
    CODE --> PR["Pull Request<br/>#482"]
    TEST --> PR
    PR --> COMMIT["Commit history"]

Storing this graph as structured data, not prose, is what lets you query it later. GitHub's Spec Kit ecosystem has moved in exactly this direction: extensions like spec-kit-trace scan REQ-XXX tokens embedded in spec files and test files and generate a deterministic matrix from that literal text match, deliberately avoiding fuzzy name-based guessing that produces silent false positives.

Spec-To-Test Mapping: Turning Acceptance Criteria Into Test Cases

Every acceptance criterion in a spec is, by construction, a behavioral assertion: given this state, when the actor does this, then the system should respond that way. That is already the shape of a test case, which is why the strongest SDD workflows generate tests from the same acceptance criteria that generate the code, instead of asking the code-generating agent to also invent its own tests after the fact.

A widely used format for writing these criteria is EARS (Easy Approach to Requirements Syntax), which forces each requirement into an unambiguous, testable pattern such as "When <trigger>, the system shall <response>." That structure maps cleanly onto four categories of test that every requirement should carry:

Positive tests — the happy path the requirement explicitly describes.
Negative tests — inputs or states the requirement says must be rejected.
Boundary tests — the edges of ranges, limits, and thresholds mentioned in the acceptance criteria.
Migration tests — behavior for data or state that predates the requirement, so an old record does not silently bypass a new rule.

Requirement type	Test category to add	Common miss
"System shall reject X"	Negative	Only the accept path is tested
"Limit is N items"	Boundary	N-1, N, and N+1 are not all covered
"New field replaces old field"	Migration	Old records with no new field crash silently
"Within 60 seconds"	Boundary + timing	Test asserts logic, not the actual time budget

Unit tests written this way still matter as the fast, cheap layer of the pyramid; the practical patterns for structuring them are covered in the Go unit testing guide and the Python unit testing guide. What traceability adds on top is a literal, stable requirement token embedded in the test name or a test comment, so a later query can prove — not assume — that REQ-014 has coverage.

Spec-To-Code Mapping: From Design Plans To A Trace Table

Spec-to-test mapping proves behavior; spec-to-code mapping proves scope. It answers a different question: which files were actually supposed to change for this requirement, and did the diff stay inside that boundary or spill into unrelated modules?

A design plan that lists affected files up front — even a rough list — gives you something to diff the real pull request against later. Comments in code should only reference a requirement ID when doing so adds information a reviewer cannot get from the spec itself; a comment repeating the requirement text verbatim is noise, but // enforces REQ-014 boundary: max 5 reset attempts per hour earns its place because the number is otherwise invisible in the diff.

A generated trace table turns this into something reviewable in seconds rather than something a reviewer has to reconstruct by reading both documents side by side:

Requirement	Design decision	Files changed	Tests	Status
REQ-014	ADR-0032	`auth/reset.go`, `auth/reset_test.go`	`test_req_014_*` (4)	Covered
REQ-015	ADR-0032	`auth/reset.go`	none	Gap
REQ-016	—	`auth/notify.go`	`test_notify_basic`	Orphan spec link

That single table surfaces two of the most common failure patterns at a glance: REQ-015 changed code with zero matching tests, and the test attached to REQ-016 does not actually reference a requirement ID, which means either the spec is missing or the test was misfiled.

The Pull Request Workflow: Reviewing Spec, Code, And Test Diffs Together

A pull request built around traceability reviews three diffs side by side instead of one: what changed in the spec, what changed in the code, and what changed in the tests. The review question stops being "does this look right?" and becomes the far more specific "which requirement does this change satisfy, and does the evidence prove it?"

sequenceDiagram
    participant Dev as Developer or Agent
    participant PR as Pull Request
    participant CI as CI Pipeline
    participant Rev as Reviewer

    Dev->>PR: Open PR with spec diff + code diff + test diff
    PR->>CI: Trigger traceability checks
    CI->>CI: Verify REQ-ID present in PR description
    CI->>CI: Run spec-to-test coverage check
    CI->>CI: Run spec-to-code file-scope check
    CI-->>PR: Post trace report as PR comment
    Rev->>PR: Review against "which requirement does this satisfy?"
    Rev->>PR: Approve or request changes

A short, concrete reviewer checklist works better here than a long one, because reviewers skip long checklists under deadline pressure:

Does the PR description name the requirement ID(s) it satisfies?
Does every changed file appear in the design plan's affected-files list, or is the extra scope explained?
Does at least one new or existing test reference each requirement ID touched by this PR?
If the spec changed, did the code and tests change in the same PR, or is there a tracked follow-up?

Automating Traceability In CI

Manual review catches drift only as often as reviewers remember to look for it, which is why the checks above belong in CI rather than in a wiki page nobody re-reads. The same GitHub Actions cheatsheet patterns you already use for build and test jobs apply directly here — traceability checks are just another job in the same pipeline.

Practical automation ideas, roughly in order of effort:

CI checks for spec files — fail the build if a spec file was edited without a corresponding code or test change in the same PR, or vice versa.
Require requirement IDs in PR titles or descriptions — a lightweight regex check (REQ-\d+) blocks merges that don't name what they implement.
Agent-generated trace summaries — have an agent produce a short summary of which requirements a PR touches, for a human to confirm rather than write from scratch.
Test coverage by acceptance criterion, not just by line — line coverage tells you code ran; requirement coverage tells you a claim was checked.
Stale spec warnings — flag specs that have not been touched in N commits touching their linked files, since long-silent specs are the ones most likely to have quietly rotted.

Extensions built on top of GitHub's Spec Kit already implement several of these mechanically: one scans literal REQ-XXX tokens across spec and test files to build a matrix and flag orphaned tests, and a stricter V-Model-oriented pack goes further, generating a paired test specification for every development specification and producing multiple traceability matrices for teams working under regulatory frameworks such as IEC 62304 or ISO 26262. You do not need that level of ceremony for most projects, but the underlying idea — a deterministic, script-generated matrix rather than a hand-maintained spreadsheet — scales down just as well as it scales up.

Using AI Agents For Traceability, Not As An Oracle

AI agents are well suited to the mechanical parts of traceability and poorly suited to being the final judge of whether a requirement was actually satisfied. Three tasks fit an agent's strengths directly:

Compare spec and diff — ask the agent to list every requirement mentioned in the spec files touched by a PR, and every one it did not find corresponding code for.
Find uncovered requirements — ask the agent to scan the test suite for requirement tokens and report which requirements in the spec have none.
Detect code not described by spec — ask the agent to flag changed files or functions that touch requirement-bearing modules but do not correspond to any requirement ID in the diff.

The failure mode to guard against is trusting the agent's summary as ground truth instead of as a reviewer's starting point. An agent can misread a comment, miss a requirement token split across two files, or confidently declare coverage for a test that only exercises the code path superficially. Treat every agent-generated trace report the way you would treat a junior reviewer's pass: useful, fast, and still subject to a second look before it gates a merge. This is the same caution that applies to decision records for AI-driven development — the record only stays trustworthy if something other than the agent that wrote it eventually checks it.

A Minimal Traceability Template You Can Copy

You do not need a heavyweight framework to start. A five-file template, checked into the repository next to the code it describes, covers the essentials:

docs/
  requirements.md     # REQ-IDs with EARS-style acceptance criteria
  design.md           # ADR-IDs, affected files, architecture decisions
  tasks.md            # TASK-IDs mapped to one or more REQ-IDs
  tests.md            # which test files/functions reference which REQ-IDs
  traceability.md     # generated table: REQ -> ADR -> TASK -> files -> tests -> PR

requirements.md, design.md, and tasks.md are written or edited by humans and agents together, the same way the spec-driven development workflow already describes. tests.md and traceability.md should be generated, not hand-maintained, even if the generator is a short script that just greps for REQ-\d+ across the test directory and the spec files — hand-maintained trace tables are themselves a form of drift risk, because nobody updates a spreadsheet under deadline pressure.

Conclusion

Spec-driven development is not finished the moment code comes out of an agent; it is only useful once code, tests, and specs keep each other honest over time, through PRs, refactors, and requirement changes that arrive months apart. A traceability model built from six plain identifiers, enforced by a handful of CI checks, and reviewed with a short PR checklist gets you most of the benefit without the overhead of a full compliance framework. Start with the minimal template, wire the cheapest CI check first — requirement IDs in PR descriptions — and add the trace table and stale-spec warnings once that habit sticks.

Traceability is one piece of a larger testing and documentation discipline covered across the App Architecture in Production cluster, and it sits alongside the tooling questions explored in the AI developer tools cluster for teams choosing which agent workflows to standardize on.

Dead Letter Queues: Handling Poison Messages in Distributed Systems

Rost — Fri, 31 Jul 2026 13:26:50 +0000

A dead-letter queue is the safety net that catches messages your consumers cannot process, so one broken payload does not block or silently drop everything behind it in the queue.

Every message-driven system eventually receives a message it cannot handle: a malformed payload, a schema that changed underneath the consumer, or a downstream call that fails no matter how many times you retry it. Without a dead-letter queue, that message either blocks the head of the queue forever or gets silently discarded, and both outcomes are worse than knowing about the failure.

A DLQ turns an invisible failure into a visible, inspectable one. It gives you a place to quarantine the message, alert on it, and decide — deliberately, not by accident — whether to fix and replay it or discard it for good.

The mechanics differ across brokers, but the underlying pattern is the same everywhere: a delivery-attempt counter, a threshold, and a destination for messages that cross it. This guide covers what a DLQ actually does, how to tell a poison message from a transient failure, when to retry versus discard, and how to replay safely once you have fixed the root cause. For the broader integration-patterns context this pattern sits inside, see the App Architecture.

What Is a Dead Letter Queue

A dead-letter queue is a separate, ordinary queue that a broker or consumer routes a message to after that message fails processing too many times. It is not a special construct — RabbitMQ's dead-letter queue is a regular queue bound to a regular exchange, and an SQS DLQ is a regular standard or FIFO queue. What makes a queue a "DLQ" is purely that something else points failed messages at it.

flowchart LR
    P[Producer] --> Q[Main Queue]
    Q --> C[Consumer]
    C -- ack: success --> Done[Message deleted]
    C -- fail / nack / timeout --> Q
    Q -- retry budget exhausted --> DLQ[Dead Letter Queue]
    DLQ --> I[Inspect / alert]
    I -- fix root cause --> R[Replay to main queue]
    I -- unrecoverable --> D[Archive / discard]

Each broker implements the redirect differently:

Amazon SQS uses a redrive policy with a maxReceiveCount. Once a message has been received that many times without being deleted, SQS moves it to the configured deadLetterTargetArn. AWS explicitly recommends keeping the DLQ's message retention period longer than the source queue's, because the original enqueue timestamp — not the move time — still governs expiry.
RabbitMQ dead-letters a message when it is rejected with requeue=false, its per-message TTL expires, the queue hits a length limit, or a quorum queue exceeds its delivery-limit. You configure this with the x-dead-letter-exchange (and optionally x-dead-letter-routing-key) queue arguments, and RabbitMQ attaches x-death headers recording the reason, the origin queue, and how many times it happened.
Apache Kafka has no broker-native DLQ. Kafka only tracks offsets; it has no concept of a "failed" message. The dead-letter topic pattern is something you build in the consumer, in a Kafka Streams topology, or in a Kafka Connect connector — commonly paired with a retry-topic tier before the terminal DLT, as Spring Kafka's @RetryableTopic and DeadLetterPublishingRecoverer do.
Azure Service Bus dead-letters automatically once a message's delivery count exceeds MaxDeliveryCount (default 10), and also for a handful of system reasons such as TTLExpiredException, HeaderSizeExceeded, and MaxTransferHopCountExceeded, each recorded in the message's DeadLetterReason property.

For a broader view of how brokers and streaming platforms fit together operationally rather than as a reliability pattern, Apache Kafka Quickstart and RabbitMQ on AWS EKS vs SQS cover the infrastructure side of running these brokers.

Poison Messages

A poison message is one that will never succeed no matter how many times a consumer retries it — a malformed JSON payload, a schema field that a producer renamed, a business rule violation, or a bug that throws on a specific input every single time. That is different from a transient failure, where the message is fine but the environment briefly is not: a downstream timeout, a database connection blip, a rate limit response.

Treating both failure types the same way is the most common DLQ mistake. If you dead-letter on the first failure, you punish transient errors that would have succeeded on retry. If you retry poison messages dozens of times before giving up, you waste compute, delay unrelated messages behind them (on ordered queues and partitions), and flood your logs with the same stack trace.

A few detection signals help separate the two:

Exception type. Deserialization errors, validation errors, and ClassCastException-style failures are almost always permanent. Spring Kafka's DefaultErrorHandler explicitly treats certain exceptions as fatal and skips retries for them rather than exhausting the retry budget first.
Repeat count with no variance. RabbitMQ's x-death header array lets you see exactly how many times a message has been dead-lettered and why; a message with a growing count and an identical x-first-death-reason on every cycle is poison, not unlucky.
Consistent failure across replicas. If every consumer instance fails on the same message while succeeding on everything around it, the message itself is the problem, not the infrastructure.

For distinguishing retryable from non-retryable failures at the code level — the same classification a DLQ policy depends on — see Go Error Handling Architecture: Boundaries and Patterns.

Retry vs Discard

The core policy decision behind every DLQ is the retry threshold: how many delivery attempts a message gets before it is quarantined. Get this too low and you dead-letter messages that would have succeeded after a brief downstream hiccup. Get it too high and a poison message sits in the main queue for a long time, consuming worker capacity and — on ordered systems — blocking everything queued behind it.

Current guidance across the major brokers converges on similar numbers:

Broker	Mechanism	Typical threshold
Amazon SQS	`maxReceiveCount` in redrive policy	3–5 for mixed transient/permanent workloads
RabbitMQ (quorum queues)	`delivery-limit` policy argument	3–5, tuned per queue
Azure Service Bus	`MaxDeliveryCount`	Default 10, often reduced for latency-sensitive queues
Kafka (via retry topics)	Retry-count header + retry-topic tier	3–4 retry-topic hops before the terminal DLT

A practical middle ground many teams land on is: start conservative (2–3 attempts), watch the actual failure mix in production, and raise the threshold only for queues where you can show most failures resolve within a few retries. Pair the retry count with exponential backoff and jitter between attempts so a downstream outage does not turn into a retry storm — the same discipline covered in backoff and circuit-breaker design. A circuit breaker at the integration boundary complements this: it stops sending requests to an unhealthy dependency instead of letting every message in the queue individually discover the outage and dead-letter one by one.

Once a message is in the DLQ, "discard" should still be a deliberate action, not neglect. Set a retention period on the DLQ itself — long enough to investigate (AWS recommends the DLQ retention exceed the source queue's; a week is a common floor for RabbitMQ DLQs) — and alert on DLQ depth and age so failures get triaged instead of silently expiring. A message that ages out of the DLQ unexamined is a message you decided to lose without deciding to lose it.

Idempotency matters just as much here as it does anywhere else duplicates can occur: a message that gets redriven from a DLQ back to the main queue is, functionally, a duplicate delivery. If your consumer is not safe to run twice on the same message, redriving from a DLQ can create the exact duplicate-side-effect bug you were trying to avoid. See Idempotency in Distributed Systems That Actually Works for the consumer-side patterns that make redrive safe.

Replay Strategies

Getting a message out of the DLQ correctly is its own discipline, separate from getting it in.

Fix the root cause first. Deploying the consumer fix before replaying is the difference between a clean recovery and re-poisoning the queue with the same failure a second time.
Redrive deliberately, not automatically. SQS supports a redrive-to-source feature that moves messages back to their original queue (or another destination) on demand; RabbitMQ and Kafka require you to build the equivalent consumer or tooling yourself. Either way, treat replay as an operator-triggered action with a record of what was replayed and when.
Preserve ordering where it matters. For Kafka, the dead-letter topic should have at least as many partitions as the source topic and should retain the original message key, so that replayed messages land back on the correct partition and preserve per-key ordering.
Cap replay attempts. A message that fails again after a fix-and-replay cycle is not transient — route it to a permanent archive (a database table, an object-storage bucket) instead of looping it through the DLQ indefinitely. RabbitMQ's own docs warn that a dead-lettered message can be routed between queues only a limited number of times (16) before further TTL-based dead-lettering is disabled.
Never let a DLQ dead-letter into itself. If your DLQ has its own x-dead-letter-exchange (RabbitMQ) or its own redrive policy (SQS) pointed back at the same chain, a replay failure can create an infinite loop. Keep the DLQ's own dead-letter configuration empty, or point it at a strictly terminal archive.
Alert on volume, not just presence. A single message in a DLQ is a data point; a sudden spike is an incident. Wire DLQ depth and message age into the same alerting pipeline you use for everything else — see Modern Alerting Systems Design for Observability Teams for routing and noise-reduction practices that apply directly to DLQ alerts.

If your workflow involves multi-step, long-running processes rather than single messages, the same dead-letter thinking applies at the workflow layer — a saga's compensation logic needs the same "quarantine, inspect, decide" discipline when a step fails permanently instead of transiently. And when the events themselves originate from a database write, the transactional outbox pattern already builds dead-letter handling into the relay worker, so the pattern shows up one layer earlier than the broker.

Where DLQs Fit in the Bigger Picture

A dead-letter queue does not make failures go away — it makes them survivable and reviewable instead of silent. It works best alongside retries with backoff for the transient case, idempotent consumers so redrive is safe, and a circuit breaker so a struggling dependency does not flood the main queue (and, eventually, the DLQ) with the same failure thousands of times over. Treat the DLQ threshold, retention, and alerting as first-class configuration decisions, not defaults you leave untouched, and dead letters become a diagnostic tool instead of a place where data quietly disappears.

Useful Links

LLM Wiki Maintenance: Drift, Contradictions and Review

Rost — Mon, 20 Jul 2026 08:29:50 +0000

An LLM Wiki fails when old facts remain plausible, contradictions become polished, and generated summaries drift from their sources.

Maintenance is the real product of any compiled knowledge system. Creating wiki pages is straightforward compared with keeping them trustworthy across months of ingest, edits, rewrites, and new sources.

This article covers the operational side of LLM Wiki systems: drift detection, contradiction checks, citation discipline, linting, Git review, and maintenance workflows. It assumes you already understand the basic pattern described in LLM Wiki - Compiled Knowledge That RAG Cannot Replace: raw sources are compiled into durable Markdown pages that humans and agents can query later. The calm but opinionated view is that an LLM Wiki without maintenance is just a nicer-looking knowledge graveyard.

What Makes LLM Wiki Maintenance Different

Traditional wikis rot because people stop updating them. RAG systems drift because the corpus changes, chunks get stale, metadata is weak, and retrieval keeps finding plausible but outdated fragments. The deeper reason is that retrieval and representation solve different problems: retrieval can be re-run against fresh data, but a representation you already compiled has to be actively kept honest.

An LLM Wiki has a different failure mode. It can look clean even when it is wrong.

The pages may be well formatted. The links may work. The summaries may sound balanced. But underneath that neat surface, the system may have dropped critical facts, merged incompatible concepts, cited summaries instead of sources, or preserved an old decision as if it still applied.

That is why LLM Wiki maintenance must check both structure and meaning.

The Maintenance Goal

The goal is not to make every page perfect. That is too expensive, and it usually leads to abandoned systems.

The goal is to keep the wiki useful, inspectable, and recoverable.

A maintained LLM Wiki should make it easy to answer these questions:

What sources support this claim?
When was this page last reviewed?
Has this concept changed?
Are there conflicting pages?
Is this summary still current?
Did the agent rewrite more than it should?
Can we roll back a bad update?
Can a human understand why the page says what it says?

If the answer is no, the problem is not only content quality. It is system design.

A Simple Maintenance Loop

A useful LLM Wiki needs a repeatable loop. The loop should be simple enough to run often and strict enough to catch drift before it becomes invisible.

flowchart TD
    A[Add or update source] --> B[Compile into wiki pages]
    B --> C[Update links and indexes]
    C --> D[Run structural lint checks]
    D --> E[Run semantic review checks]
    E --> F[Human review of risky changes]
    F --> G[Commit approved changes]
    G --> H[Schedule stale-page review]
    H --> A

This loop is not glamorous. That is the point.

A knowledge system becomes durable through boring maintenance: source preservation, explicit review, predictable structure, and small safe updates. The same instinct drives evergreen notes: a note or a wiki page only compounds in value if someone keeps refining it instead of leaving it to rot after the first draft.

Types of Drift in an LLM Wiki

Knowledge drift is not one thing. Different drift types require different checks. A good maintenance system should name them clearly.

Source Drift

Source drift happens when the underlying source material changes.

For example, a tool releases a new version, a policy is updated, an API changes, or a vendor deprecates an old feature. The old wiki page may still be accurate for the previous version, but wrong for current use.

Source drift is dangerous because the old claim may still be true in historical context. The problem is not that the claim is fake. The problem is that it no longer answers the current question.

Maintenance response:

Record source dates
Record last reviewed dates
Mark version-specific pages clearly
Link old pages to superseding pages
Avoid mixing old and new versions without labels

Concept Drift

Concept drift happens when the meaning of a term changes over time.

This is common in AI and software architecture. Terms like "agent", "memory", "RAG", "workflow", "tool use", "structured output", and "context engineering" can shift meaning quickly.

A wiki can accidentally preserve several meanings of the same term without explaining the difference. That creates confusing pages that sound coherent but combine incompatible ideas.

Maintenance response:

Maintain glossary pages
Add "meaning in this wiki" sections
Separate overloaded concepts into distinct pages
Link related meanings explicitly
Avoid letting the model silently merge terms

Terminology Drift

Terminology drift is smaller than concept drift but still harmful.

It happens when the wiki uses multiple names for the same thing: "LLM Wiki", "compiled knowledge base", "AI-maintained wiki", "agent-maintained wiki", and "Markdown knowledge base".

Some variation is fine. Too much variation breaks search, linking, and review.

Maintenance response:

Define canonical page names
Keep aliases in front matter or page metadata
Redirect duplicate pages
Lint for near-duplicate titles
Use consistent anchor text in internal links

Decision Drift

Decision drift happens when a past decision remains documented but no longer reflects current practice.

For example, a page might say that the project uses vector RAG for all document search, while newer pages describe an LLM Wiki workflow. Both statements may be historically true, but the wiki must show which one is current.

This matters for architecture notes, engineering processes, content strategy, and internal tooling. The same discipline applies to decision records in AI-driven development: a decision page is only trustworthy if superseded choices are marked as superseded rather than silently overwritten.

Maintenance response:

Mark decisions as proposed, accepted, superseded, or rejected
Keep decision dates
Link superseded decisions to replacements
Preserve historical context
Avoid deleting old decisions without trace

Citation Drift

Citation drift happens when a page cites a source, but the claim no longer matches what the source says.

This can happen after a rewrite. The citation remains in place, but the sentence around it changes. The page still looks sourced, yet the citation no longer supports the claim.

This is one of the most serious LLM Wiki failure modes because it creates false confidence.

Maintenance response:

Check claim-level citation support
Avoid citing only summaries
Keep raw sources
Require citations for important claims
Flag paragraphs with sources but no direct support
Review citation changes in Git diffs

Structure Drift

Structure drift happens when the wiki slowly loses navigability.

New pages are created instead of updating old ones. Index pages fall behind. Duplicate pages appear. Orphan pages accumulate. Related pages stop linking to each other.

The wiki still contains useful knowledge, but finding and trusting it becomes harder.

Maintenance response:

Lint for orphan pages
Maintain index pages
Detect duplicate topics
Require backlinks for canonical pages
Archive rather than scatter
Keep folder roles clear

The Core Maintenance Files

An LLM Wiki should not rely on memory, vibes, or repeated prompting. It needs operating files that tell humans and agents how to maintain it.

A practical structure can look like this:

llm-wiki/
  raw/
    sources/
    transcripts/
    documents/
  wiki/
    index.md
    concepts/
    entities/
    projects/
    decisions/
  maintenance/
    review-log.md
    lint-report.md
    stale-pages.md
    contradiction-report.md
  rules/
    AGENTS.md
    schema.md
    style-guide.md
    citation-policy.md
    source-policy.md

The exact names do not matter. The roles do.

The wiki needs source storage, compiled pages, review artifacts, and rules that survive across sessions.

AGENTS.md

AGENTS.md should explain how the agent is allowed to work.

It should answer questions like:

Should the agent update existing pages before creating new ones?
When should it ask for review?
How should it cite sources?
Can it reorganize folders?
Can it rewrite old pages?
What should it do with contradictions?
What should it never delete?

A useful rule is:

Prefer small, source-backed updates over broad rewrites.

That one sentence prevents many bad maintenance habits. It also draws a useful boundary relative to agent memory: memory shapes how an agent behaves in the moment, while the wiki's rules file shapes what the agent is allowed to change in the shared knowledge base.

schema.md

schema.md should define the page structure.

For example:

# Page Title

## Summary
Short current summary.

## Key Claims
- Claim with source reference.
- Claim with source reference.

## Current Status
Current, historical, superseded, draft, or uncertain.

## Details
Main compiled explanation.

## Related Pages
Internal links.

## Sources
Raw source references.

## Review Notes
Last reviewed date and reviewer.

A schema does not need to be rigid. But without a schema, the wiki becomes a pile of inconsistent essays. The schema gives the agent something to preserve.

citation-policy.md

The citation policy should say what requires a source. At minimum, require citations for:

Technical claims
Comparisons
Product behavior
Version-specific statements
Benchmark results
Pricing
Legal or compliance claims
Current status claims
Claims copied or derived from a source

A good citation policy also says what not to do. Do not cite a compiled page as if it were the original source. Do not attach citations to paragraphs they do not support. Do not keep a citation after rewriting a claim unless the source still supports it.

Citation discipline is the difference between a useful LLM Wiki and decorative confidence.

source-policy.md

The source policy should define how raw sources are stored.

For example:

Keep original files when possible
Preserve URLs and access dates
Store copied notes separately from generated summaries
Record source type
Mark low-confidence sources
Avoid overwriting raw material
Keep superseded sources when they explain historical decisions

This policy protects the wiki from a common mistake: summarizing a source, discarding the original, and then treating the summary as evidence.

That is not knowledge management. That is lossy compression.

What an LLM Wiki Linter Should Check

A linter should check structure first. Structural problems are easier to automate and often reveal deeper semantic problems.

Useful structural checks include:

Broken internal links
Orphan pages
Duplicate titles
Missing source sections
Missing review dates
Pages without backlinks
Empty or placeholder sections
Very long pages without section structure
Pages with no incoming links from an index
Inconsistent naming conventions
Invalid front matter
Stale "current" pages older than a review threshold

These checks do not prove the knowledge is true. They prove the wiki is still maintainable. That is a necessary starting point.

Semantic Checks

Semantic checks are harder but more valuable.

Useful semantic checks include:

Claims without direct source support
Two pages making incompatible claims
Old decisions presented as current
Duplicate concepts with different names
One concept page mixing several meanings
Summaries that omit known constraints
"Best" or "recommended" claims without criteria
Version-specific claims without version labels
Pages that contradict newer sources

These checks should not automatically rewrite the wiki. They should usually produce a report for review.

The safer pattern is:

Detect automatically.
Explain clearly.
Update deliberately.
Review risky changes.

Contradiction Detection

Contradiction detection is not just asking an LLM whether two pages contradict each other.

That can help, but it is too vague. A better approach is to compare claims.

A contradiction workflow can look like this:

flowchart TD
    A[Extract claims from page] --> B[Find related pages and sources]
    B --> C[Extract claims from related material]
    C --> D[Group claims by subject]
    D --> E[Compare status, date, version, and scope]
    E --> F{Conflict found?}
    F -->|No| G[No action]
    F -->|Yes| H[Classify conflict]
    H --> I[Create contradiction report]
    I --> J[Human or agent-assisted resolution]

This works better because many apparent contradictions are not real contradictions.

One page may describe version 1.0 and another version 2.0. One page may describe personal use and another enterprise use. One page may describe the design goal and another the implementation reality.

The contradiction report should include:

Conflicting claims
Pages involved
Source references
Date or version context
Likely explanation
Suggested resolution
Whether human review is required

Do not let the agent silently resolve contradictions by blending both claims into a vague compromise. That creates smooth nonsense.

How to Resolve Contradictions

There are several valid ways to resolve a contradiction.

If one claim is outdated, mark it as superseded and link to the newer page.
If both claims are true in different contexts, split the context clearly.
If the sources disagree, preserve the disagreement and explain it.
If the wiki page invented or overgeneralized a claim, remove or narrow the claim.
If the contradiction reflects a real unresolved decision, create or update a decision page.

The goal is not to remove all tension. Some tension is useful. A good wiki shows where knowledge is settled and where it is still uncertain.

Stale Page Review

Every compiled page should have a review status.

A simple metadata block can be enough:

status: current
last_reviewed: 2026-07-10
review_after: 2026-10-10
source_confidence: medium

For fast-moving topics, use short review windows. For stable concepts, use longer windows.

Page type	Review interval
Tool version pages	30 to 90 days
Pricing or availability pages	7 to 30 days
Architecture principles	6 to 18 months
Historical decision records	Only when superseded
Glossary pages	3 to 12 months
Source summaries	When source changes
Comparison pages	30 to 180 days

The interval is less important than the habit. A page without a review date is a page that will eventually lie quietly.

Source-Aware Updates

When a new source arrives, the agent should not simply write a new summary page.

It should ask:

Does this update an existing concept?
Does it supersede a previous source?
Does it contradict an existing page?
Does it add a new entity or term?
Does it change a recommendation?
Does it require an index update?
Does it affect decision pages?

A good ingest workflow updates the wiki as a graph, not as a stack of isolated summaries.

This is why one source may touch many pages. That is normal. The maintenance challenge is making those updates small, reviewable, and traceable.

Review with Git Diffs

Git is one of the best maintenance tools for an LLM Wiki.

Not because Git is fashionable, but because generated knowledge needs reviewable change history. A Git diff shows what the agent changed, deleted, moved, or rephrased.

Use Git to review:

New pages
Deleted sections
Changed claims
Changed citations
Renamed pages
Index updates
Link changes
Status changes
Broad rewrites

The most important review habit is to inspect deletions. LLMs often remove details while making prose cleaner.

Clean prose is not always better knowledge.

Safe Commit Strategy

Avoid giant commits like:

Update wiki

Use commits that explain the operation:

ingest: add source notes for Qwen embedding release
maint: mark old vector store comparison as superseded
lint: fix broken internal links in RAG pages
review: update LLM Wiki page citations
refactor: split agent memory concept from long-term memory

This gives future humans and agents a useful history.

Commit messages are part of the knowledge system. They explain why the wiki changed.

Review Levels

Not every page needs the same review level. Use risk-based review.

Low-risk changes:

Broken link fixes
Formatting cleanup
Adding backlinks
Adding index entries
Correcting typos
Adding aliases

Medium-risk changes:

Adding source summaries
Creating new concept pages
Updating recommendations
Merging duplicate pages
Changing page status

High-risk changes:

Deleting claims
Rewriting canonical pages
Changing decision records
Updating security, legal, pricing, or benchmark claims
Resolving contradictions
Marking content as current or superseded

High-risk changes should get human review. That may be you, a maintainer, or a domain owner.

The agent can prepare the change, but it should not always be the final authority.

A Maintenance Dashboard

A simple maintenance dashboard can live in Markdown.

Example:

# LLM Wiki Maintenance Dashboard

## Needs Review
- wiki/concepts/agent-memory.md - review overdue
- wiki/projects/search-index.md - conflicting claims found

## Stale Pages
- wiki/tools/ollama.md - last reviewed 120 days ago
- wiki/concepts/rag.md - fast-moving topic, review due

## Broken Links
- wiki/index.md -> wiki/concepts/old-page.md

## Orphan Pages
- wiki/entities/vendor-x.md

## Contradiction Reports
- reports/contradictions/2026-07-10-agent-memory.md

## Recent Ingests
- raw/sources/2026-07-09-new-paper.md

This is not a fancy admin panel. It is a working surface.

For many solo or small-team systems, a Markdown dashboard is enough.

Metrics for a Healthy LLM Wiki

You can track wiki health without overengineering it.

Useful metrics include:

Number of pages
Pages without sources
Pages without review dates
Pages past review date
Broken internal links
Orphan pages
Duplicate titles or aliases
Contradiction reports open
Contradiction reports resolved
Average page age by cluster
Raw sources not yet ingested
Pages changed without human review
Canonical pages changed in the last 30 days

These metrics are not vanity numbers. They tell you whether the system is becoming easier or harder to trust.

A growing wiki with rising orphan pages and stale reviews is not compounding knowledge. It is compounding maintenance debt.

Maintenance Prompts

Maintenance prompts should be narrow. Broad prompts produce broad rewrites.

Structural Lint Prompt

Review the wiki structure.

Check for:
- broken internal links
- orphan pages
- duplicate or near-duplicate page titles
- pages missing source sections
- pages missing last_reviewed metadata
- index pages that do not link to new pages

Do not rewrite content.
Produce a Markdown report with findings and suggested fixes.

Citation Review Prompt

Review citations for the selected page.

For each important claim:
- identify the supporting source
- check whether the source directly supports the claim
- flag claims with weak, missing, or mismatched support
- do not rewrite the page yet

Return a table with:
claim, current citation, support level, issue, suggested action.

Contradiction Check Prompt

Check this page for contradictions against related pages.

Steps:
1. Extract the main claims from the page.
2. Find related pages through links, backlinks, aliases, and search.
3. Extract potentially conflicting claims.
4. Classify each conflict as:
   - real contradiction
   - version difference
   - scope difference
   - terminology difference
   - unresolved uncertainty
5. Recommend a resolution.

Do not edit files until the contradiction report is reviewed.

Stale Page Review Prompt

Review this page for staleness.

Check:
- dates
- version-specific claims
- tool behavior
- recommendations
- links to superseded pages
- sources newer than the page review date
- claims marked current without recent support

Return:
- keep current
- update needed
- mark historical
- mark superseded
- split into versioned pages

Safe Update Prompt

Update the selected wiki page using the provided source.

Rules:
- preserve existing useful structure
- do not remove sourced claims unless the source is superseded
- add citations for new claims
- mark uncertainty explicitly
- update related pages only when necessary
- add a review note explaining what changed
- keep the diff small

These prompts are not magic. They are guardrails. The real value comes from making the maintenance action explicit.

Common Maintenance Mistakes

The first mistake is letting the agent rewrite too much.

Large rewrites feel productive because they make pages smoother. They also make it harder to see what changed and easier to lose sharp details.

The second mistake is treating generated summaries as sources.

A summary can be useful, but it is not the evidence. Keep raw sources and cite them when claims matter.

The third mistake is creating new pages instead of updating old ones.

This produces duplicate concepts, conflicting recommendations, and search confusion. The agent should search first, update second, and create new pages only when the concept is genuinely new.

The fourth mistake is deleting historical context.

Old decisions, failed experiments, and superseded recommendations can be valuable. Mark them clearly instead of erasing them.

The fifth mistake is forgetting indexes.

A page that is not linked from the right place is half-lost. Index maintenance is not clerical work. It is navigation architecture.

The sixth mistake is using the same review policy for every page.

A stable conceptual page and a fast-moving tool comparison should not have the same review schedule.

Archive, Supersede, or Delete

Most stale pages should not be deleted immediately.

Outcome	Use when
Archive	The page is historical but still useful
Supersede	A newer page replaces the old answer
Delete	The page is duplicate, empty, wrong, or unrecoverable

A superseded page should link to the replacement.

For example:

Status: superseded
Superseded by: wiki/concepts/agent-memory-architecture.md
Reason: This page used an older definition of agent memory before the project separated session memory, user memory, and compiled knowledge.

This preserves context and reduces confusion.

Deletion should be rare and reviewable. Knowledge systems need pruning, but invisible pruning is dangerous.

Canonical Pages Need Extra Care

Every LLM Wiki eventually develops canonical pages.

These are pages that define core concepts, architecture choices, workflows, or project vocabulary. They are more important than ordinary notes because many other pages depend on them.

Examples:

What is LLM Wiki
RAG vs compiled knowledge
Agent memory
Source policy
Review workflow
Project architecture
Glossary
Index

Changes to canonical pages should require stricter review.

A small error on an obscure page is local. A small error on a canonical page can distort the whole wiki.

How Often Should You Run Maintenance

For an active LLM Wiki, run small maintenance frequently and deep maintenance occasionally.

Frequency	Maintenance task
Every ingest	Update links, sources, indexes, and review notes
Weekly	Run structural lint checks
Monthly	Review stale pages and orphan pages
Monthly	Check contradictions in active clusters
Quarterly	Review canonical pages
Quarterly	Archive or supersede old pages
Before major use	Run citation checks on pages used for reports, articles, or decisions

The schedule should match risk. A personal research wiki can be lighter. A wiki used for technical publishing, customer support, or engineering decisions needs more discipline.

Using an LLM Wiki for Technical Publishing

For a technical blog or documentation site, the LLM Wiki can become the knowledge layer behind article planning.

It can track:

Canonical explanations
Repeated definitions
Internal link opportunities
Stale comparisons
Article gaps
Topic clusters
Source notes
Decisions about structure
Claims that need verification

This is useful, but it raises the maintenance bar.

If the wiki feeds published articles, then stale wiki pages can become stale public content. If the wiki contains contradictions, those contradictions may leak into articles. If citations are weak, generated drafts will inherit that weakness. The scoped summaries, schema-based extraction, and human review loops described in AI for Knowledge Management: Real Workflows That Hold Up are a good template for keeping that editorial pipeline honest.

For publishing workflows, treat the LLM Wiki as an editorial system, not just a private notebook. A digital garden makes this explicit by showing readers which pages are still growing and which are mature, which is exactly the kind of status signal a well-maintained LLM Wiki should also expose.

Maintenance Is Not Optional

An LLM Wiki is attractive because it promises compounding knowledge.

But compounding only works if old knowledge remains useful. Otherwise, the system compounds noise, not insight.

The maintenance burden does not disappear because an agent is involved. It changes shape.

Humans should not have to do every tedious task. Agents can check links, find stale pages, compare claims, draft reports, and propose updates. But humans still need to set policy, review risky changes, and decide what the wiki is allowed to mean.

That is the real bargain.

Conclusion

LLM Wiki maintenance is about keeping compiled knowledge honest.

The important practices are not complicated: preserve sources, cite claims, keep indexes current, detect contradictions, review Git diffs, mark stale pages, and avoid broad unreviewed rewrites.

A good LLM Wiki does not pretend to be magically self-correcting. It makes correction easier.

That is the standard worth aiming for: not perfect knowledge, but inspectable knowledge that can be reviewed, repaired, and trusted over time.

Syncthing File Sync for Self-Hosted Knowledge Systems

Rost — Sun, 19 Jul 2026 12:19:06 +0000

Syncthing keeps files synchronized across devices you control, making it one of the most practical tools for a self-hosted knowledge infrastructure that avoids cloud lock-in.

Unlike cloud storage platforms, Syncthing uses a peer-to-peer model where each device holds its own copy of synced folders and exchanges changes directly with trusted peers. There is no central server that owns your data, no subscription account, and no vendor lock-in. The project is open-source and community-driven, with more details at syncthing.net.

This architecture makes Syncthing especially useful for knowledge workers who manage markdown notes, research documents, PDFs, and project files across a desktop, laptop, home server, and possibly a phone. The tool is simple in concept but requires careful setup to avoid common pitfalls like treating sync as backup or syncing folders that should remain isolated.

What Syncthing Is and Is Not

Syncthing synchronizes files between two or more devices. Each device maintains its own copy of a folder, and changes propagate between trusted peers. Discovery and relay services may help devices find each other across networks, but the storage model remains local-first. The Syncthing documentation covers installation and configuration in detail.

The calm but important opinion is this: Syncthing is excellent when treated as sync infrastructure. It becomes dangerous when treated as backup.

It is not a cloud drive. It is not a complete backup system. It is not a collaboration suite. It is a private, peer-to-peer file synchronization tool.

Why Syncthing Matters for Knowledge Management

Knowledge management is not only about note-taking. It is also about where knowledge lives, how it moves, and whether it remains accessible over time — see the knowledge management guide for the broader picture of tools, methods, and self-hosted platforms this fits into.

A useful personal or team knowledge system often contains:

markdown notes
PDFs and papers
diagrams and screenshots
exported web pages
source snippets and configuration files
meeting notes and project documents
scanned documents and plain text logs

Many of these are just files. That is good. Files are durable, portable, searchable, scriptable, and easy to back up.

Syncthing gives those files movement without forcing them into a vendor platform. You can write notes on one machine, read them on another, keep a copy on a home server, and still use normal tools like grep, ripgrep, Obsidian, VS Code, DokuWiki imports, static site generators, or custom scripts.

For a self-hosted knowledge system, that is a strong architectural property. See personal knowledge management foundations for a broader view of PKM goals and methods, and PKM vs RAG vs Wiki vs Memory Systems for how different knowledge systems operate at different layers.

Distinct System Roles

graph LR
    S["Syncthing<br/>moves files"] --> B["Backup<br/>preserves history"]
    S --> K["Knowledge tools<br/>create and edit"]
    B --> R["Recovery<br/>when things go wrong"]
    K --> S

Keep those roles separate.

Syncthing should not be your only copy of important data. It should not be the only thing standing between you and accidental deletion. It should not be the only recovery mechanism after corruption, ransomware, filesystem failure, or a bad script.

Used correctly, Syncthing is part of a resilient file workflow. Used alone, it can replicate mistakes very efficiently.

Good Use Cases for Syncthing

Syncthing works best when the folder has a clear owner, a predictable structure, and a small number of trusted devices.

Good use cases include:

Syncing an Obsidian or markdown notes vault
Syncing project notes between desktop and laptop
Syncing documents to an always-on home server
Syncing exported PDFs and research material
Syncing configuration files across personal machines
Syncing scanned documents from one machine to another
Syncing selected phone folders to a desktop or NAS
Syncing static-site source notes before publishing
Syncing knowledge archives between a workstation and server

These are not exotic use cases. They are exactly the kind of boring file movement that knowledge workers need every day.

Syncthing is strongest when the folder remains understandable without Syncthing. If you can open the folder in a file manager and understand what it is, you are probably using the tool well.

Risky Use Cases for Syncthing

Syncthing becomes risky when users expect it to behave like backup, collaboration software, or managed cloud storage.

Be careful with:

Using Syncthing as your only backup
Syncing huge folders without thinking about deletion risk
Syncing the same files while multiple apps edit them
Syncing application databases that expect exclusive local access
Syncing browser profiles
Syncing mail stores
Syncing build directories or cache folders
Syncing folders with frequent generated files
Syncing very large photo libraries to low-storage devices
Relying on mobile background sync without testing it

The issue is not that Syncthing is unreliable. The issue is that sync is powerful. It does what you ask, including syncing deletions, conflicts, corrupted files, and accidental edits.

That is why a knowledge system should combine Syncthing with versioning, snapshots, and real backup.

Syncthing Is Not Backup

This point deserves its own section.

Backup is about recovery. Sync is about convergence. These are related, but they are not the same.

If you delete a file on one device, a sync tool may delete it everywhere. If a script corrupts a folder, the corrupted version may sync to other devices. If ransomware encrypts local files, the encrypted versions may be treated as changed files.

File versioning can reduce this risk, but it does not turn Syncthing into a complete backup system. Syncthing versioning is configured per folder and per device, and it mainly protects old versions when changes are received from other devices. It does not magically preserve every local edit before it happens.

A safer design is:

Syncthing for active file movement.
File versioning for short-term mistake recovery.
Filesystem snapshots for local rollback.
Restic, Borg, ZFS send, Btrfs snapshots, or another backup system for real recovery.

That is more boring than pretending sync is backup. It is also much safer.

Recommended Knowledge Workflow

A practical self-hosted knowledge setup can look like this:

graph TD
    subgraph "Desktop Workstation"
        D["Main editing device<br/>Full notes and documents"]
    end
    subgraph "Laptop"
        L["Mobile editing device<br/>Same knowledge folders"]
    end
    subgraph "Home Server or NAS"
        H["Always-on sync target<br/>Receive-side versioning<br/>Filesystem snapshots<br/>Separate backup job"]
    end
    subgraph "Phone"
        P["Selected folders only<br/>Camera scans or quick capture<br/>Avoid full archive sync"]
    end
    D <--> H
    L <--> H
    P --> H

This shape works because the always-on server becomes the stable point in the system. It does not need to be a central cloud server, but it gives your sync topology a reliable anchor.

For knowledge management, that anchor matters. Laptops sleep. Phones throttle background services. Desktops are not always on. A small home server or NAS gives the system somewhere steady to converge.

Folder Design Matters

Do not create one giant "sync everything" folder.

Create separate folders for separate purposes:

knowledge-notes
knowledge-documents
research-papers
project-notes
scans-inbox
static-site-drafts
configs

This gives you better control over:

Which devices receive which files
Which folders need versioning
Which folders need snapshots
Which folders can be send-only
Which folders should avoid mobile devices
Which folders have privacy or size concerns

Folder boundaries are architecture. They define trust, storage, recovery, and operational behavior.

For example, an Obsidian vault might sync to laptop, desktop, and server. A large PDF archive might sync only to desktop and server. A phone scan inbox might be send-only from the phone to the server.

That is much cleaner than treating every device as equal.

Understanding Syncthing Folder Types

Syncthing supports different folder behavior depending on how you want changes to flow.

Send and receive is the default mode. A device both sends local changes and receives remote changes. This is the normal choice for active editing across trusted devices.

Send-only sends local changes to other devices but does not accept remote changes as authoritative. This can be useful for source folders where one device should be treated as the origin.

Receive-only receives changes but does not publish local changes to the rest of the cluster. This can be useful for mirrors, replication targets, and backup-adjacent destinations where local edits should not be sent back.

These modes are powerful, but they can also make the system harder to reason about. Use them intentionally. For most everyday folders, send and receive is simpler.

A Good Folder Layout for Notes

For markdown-based notes, keep the structure predictable.

Example:

knowledge-notes/
  inbox/
  projects/
  areas/
  references/
  archive/
  attachments/

This works well with tools like Obsidian, VS Code, ripgrep, static site generators, and command-line scripts.

Avoid putting temporary exports, generated indexes, and application caches into the same folder unless you really want them synced. Use ignore patterns for files that are noisy or machine-specific.

Examples of files you may want to ignore:

.DS_Store
Thumbs.db
*.tmp
*.swp
.cache/
node_modules/
dist/
build/

For Obsidian specifically, think carefully before syncing all plugin state. Some settings are useful across devices, but workspace state can be annoying if it constantly changes window layouts or active panes.

Syncthing and Obsidian

Syncthing is often used to sync Obsidian vaults because Obsidian stores notes as local markdown files.

This is a good pairing. Obsidian gives you the writing and linking interface. Syncthing moves the files. Your notes remain plain files.

The main risks are conflicts and mobile behavior.

If you edit the same note on two devices before they sync, Syncthing may create conflict files. That is better than silent data loss, but it still requires cleanup. The practical habit is to let devices sync before editing the same active note elsewhere.

For mobile devices, do not assume background sync behaves exactly like desktop sync. Test it. Open the app, let sync complete, edit a note, and verify the change appears on your other devices.

See the Obsidian vs Logseq comparison for a broader discussion of sync considerations in PKM tools.

Syncthing and DokuWiki or Static Knowledge Sites

Syncthing also works well around file-based or file-friendly knowledge systems.

For DokuWiki, you might use Syncthing to move exported documents, media files, or staging content between machines. Be more careful with live server data, permissions, and concurrent edits. See DokuWiki and self-hosted wiki alternatives for platform options.

For Hugo or other static site generators, Syncthing can be useful for drafts, research notes, and content source files. It should not replace Git for source control, but it can complement Git for non-code knowledge material.

A good rule is:

Use Git for history and collaboration.
Use Syncthing for private file movement.
Use backup for recovery.

Each tool has a job.

Syncthing vs Nextcloud

Syncthing and Nextcloud are often compared, but they solve different problems.

Feature	Syncthing	Nextcloud
Architecture	Peer-to-peer sync	Central server
Best for	Private device-to-device folders	Web-accessible file platform
Storage model	Local copies on devices	Server-first storage
Web UI	Minimal local admin UI	Full web file interface
Sharing links	Not the main purpose	Built in
Users and permissions	Device trust model	User and group model
Calendars and contacts	No	Yes
Office collaboration	No	Possible with add-ons
Knowledge notes	Good for local-first notes	Good for server-centered sharing
Operational style	Lightweight but manual	Heavier but more complete

Use Syncthing when you want private file sync between your own devices.

Use Nextcloud when you want a self-hosted cloud platform with users, web access, sharing, calendars, contacts, and broader collaboration features. See the Nextcloud self-hosting guide for a detailed setup walkthrough.

Use both if you have both needs. For example, Syncthing can handle your local-first notes, while Nextcloud handles family file sharing or browser-based access.

Syncthing vs rsync

Rsync is excellent for one-off or scheduled file copying. It is simple, scriptable, and widely available.

Syncthing is better when you want continuous synchronization across multiple devices without writing your own scheduling, conflict detection, and device discovery logic.

Use rsync for:

Scripted deployments
One-way copies
Server maintenance
Simple backup jobs
Predictable batch transfers

Use Syncthing for:

Continuous multi-device sync
Local-first notes
Personal document movement
Always-on folder convergence
Cross-platform device sync

Rsync is a tool. Syncthing is a small synchronization system.

Syncthing vs Seafile

Seafile is closer to a file sync platform. It can be a good fit when you want a central service, clients, libraries, and a more managed team file sync experience.

Syncthing is more decentralized and simpler in concept. There is no main server that owns the truth. Devices share folders with each other.

For personal knowledge management, Syncthing is usually easier to reason about if you already like local files. For team file sharing, Seafile or Nextcloud may be more appropriate.

The question is not which tool is better. The question is whether you want local-first peer sync or a central file platform.

Android in 2026

The Android story needs caution.

The original Syncthing Android app was discontinued after the December 2024 release. Community forks and alternative approaches may exist, but Android should not be treated as the most stable part of a Syncthing knowledge system.

This does not mean Syncthing is useless on Android. It means you should design mobile sync as a convenience layer, not the only reliable copy.

For Android, prefer narrower use cases:

Scan inbox
Camera import folder
Quick notes folder
Read-only reference folder
Selected documents

Avoid making your phone responsible for the only complete sync path of your knowledge archive. Phones are battery-managed, storage-limited, and increasingly restrictive about background file access.

The boring recommendation is best: keep the authoritative knowledge set on desktop, laptop, and server. Let the phone participate selectively.

Versioning Strategy

Syncthing supports file versioning, and you should usually enable it on at least one stable device.

For a knowledge-management setup, the home server or NAS is often the best place for versioning. It is always on, has more storage, and is easier to include in backup jobs.

Common versioning approaches include:

Trash can versioning — moved files go to a trash folder
Simple versioning — keeps a fixed number of old versions
Staggered versioning — keeps more versions near the present, fewer as files age
External versioning — delegates versioning to an external tool

For most personal knowledge folders, staggered versioning is a reasonable starting point. It keeps more versions near the present and fewer versions as files get older.

Versioning is not free. It consumes storage and needs occasional review. But storage is cheaper than losing a year of notes.

Snapshots and Backup

If your Syncthing target is a NAS or Linux server, add snapshots.

Good options include:

ZFS snapshots
Btrfs snapshots
LVM snapshots
restic
BorgBackup
Kopia
Filesystem-level backup tools from your NAS platform

A strong setup might be:

Syncthing syncs files to the server.
The server keeps Syncthing file versions.
The filesystem keeps snapshots.
A backup tool copies encrypted backups off-device.

This gives you multiple recovery layers.

If you accidentally delete a note, Syncthing versioning may help. If a folder is corrupted, filesystem snapshots may help. If the server disk dies, off-device backup may help.

That is the difference between a sync setup and a recovery strategy.

Privacy and Trust

Syncthing is attractive because it lets you avoid placing all knowledge files into a third-party cloud account.

However, privacy is not automatic. You still need to think about device trust.

Every normal trusted device that participates in a folder can read that folder. If you sync your notes to an old laptop, that laptop is now part of your security boundary. If you sync documents to a VPS, that server matters too.

Syncthing also has an untrusted encrypted device feature, but treat it carefully. It can be useful when you want an encrypted replica on a device that should not see plaintext file content, but it is more advanced and should be tested before relying on it.

For most people, the simpler model is better:

Only sync sensitive folders to devices you actually trust.
Encrypt disks on laptops and servers.
Back up important data separately.
Do not sync private archives everywhere.

Self-hosted does not automatically mean secure. It means you own the responsibility.

Conflict Handling

Conflicts happen when different devices change the same file before synchronization converges.

For notes, this can happen if you edit the same markdown file on a laptop and desktop while one of them is offline. Syncthing will preserve conflict copies rather than silently choosing one version.

That is the right behavior, but it still leaves you with cleanup work.

To reduce conflicts:

Avoid editing the same note on two offline devices
Let sync finish before switching machines
Keep frequently edited inbox notes small
Avoid syncing application state files unnecessarily
Use ignore patterns for volatile files
Review conflict files periodically

Conflict files are not a sign that Syncthing is broken. They are a sign that two devices changed reality at the same time.

Ignore Patterns

Ignore patterns are important for knowledge folders.

They keep generated, temporary, or machine-specific files out of the sync set. This reduces conflicts, saves bandwidth, and avoids polluting other devices.

Common examples:

.DS_Store
Thumbs.db
*.tmp
*.swp
*.bak
.cache/
node_modules/
dist/
build/
__pycache__/

For a notes vault, consider whether plugin caches, workspace layout files, or generated indexes should really sync. Some should. Some should not.

The principle is simple: sync source material, not noise.

Suggested Syncthing Topologies

For one person, a star-like topology around a home server is often easiest.

desktop <-> home-server
laptop  <-> home-server
phone   -> home-server

The devices can still connect directly, but the server gives them a stable meeting point.

For a small household, keep personal folders separate. Do not create one shared mega-folder unless everyone understands the consequences.

For a small technical team, be careful. Syncthing can work for shared files among trusted peers, but it does not replace proper collaboration tools, permissions, review workflows, or version control.

The more people you add, the more attractive a server-centered platform like Nextcloud, Seafile, Git, or a document system becomes.

Practical Setup Example

Here is a reasonable self-hosted knowledge setup.

Folders:
  knowledge-notes
  knowledge-documents
  research-papers
  scans-inbox

Devices:
  desktop
  laptop
  home-server
  phone

Rules:
  desktop and laptop use send-receive for notes
  home-server uses send-receive with versioning enabled
  phone sends scans-inbox only
  large PDFs do not sync to phone
  home-server snapshots all synced folders
  home-server backup runs nightly

This setup is not fancy, but it is robust.

It supports local-first work, mobile capture, server-side recovery, and off-device backup. It avoids making the phone responsible for the whole archive. It also avoids confusing sync with backup.

Operational Checklist

Before trusting Syncthing with important knowledge files, check the basics.

Are all important folders synced to at least two real devices?
Is one device always on or frequently online?
Is file versioning enabled on at least one stable device?
Are filesystem snapshots enabled on the server or NAS?
Is there an off-device backup?
Are noisy files ignored?
Are mobile folders limited?
Have you tested restore?
Have you tested conflict behavior?
Do you understand which folders are send-only or receive-only?

The most important item is restore testing.

A backup strategy you have never restored from is a theory. A sync strategy you have never tested under conflict is also a theory.

Common Mistakes

The most common mistake is syncing too much.

People start with one useful folder, then add every document, every photo, every export, every cache, and every application directory. The system becomes noisy and hard to reason about.

The second mistake is treating receive-only folders as magical backup. They are useful, but they are not a full historical recovery system.

The third mistake is ignoring deletion behavior. If deletion syncs everywhere, then deletion is part of the design.

The fourth mistake is trusting mobile sync too much. Mobile operating systems are not friendly to long-running background file synchronization.

The fifth mistake is not using versioning. If the data matters, keep versions somewhere.

The sixth mistake is not having a real backup. Syncthing can help populate a backup target, but it should not be the only protection.

Where Syncthing Fits in a Self-Hosted Knowledge Stack

In a self-hosted knowledge system, Syncthing is infrastructure.

It sits below note-taking apps, wikis, search tools, static sites, scripts, and AI-assisted workflows. It moves the files those systems use.

A simple stack might look like this:

graph TB
    subgraph "Capture"
        C1["Phone scanner"]
        C2["Browser save"]
        C3["Quick notes"]
    end
    subgraph "Sync"
        S["Syncthing"]
    end
    subgraph "Storage"
        ST1["Desktop"]
        ST2["Laptop"]
        ST3["Home server / NAS"]
    end
    subgraph "Authoring"
        A1["Obsidian"]
        A2["VS Code"]
        A3["Vim"]
        A4["DokuWiki"]
        A5["Static site generator"]
    end
    subgraph "Search"
        SR1["ripgrep"]
        SR2["Desktop search"]
        SR3["Local index"]
        SR4["RAG pipeline"]
    end
    subgraph "Recovery"
        R1["Versioning"]
        R2["Snapshots"]
        R3["Encrypted backup"]
    end
    C1 --> S
    C2 --> S
    C3 --> S
    S --> ST1
    S --> ST2
    S --> ST3
    ST1 --> A1
    ST1 --> A2
    ST2 --> A3
    ST3 --> A4
    ST3 --> A5
    A1 --> SR1
    A2 --> SR2
    A3 --> SR3
    A4 --> SR4
    ST3 --> R1
    ST3 --> R2
    ST3 --> R3

This separation is healthy.

You can replace the editor without replacing the storage. You can replace the backup tool without replacing the note format. You can stop Syncthing and still have ordinary files.

That is the value of local-first knowledge infrastructure.

When Syncthing Is the Right Tool

Syncthing is a strong choice when the problem is private file synchronization between devices you control. It is less suitable when the real problem is collaboration, web access, permissions, or long-term recovery.

This distinction matters because many self-hosted file workflows fail from unclear expectations. A sync tool can move files very well, but it should not be asked to behave like a cloud suite, a backup archive, and a team collaboration platform at the same time.

Need	Syncthing fit	Better fit when Syncthing is not enough
Private sync between your own devices	Excellent	Usually none needed
Local-first markdown notes	Excellent	Obsidian Sync if you prefer managed sync
Personal document folders	Good	Nextcloud if you need browser access
Family file sharing	Possible, but awkward	Nextcloud
Team document collaboration	Limited	Nextcloud, Seafile, Google Drive, Microsoft 365
One-way server copy	Good	rsync may be simpler
Full backup and restore	Not enough alone	restic, BorgBackup, Kopia
Web file access	Weak	Nextcloud or Seafile
Calendar and contacts	Not supported	Nextcloud
Mobile photo backup	Possible with caution	Immich, Nextcloud, platform photo tools
Encrypted offsite backup	Not the main role	restic, BorgBackup, Kopia
Encrypted sync to untrusted device	Advanced use case	Syncthing untrusted devices, tested carefully

This is the main reason Syncthing remains useful. It does not try to become a full cloud platform. It works best when the job is clear: keep selected folders synchronized across trusted devices.

Final Recommendation

Use Syncthing for self-hosted knowledge management when you want private, local-first file sync across trusted devices.

Use it for notes, documents, research folders, scans, and personal knowledge archives. Pair it with markdown, plain files, local search, and self-hosted services. Keep the structure simple enough that you can understand it without a dashboard.

But do not confuse sync with backup.

The best setup is not Syncthing alone. The best setup is Syncthing plus file versioning, filesystem snapshots, and real backup.

That combination gives you the thing self-hosted knowledge systems should aim for: local control, practical convenience, and recoverable data.

Circuit Breaker Pattern in Go: Stop Cascading Failures

Rost — Sat, 18 Jul 2026 07:49:30 +0000

A circuit breaker stops your Go service from hammering a failing dependency,
preventing cascading failures that consume goroutines, sockets, and memory until the entire system collapses.

The hard part is not the state machine. It is deciding where the breaker belongs, what counts as failure, how it interacts with timeouts and retries, and what your service should do when the circuit is open.

In Go, the circuit breaker pattern is especially useful around outbound calls: HTTP APIs, payment gateways, search services, email providers, LLM gateways, internal microservices, and other dependencies that can become slow, overloaded, or partially unavailable. Used well, a circuit breaker reduces cascading failures. Used badly, it becomes another obscure failure mode.

What Problem Does a Circuit Breaker Solve?

Distributed systems rarely fail cleanly.

A dependency might not be fully down. It might be:

returning 500 errors
returning 429 rate limit responses
accepting TCP connections but never replying
responding in 30 seconds instead of 300 milliseconds
failing only for some requests
overloaded because every client is retrying at once

The worst case is often not a hard failure. It is a slow dependency.

Slow calls consume goroutines, sockets, database connections, memory, and worker capacity. If your service keeps waiting on a dependency that is already unhealthy, your service can become unhealthy too.

A circuit breaker prevents that by failing fast after the dependency crosses a failure threshold.

Instead of doing this forever:

request -> call dependency -> wait -> timeout -> retry -> wait -> fail

the service eventually does this:

request -> circuit open -> return fallback or error immediately

That fast failure is not always pleasant, but it is predictable. Predictable failure is easier to operate than a slow collapse.

The Three Circuit Breaker States

Most circuit breakers use three states.

Closed

The circuit is closed during normal operation.

Requests are allowed through. The breaker records successes and failures. If the number or ratio of failures crosses a threshold, the breaker opens.

Closed does not mean "safe forever." It means "traffic is currently allowed."

Open

The circuit is open when the dependency is considered unhealthy.

Requests are rejected immediately. The service should return a fallback, cached response, degraded response, or a clear upstream error.

Open does not fix the dependency. It gives the dependency time to recover and protects the caller from wasting resources.

Half-Open

After a cool-down period, the breaker enters a half-open state.

Only a limited number of trial requests are allowed through. If they succeed, the breaker closes. If they fail, the breaker opens again.

Half-open is important because it avoids two bad extremes:

never trying the dependency again
sending full traffic back too quickly

The state transitions look like this:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: Failure threshold reached
    Open --> HalfOpen: Timeout elapsed
    HalfOpen --> Closed: Trial succeeds
    HalfOpen --> Open: Trial fails

Circuit Breaker vs Timeout vs Retry

A common mistake is treating circuit breakers, retries, and timeouts as interchangeable. They are related, but they solve different problems.

Timeout

A timeout limits how long one operation can run.

In Go, this usually means passing a context.Context with a deadline or timeout into the outbound call.

A timeout answers this question:

How long am I willing to wait for this one call?

Retry

A retry repeats an operation when the failure might be temporary.

Retries are useful for short network glitches, temporary 503 responses, connection resets, and other transient failures.

A retry answers this question:

Should I try this call again?

Circuit Breaker

A circuit breaker stops calls when the dependency is probably unhealthy.

It answers this question:

Should I call this dependency at all right now?

Rate Limiter

A rate limiter controls how much traffic is allowed over time.

It answers this question:

How much traffic should this caller send?

Bulkhead

A bulkhead isolates resources so one dependency cannot consume everything.

It answers this question:

How much of my service can this dependency damage?

These patterns are strongest when used together. A circuit breaker without timeouts is weak. Retries without jitter can create retry storms. A fallback without metrics can hide an outage.

When to Use a Circuit Breaker in Go

Use a circuit breaker when your service calls a dependency that can fail independently from your service.

Good candidates include:

external HTTP APIs
payment processors
email and SMS providers
search services
recommendation services
LLM inference gateways
internal microservice endpoints
third-party SaaS APIs
slow or overloaded read-side services

Circuit breakers are especially useful when the caller can degrade gracefully.

For example:

return cached product data
skip a recommendation block
mark a payment provider as temporarily unavailable
queue work for later
return a partial response
fail fast with a clear temporary error

The important question is not "can this call fail?" Everything can fail. The better question is:

If this dependency is failing, should we continue sending full traffic to it?

If the answer is no, a circuit breaker may help.

When Not to Use a Circuit Breaker

Do not add a circuit breaker to every function just because the pattern sounds responsible.

A circuit breaker is usually not useful for:

local in-process function calls
simple CRUD inside a monolith
validation logic
deterministic business rules
CPU-only local operations
code paths where no useful fallback exists
write operations that are not idempotent
dependencies already protected by a stronger workflow layer

A circuit breaker also does not replace basic hygiene:

set timeouts
propagate context
use connection pools correctly
handle errors explicitly
make retries safe
observe failure rates

A bad circuit breaker can make a system harder to reason about. It can hide the real problem, reject traffic too aggressively, or create confusing behavior during recovery.

The slightly opinionated rule is simple:

Add circuit breakers at dependency boundaries, not everywhere.

Choosing a Go Circuit Breaker Library

You can implement a basic circuit breaker yourself, but most production Go services should use a library.

The most common simple choice is sony/gobreaker.

It gives you:

closed, open, and half-open states
configurable failure thresholds
configurable open-state timeout
state change callbacks
request counters
generic support in v2
a small API surface

For larger resilience pipelines, you may also look at libraries that compose multiple policies, such as retry, timeout, fallback, rate limiting, bulkhead isolation, and circuit breaking. That can be useful when you want a single resilience layer around an operation.

For many Go services, though, gobreaker is enough.

Go Circuit Breaker Packages Compared

Go does not include a built-in circuit breaker in the standard library. In practice, you usually choose between a small circuit breaker library, a larger resilience framework, or an older Hystrix-style package.

For most new Go services, the decision is simple:

use sony/gobreaker if you want a small, focused circuit breaker
use failsafe-go if you want circuit breakers composed with retries, timeouts, fallbacks, bulkheads, rate limits, and other resilience policies
avoid starting new projects on hystrix-go unless you already have legacy code using it

Package	Best for	Strengths	Tradeoffs
`sony/gobreaker/v2`	Simple circuit breakers around HTTP/RPC clients	Small API, generic v2 support, clear state model, easy to wrap dependency clients	Only solves circuit breaking; retries, timeouts, and fallbacks must be composed separately
`failsafe-go`	Full resilience policy composition	Retry, fallback, circuit breaker, timeout, bulkhead, rate limiter, cache, hedge, adaptive limiter, and adaptive throttler policies	More concepts to learn; heavier than needed if you only want a basic breaker
`afex/hystrix-go`	Legacy Hystrix-style systems	Familiar Hystrix concepts, command-style execution, historical usage	Older design; not the best default for new Go services
`go-kit/kit/circuitbreaker`	Go kit endpoint-based services	Fits Go kit middleware style and endpoint architecture	Mostly useful if your service already uses Go kit
`cep21/circuit`	Hystrix-like circuit breaker behavior	More featureful Hystrix-style approach	Less common as the simple default; may be more than needed for small services

My default recommendation is boring on purpose: start with sony/gobreaker/v2 when you only need a circuit breaker. Reach for failsafe-go when you want to express a complete resilience policy in one place.

That split keeps the architecture clean. A small service client does not need a full resilience framework just to stop calling a failing dependency. But a gateway, aggregator, API client SDK, or high-traffic integration layer may benefit from composed policies.

Installing gobreaker

Use the v2 package for new code:

go get github.com/sony/gobreaker/v2

Then import it:

import "github.com/sony/gobreaker/v2"

A Basic Circuit Breaker in Go

Here is a small example around an HTTP call.

package main

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "time"

    "github.com/sony/gobreaker/v2"
)

var ErrTemporaryUnavailable = errors.New("dependency temporarily unavailable")

type UserClient struct {
    baseURL string
    http    *http.Client
    cb      *gobreaker.CircuitBreaker[[]byte]
}

func NewUserClient(baseURL string) *UserClient {
    settings := gobreaker.Settings{
        Name:        "user-service",
        MaxRequests: 3,
        Interval:    30 * time.Second,
        Timeout:     10 * time.Second,
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            return counts.ConsecutiveFailures >= 5
        },
        OnStateChange: func(name string, from gobreaker.State, to gobreaker.State) {
            fmt.Printf("circuit breaker %s changed from %s to %s\n", name, from, to)
        },
    }

    return &UserClient{
        baseURL: baseURL,
        http: &http.Client{
            Timeout: 3 * time.Second,
        },
        cb: gobreaker.NewCircuitBreaker[[]byte](settings),
    }
}

func (c *UserClient) GetUser(ctx context.Context, userID string) ([]byte, error) {
    result, err := c.cb.Execute(func() ([]byte, error) {
        req, err := http.NewRequestWithContext(
            ctx,
            http.MethodGet,
            c.baseURL+"/users/"+userID,
            nil,
        )
        if err != nil {
            return nil, err
        }

        resp, err := c.http.Do(req)
        if err != nil {
            return nil, err
        }
        defer resp.Body.Close()

        if resp.StatusCode >= 500 {
            return nil, fmt.Errorf("user service returned %d", resp.StatusCode)
        }

        if resp.StatusCode == http.StatusNotFound {
            return nil, fmt.Errorf("user not found")
        }

        if resp.StatusCode >= 400 {
            return nil, fmt.Errorf("user service client error: %d", resp.StatusCode)
        }

        return io.ReadAll(resp.Body)
    })

    if errors.Is(err, gobreaker.ErrOpenState) {
        return nil, ErrTemporaryUnavailable
    }

    if errors.Is(err, gobreaker.ErrTooManyRequests) {
        return nil, ErrTemporaryUnavailable
    }

    return result, err
}

This is not a complete production client, but it shows the shape:

the breaker wraps the outbound call
the HTTP request receives a context
the HTTP client has a timeout
server-side failures count as breaker failures
open circuit errors are mapped into an application error

Configuring gobreaker Settings

The key settings are worth understanding.

Name

Name identifies the breaker.

Use a stable, specific name:

payment-api
search-service
llm-gateway
user-service

Avoid vague names like:

http-client
external-call
default

You will want this name in logs and metrics.

MaxRequests

MaxRequests controls how many requests are allowed while the breaker is half-open.

A small number is usually safer. The purpose of half-open is to test recovery, not to send full traffic immediately.

Interval

Interval controls when internal counts are cleared while the breaker is closed.

If it is zero, counts are not cleared automatically. A non-zero interval gives the breaker a rolling-ish memory window, although it is not the same as a full sliding window implementation.

Timeout

Timeout controls how long the breaker stays open before moving to half-open.

If the timeout is too short, your service will keep probing a dependency that has not recovered. If it is too long, recovery will be delayed.

Start with something conservative, such as 10 to 30 seconds, then tune from production metrics.

ReadyToTrip

ReadyToTrip decides when the breaker should open.

A simple rule is consecutive failures:

ReadyToTrip: func(counts gobreaker.Counts) bool {
    return counts.ConsecutiveFailures >= 5
}

That is easy to reason about, but it may not be right for high-volume services.

Another option is failure ratio after a minimum number of requests:

ReadyToTrip: func(counts gobreaker.Counts) bool {
    total := counts.Requests
    failures := counts.TotalFailures

    if total < 20 {
        return false
    }

    return float64(failures)/float64(total) >= 0.5
}

This avoids opening the circuit after a tiny sample size.

OnStateChange

OnStateChange is where you should emit logs or metrics.

At minimum, record:

breaker name
old state
new state
timestamp

For production systems, expose breaker state as a metric. Logs are useful for debugging, but metrics are better for alerting and dashboards.

IsSuccessful

IsSuccessful lets you decide which errors count as failures.

This is important.

Not every error should open the breaker. For example, a 404 Not Found from a user service may be a valid business result. A 400 Bad Request might be the caller's fault, not the dependency's fault.

A 503 Service Unavailable, timeout, connection reset, or 429 Too Many Requests may be a real dependency health signal.

Be careful here. Counting the wrong errors is one of the easiest ways to build a noisy circuit breaker.

What Should Count as Failure?

This is where engineering judgement matters.

Usually count these as failures:

network timeouts
connection refused
connection reset
HTTP 500
HTTP 502
HTTP 503
HTTP 504
repeated 429 responses
malformed responses from the dependency
context deadline exceeded during the outbound call

Usually do not count these as dependency failures:

validation errors
local serialization errors
expected 404 responses
caller-side authorization failures
business rule rejections
user input errors

The breaker should represent dependency health, not general application failure.

Circuit Breakers and context.Context

In Go, circuit breakers should not replace context.Context.

A circuit breaker decides whether to attempt a call. A context controls how long that call may run and whether it should stop when the caller is gone.

A good outbound call should usually have both:

ctx, cancel := context.WithTimeout(parentCtx, 2*time.Second)
defer cancel()

data, err := client.GetUser(ctx, userID)

The context should flow through the call chain:

incoming request context
-> service method
-> client method
-> HTTP request
-> dependency

Avoid creating detached background contexts inside request-scoped code. If the user request is canceled, the downstream work should usually stop too.

The calm rule is:

The breaker protects the system. The context protects the request.

You normally need both.

Circuit Breakers and Retries

Retries and circuit breakers can work well together, but the order matters.

The safest default is:

timeout per attempt
retry with backoff and jitter
circuit breaker around the dependency call

But there is no universal answer. Think about what you want to count.

If each retry attempt passes through the breaker, one user request can contribute multiple failures. That may open the breaker faster, which can be good or bad.

If the breaker wraps the whole retry operation, the breaker sees one final success or failure per user request. That is calmer, but it may hide the number of failed attempts.

For many application services, this shape is reasonable:

user request
-> circuit breaker
   -> retry policy
      -> one HTTP attempt with timeout

That means the breaker tracks whether the dependency operation ultimately worked for the caller.

For lower-level clients, this shape can also make sense:

user request
-> retry policy
   -> circuit breaker
      -> one HTTP attempt with timeout

That means the breaker protects each attempt.

The more important rule is this:

Do not retry blindly.

Use:

a small maximum retry count
exponential backoff
jitter
per-attempt timeouts
an overall request deadline
idempotency for writes
metrics for retry attempts

Without those, retries can turn a small outage into a larger one. For a deeper treatment of retry safety, see Idempotency in Distributed Systems That Actually Works.

Circuit Breakers and Idempotency

Circuit breakers often appear next to retries, and retries raise the question of idempotency.

For read operations, retrying is usually safe.

For write operations, retrying can be dangerous.

Consider this payment call:

POST /charge

If the request times out, did the payment fail? Maybe. Did it succeed but the response was lost? Also maybe.

If you retry without an idempotency key, you might charge twice.

For write operations, use one or more of these:

idempotency keys
request IDs
operation IDs
unique constraints
transactional outbox
workflow orchestration
explicit reconciliation

A circuit breaker can stop you from continuing to call a failing payment provider, but it cannot make unsafe retries safe.

Circuit Breakers and Fallbacks

When the circuit is open, your service needs a plan.

Possible fallback strategies include:

return cached data
return stale data with a warning
omit a non-critical section
queue work for later
switch to another provider
return a temporary error
show degraded functionality
fail the request quickly

A fallback should be honest.

For example, this is usually good:

{
  "status": "temporary_unavailable",
  "message": "Recommendations are temporarily unavailable"
}

This is risky:

{
  "recommendations": []
}

An empty list may look like a valid result. It can hide an outage, confuse users, and make debugging harder.

Silent fallbacks are tempting. They are also dangerous.

Circuit Breakers and Observability

A circuit breaker without observability is mostly a surprise generator.

Track at least these metrics:

current breaker state
state changes
calls allowed
calls rejected
successes
failures
timeouts
fallback responses
retry attempts
downstream latency
downstream status codes

Useful labels include:

breaker name
dependency name
operation name
status class
error category

Avoid high-cardinality labels such as user ID, full URL, request ID, or raw error messages.

You should be able to answer these questions from dashboards:

Which circuit breakers are open right now?
How often do they open?
Which dependency caused the opening?
Are users seeing fallback responses?
Did latency improve after the breaker opened?
Did retry volume spike before the breaker opened?
Did the dependency recover?

If you cannot observe the breaker, you cannot tune it. For structured logging that pairs well with metrics, see Structured Logging in Go with slog.

A More Production-Friendly HTTP Client Shape

For real services, avoid scattering circuit breaker logic across handlers.

Create a small client package around the dependency.

Example structure:

internal/
  userservice/
    client.go
    errors.go
    metrics.go

The handler should not know the details of gobreaker. It should depend on a domain-level client method:

type UserService interface {
    GetUser(ctx context.Context, userID string) (*User, error)
}

Then the implementation can contain:

HTTP request creation
context propagation
breaker execution
status code handling
response decoding
metrics
error mapping

This keeps the resilience policy close to the dependency boundary. For more on error classification at boundaries, see Go Error Handling Architecture: Boundaries and Patterns.

Where Circuit Breakers Fit in Application Architecture

The circuit breaker pattern belongs at integration boundaries.

In a Go application, that usually means:

graph LR
    A[Handler] --> B[Application Service]
    B --> C[Dependency Client]
    C --> D[Circuit Breaker]
    D --> E[HTTP / RPC / DB / Queue]

Keep the breaker out of business logic when possible.

The business layer should understand domain errors like:

payment provider unavailable
recommendations unavailable
profile service timeout

It should not need to understand gobreaker states.

This separation keeps the architecture clean:

transport concerns stay in clients
resilience policy stays near dependencies
domain logic stays readable
handlers stay thin
tests are easier to write

Common Mistakes

Mistake 1: No Timeout

A circuit breaker does not magically stop slow calls unless the calls return.

If the outbound operation can hang forever, the breaker may not see a failure quickly enough.

Always use timeouts.

Mistake 2: One Global Breaker for Everything

Do not use one breaker for all dependencies.

A failing email provider should not open the circuit for your payment provider. A slow search endpoint should not block user profile calls.

Use separate breakers for separate dependency operations when their failure modes differ.

Mistake 3: Counting Caller Errors as Dependency Failures

If your service sends bad input and receives 400 Bad Request, that is usually not a downstream outage.

Do not train the breaker on your own bugs.

Mistake 4: Retrying Non-Idempotent Writes

Retries are not free. They can duplicate writes, payments, messages, or side effects.

Make writes idempotent before retrying them.

Mistake 5: Hiding Outages Behind Fallbacks

Fallbacks should degrade gracefully, not falsify reality.

If a dependency is down, your metrics and logs should make that obvious.

Mistake 6: Tuning Without Production Data

Thresholds copied from examples are only starting points.

Tune based on:

request volume
normal error rate
dependency latency
user impact
recovery time
fallback quality

Mistake 7: Using Circuit Breakers Instead of Capacity Management

A circuit breaker is not a substitute for:

load shedding
rate limiting
queue limits
autoscaling
database tuning
connection pool limits
upstream quotas

It is one part of a resilience strategy.

Practical Defaults

For a typical Go service calling an internal HTTP dependency, a reasonable starting point might be:

HTTP client timeout: 2 to 5 seconds
per-request context timeout: based on caller SLA
breaker failure rule: 5 consecutive failures or 50 percent failure after 20 requests
open timeout: 10 to 30 seconds
half-open requests: 1 to 5
retry count: 1 to 3 attempts
retry backoff: exponential with jitter

These are not universal values. They are safe-ish starting points.

For user-facing APIs, keep total latency budgets tight. For background jobs, you may tolerate longer waits. For payment providers, be much more careful with retries and idempotency.

Circuit Breaker Checklist

Before adding a circuit breaker, answer these questions:

What dependency is being protected?
What operation is being protected?
What errors count as dependency failure?
What errors should be ignored by the breaker?
What timeout applies to each call?
Are retries allowed?
Are writes idempotent?
What happens when the circuit is open?
Is there a fallback?
Is the fallback visible in metrics?
Who gets alerted if the circuit keeps opening?
How will the breaker be tuned after deployment?

If you cannot answer these, adding a breaker may create more confusion than resilience.

Testing Circuit Breakers in Go

Test behavior, not the internal state machine of the library.

Useful tests include:

dependency succeeds and response is returned
dependency fails repeatedly and circuit opens
open circuit returns a temporary error
client-side validation errors do not trip the breaker
context timeout is respected
fallback response is returned when expected
metrics are emitted on state changes

Use fake HTTP servers for integration-style tests:

server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    http.Error(w, "unavailable", http.StatusServiceUnavailable)
}))
defer server.Close()

For unit tests, hide the dependency behind an interface and inject a fake implementation.

Keep tests deterministic. Avoid sleeping for long real durations. Configure short breaker timeouts in tests. For more on testing concurrent Go code with fake time and isolated bubbles, see Testing Concurrent Go Code with testing/synctest.

Should You Build Your Own Circuit Breaker?

Building a small circuit breaker is a good learning exercise. It helps you understand the state machine.

For production code, prefer a maintained library unless your needs are very specific.

A production breaker needs to handle:

concurrency
state transitions
counters
half-open probes
callbacks
custom failure classification
race-free behavior
predictable error handling

That is not impossible, but it is easy to get subtly wrong.

The boring library is usually the better choice.

Conclusion

The circuit breaker pattern is not magic reliability dust.

In Go, it works best when it is part of a small, explicit resilience stack:

context timeout
+ retry with backoff and jitter
+ circuit breaker
+ fallback
+ metrics

The pattern is most useful at dependency boundaries, especially around remote services that can become slow or partially unavailable.

Use it to stop cascading failures. Use it to fail fast when a dependency is clearly unhealthy. Use it to give overloaded systems room to recover.

But do not use it as an excuse to ignore timeouts, idempotency, observability, or clean architecture.

A good circuit breaker makes failure clearer and cheaper. A bad one just makes failure more mysterious.

References

Go Microservices for AI/ML Orchestration — broader orchestration context where circuit breakers fit
Saga Pattern in Distributed Transactions — distributed transaction patterns that pair with circuit breakers
Idempotency in Distributed Systems — retry safety and idempotent operations
Transactional Outbox Pattern in Go — reliable event delivery alongside resilience patterns
Go Error Handling Architecture — error classification at dependency boundaries
Testing Concurrent Go Code with synctest — testing async behavior with circuit breakers
Structured Logging in Go with slog — observability alongside circuit breakers
github.com/sony/gobreaker/v2 — official gobreaker v2 package
Go Context Cancellation and Timeouts — context patterns that pair with circuit breakers

Podman Quadlet vs Docker Compose for Linux Services

Rost — Thu, 16 Jul 2026 10:13:30 +0000

Docker Compose and Podman Quadlet solve overlapping problems but come from different design centers, and choosing between them depends on whether you think in application stacks or Linux services.

The distinction matters for anyone running containers on a Linux host beyond a single afternoon of experimentation. Compose describes services, networks, and volumes in a YAML file and starts them with docker compose up. Quadlet describes containers in systemd-style unit files and lets the system service manager own the lifecycle.

Both approaches work for self-hosted services, internal tools, and small servers. The right choice comes down to your operational model, team familiarity, and whether you prefer a developer-friendly stack format or a systemd-native service model. This comparison covers the practical differences: file formats, lifecycle ownership, rootless containers, logging, updates, networking, security, and migration paths. It is part of Developer Tools: The Complete Guide to Modern Development Workflows.

Quick Recommendation

Use Docker Compose when:

You want the fastest multi-container workflow.
You already use Docker Engine.
You share stacks with developers.
You need a familiar compose.yaml for local development.
You deploy small services with Docker on one host.
You use existing Compose examples from projects.

Use Podman Quadlet when:

You want systemd-native container services.
You prefer rootless containers.
You do not want a central Docker daemon.
You run long-lived services on a Linux host.
You want systemctl, journalctl, timers, dependencies, and auto-start.
You are building a self-hosted or homelab server around systemd.
You want containers to fit into the normal Linux service model.

The practical rule:

Docker Compose is better for application stacks.
Podman Quadlet is better for Linux services.

That is not a law. It is a useful default.

Comparison Table

Area	Docker Compose	Podman Quadlet
Primary model	Multi-container application	systemd-managed container service
File format	YAML	systemd-like unit files
Runtime	Docker Engine	Podman
Daemon	Uses Docker daemon	Daemonless Podman model
Service manager	Compose manages stack lifecycle	systemd manages lifecycle
Best fit	Dev stacks, app bundles, simple deployments	Long-running Linux services
Rootless support	Possible, but not the default mental model	Strong fit
Logs	`docker compose logs`	`journalctl` and `podman logs`
Startup on boot	Usually via systemd wrapper or restart policy	Native systemd unit
Updates	`docker compose pull && docker compose up -d`	Podman auto-update or systemd workflows
Portability	Very high across Docker environments	Best on Linux with systemd
Learning curve	Easier for most developers	Easier for systemd users
Ecosystem examples	Huge	Smaller, but growing

Neither one is Kubernetes. Most small services do not need a cluster. They need a boring, understandable way to start, stop, update, log, and recover.

What Docker Compose Is Good At

Docker Compose is a tool for defining and running multi-container applications. A typical Compose file describes services, images, build contexts, ports, volumes, networks, environment variables, health checks, dependencies, and profiles.

services:
  web:
    image: nginx:stable
    ports:
      - "8080:80"
    restart: unless-stopped
    volumes:
      - ./html:/usr/share/nginx/html:ro

  redis:
    image: redis:7
    restart: unless-stopped

Run it:

docker compose up -d

Check status:

docker compose ps

Read logs:

docker compose logs -f

Stop it:

docker compose down

Compose is direct and productive. It is especially good when the unit of thought is "this application has several containers." For a comprehensive reference of Compose commands and patterns, see the Docker Compose Cheatsheet. For Docker commands beyond Compose — images, volumes, networks, and cleanup — see the Docker Cheatsheet.

What Podman Quadlet Is Good At

Podman Quadlet is a way to define Podman containers using systemd-style files. Instead of writing a full generated systemd service by hand, you write a declarative file:

[Unit]
Description=Example web container
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/library/nginx:stable
PublishPort=8080:80
Volume=/opt/example/html:/usr/share/nginx/html:ro

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

Save it as /etc/containers/systemd/example.container, then reload systemd:

sudo systemctl daemon-reload

Start it:

sudo systemctl enable --now example.service

Check it:

systemctl status example.service
journalctl -u example.service -f

The core appeal of Quadlet: the container becomes a normal Linux service.

The Philosophical Difference

Docker Compose Thinks in Stacks

Compose asks: What services make up this application?

A Compose project usually lives near application code:

myapp/
  compose.yaml
  .env
  app/
  db/

You start the project as a unit:

docker compose up -d

You update the project as a unit:

docker compose pull
docker compose up -d

This is simple, visible, and portable.

Podman Quadlet Thinks in Services

Quadlet asks: What containers should this Linux host run as services?

Quadlet files live in systemd-related container paths:

/etc/containers/systemd/
~/.config/containers/systemd/

You manage generated services with systemd:

systemctl status myapp.service
systemctl restart myapp.service
journalctl -u myapp.service -f

This feels more native on a Linux server. For general systemd service patterns, see Run any Executable as a Service in Linux.

The Important Difference: Who Owns Lifecycle?

With Docker Compose, Compose owns the application lifecycle. With Quadlet, systemd owns the service lifecycle.

This affects boot behavior, shutdown behavior, restart policy, dependency ordering, logs, health visibility, user services, updates, integration with timers, and integration with other host services.

If you already use systemd to manage everything else on the host, Quadlet fits neatly. If you think mainly in terms of application stacks, Compose is usually more comfortable.

Docker Compose Under systemd vs Quadlet

You can run Docker Compose as a systemd service. That is often a good pattern. Example systemd unit:

[Unit]
Description=MyApp Docker Compose stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecReload=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

This works well. But it is still a wrapper around Compose. systemd starts the Compose command, while Docker and Compose handle containers behind it.

With Quadlet, the unit generation is designed for Podman and systemd directly. You write container-oriented unit files, and systemd manages the generated services.

The distinction is subtle but important:

Docker Compose under systemd:
  systemd manages a Compose command.

Podman Quadlet:
  systemd manages generated container services.

For a detailed walkthrough of this pattern, see Run Docker Compose as a Linux Service with systemd.

File Format Comparison

Docker Compose YAML

Compose uses YAML. It is compact, popular, and easy to share. It is also indentation-sensitive and can grow messy when a stack becomes large.

services:
  app:
    image: ghcr.io/example/app:1.0.0
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      APP_ENV: production
    volumes:
      - app-data:/data

volumes:
  app-data:

Quadlet Unit Files

Quadlet uses systemd-like files. They are more verbose when you have many services, but readable if you already understand systemd.

[Unit]
Description=Example app container
After=network-online.target
Wants=network-online.target

[Container]
Image=ghcr.io/example/app:1.0.0
PublishPort=8080:8080
Environment=APP_ENV=production
Volume=app-data.volume:/data

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

And a volume file:

[Volume]
VolumeName=app-data

Saved as app.container and app-data.volume in the appropriate systemd container directory.

Mapping Docker Compose Concepts to Quadlet

Compose concept	Quadlet equivalent
`services`	`.container` files or `.pod` plus `.container`
`volumes`	`.volume` files or bind mounts
`networks`	`.network` files
`ports`	`PublishPort=`
`environment`	`Environment=` or `EnvironmentFile=`
`restart`	`[Service] Restart=`
`depends_on`	systemd `After=`, `Wants=`, `Requires=`
`healthcheck`	Podman healthcheck options
`profiles`	systemd enablement and separate units
`docker compose logs`	`journalctl -u service` and `podman logs`
`docker compose up -d`	`systemctl start service`
`docker compose down`	`systemctl stop service`
project directory	systemd container unit directory

The migration is conceptually simple but not mechanical. Compose describes a stack. Quadlet describes services.

Rootless Containers

Rootless containers are one of the strongest reasons to look at Podman and Quadlet.

With Docker, many users add themselves to the docker group. That is convenient, but access to the Docker daemon is effectively powerful host access. On a personal workstation, that may be acceptable. On shared servers, it deserves more caution.

Podman was designed with rootless usage as a first-class workflow. A rootless Quadlet lives under the user's config directory:

~/.config/containers/systemd/whoami.container

Then manage it with user systemd:

systemctl --user daemon-reload
systemctl --user enable --now whoami.service
systemctl --user status whoami.service
journalctl --user -u whoami.service -f

To allow the user service to keep running after logout:

sudo loginctl enable-linger "$USER"

This is a clean model for user-owned services.

Rootless Comparison

Area	Docker Compose	Podman Quadlet
Default common setup	Rootful Docker daemon	Rootless-friendly Podman
User service model	Possible, but less native	Native with `systemctl --user`
Daemon access risk	Docker socket is powerful	No central root daemon by default
Low port binding	Simple as rootful Docker	Needs extra setup when rootless
Host integration	Very common	More Linux-native
Shared server fit	Needs care	Strong fit

Rootless is not magic. It has tradeoffs around networking, privileged behavior, and low ports. But for long-running user-owned services, Quadlet is a very elegant model.

Startup and Boot Behavior

Docker Compose

Compose by itself does not create a boot service. You usually rely on Docker restart policies, a systemd wrapper around docker compose up -d, a deployment script, or a higher-level tool.

Example Compose restart policy:

services:
  app:
    image: example/app:stable
    restart: unless-stopped

A systemd wrapper gives you a host-level service. That is good, but it is still an extra wrapper. See Run Docker Compose as a Linux Service with systemd for the full pattern.

Podman Quadlet

Quadlet is already systemd-oriented. Enable on boot:

sudo systemctl enable myapp.service

For rootless:

systemctl --user enable myapp.service
sudo loginctl enable-linger "$USER"

Boot behavior is not an add-on. It is the model.

Restart Behavior

Compose commonly uses restart: unless-stopped in YAML. Quadlet commonly uses systemd restart behavior:

[Service]
Restart=always

Or:

[Service]
Restart=on-failure

This moves restart logic into the service manager. The preference: use Compose/Docker restart policies for Compose stacks, use systemd restart policies for Quadlet, and do not stack too many supervisors. Keep one clear owner of restart behavior.

Logging Comparison

Docker Compose Logs

docker compose logs -f
docker compose logs -f app

This is excellent for developers.

Quadlet Logs

Quadlet services use systemd logs:

journalctl -u app.service -f

For rootless units:

journalctl --user -u app.service -f

You can still use Podman logs:

podman logs -f container-name

For server operations, journalctl integration is a major advantage. Your containers fit into the same log workflow as other Linux services.

Updates

Updating Docker Compose

A common update flow:

cd /opt/myapp
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f

Easy to wrap in a script:

#!/usr/bin/env bash
set -euo pipefail

cd /opt/myapp

docker compose config --quiet
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f
docker compose ps

Updating Podman Quadlet

A simplified manual flow:

sudo podman pull ghcr.io/example/app:1.0.1
sudo systemctl restart app.service

Or for rootless:

podman pull ghcr.io/example/app:1.0.1
systemctl --user restart app.service

Podman can support auto-update workflows when containers are configured with the right labels and image policy. Quadlet's advantage is not that updates are always simpler. The advantage is that updates are service-manager-native.

Auto-Update Philosophy

Auto-updates are convenient. They are also a risk. For low-risk homelab services, automatic container updates can be fine. For databases, stateful apps, or business services, the preferred flow is:

Back up.
Pull.
Recreate or restart.
Check health.
Prune later.

Compose makes this explicit. Quadlet and Podman can make it systemd-native. Neither tool removes the need for a rollback plan.

Volumes and Persistent Data

Compose Volumes

Compose supports named volumes:

services:
  db:
    image: postgres:16
    volumes:
      - db-data:/var/lib/postgresql/data

volumes:
  db-data:

And bind mounts:

services:
  app:
    image: example/app
    volumes:
      - ./config:/config:ro
      - ./data:/data

Quadlet Volumes

Quadlet can use bind mounts:

[Container]
Volume=/opt/app/config:/config:ro
Volume=/opt/app/data:/data

Or a .volume file:

[Volume]
VolumeName=app-data

Then reference it:

[Container]
Volume=app-data.volume:/data

Compose is more compact for stack-level storage. Quadlet is more aligned with independently managed service units.

Secrets and Environment Files

Compose

Compose often uses env_file or environment in YAML. For a small private service, .env is common. For serious systems, treat .env as sensitive and keep it out of Git.

Quadlet

Quadlet can use:

[Container]
Environment=APP_ENV=production
EnvironmentFile=/opt/app/app.env

Restrict permissions:

chmod 600 /opt/app/app.env

Neither Compose nor Quadlet is a complete secret-management system by itself. Do not confuse "not in the command line" with "secure".

Networking

Compose Networking

Compose creates a default project network and gives services DNS names based on service names:

services:
  app:
    image: example/app
    depends_on:
      - db

  db:
    image: postgres:16

The app can usually reach the database at db. Multi-container app networking feels natural. This is one of Compose's strongest features.

Quadlet Networking

Quadlet can define networks separately with .network files or use Podman network options:

[Network]
NetworkName=appnet

Container file:

[Container]
Image=example/app:stable
Network=appnet.network

This is more explicit and systemd-like. For one or two containers, it is fine. For a large app stack, Compose is often easier to read.

Pods

Podman has a native pod concept. That matters if you like the Kubernetes mental model where multiple containers share a network namespace and lifecycle boundary. Quadlet supports .pod files:

[Pod]
PodName=myapp
PublishPort=8080:8080

A container can join that pod:

[Container]
Image=ghcr.io/example/app:stable
Pod=myapp.pod

Compose does not have the same pod model. It has services on networks. For most simple web apps, Compose networks are enough. For Podman users who like pod-style grouping, Quadlet is a better match.

Build Workflows

Compose is usually better when you build images as part of the local application workflow:

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"

Then:

docker compose up --build

This is extremely convenient for development. Quadlet is usually better when you run already-built images as services. Build images with Podman separately:

podman build -t localhost/myapp:latest .

Then reference the image:

[Container]
Image=localhost/myapp:latest

If your workflow is "edit code, rebuild, restart stack", Compose wins. If your workflow is "deploy a known image as a Linux service", Quadlet wins.

Portability

Compose files are widely shared. Many open-source projects provide a compose.yaml or docker-compose.yml. If a project says "run this with Docker Compose", you can usually start quickly with docker compose up -d. This is a major practical advantage.

Quadlet is portable across systems that have Podman, systemd, and compatible Quadlet support. That is a narrower target, but a very good one for modern Linux servers. Quadlet is not the best format for sharing an application with every possible developer. It is a good format for describing how a specific Linux host should run a containerized service.

Developer Experience

Docker Compose usually wins developer experience. More examples, more tutorials, more project templates, easier local builds, easy one-file stack, familiar docker compose up, and strong fit for dev dependencies. A developer can read this quickly:

services:
  db:
    image: postgres:16
  redis:
    image: redis:7
  app:
    build: .

Quadlet can do similar things, but it is more operations-shaped. For local development, I would rarely start with Quadlet unless the application itself is specifically about Podman or systemd.

Operations Experience

Quadlet often wins operations experience on a Linux host. Reasons:

Native systemctl
Native journalctl
Rootless user services
systemd dependencies
systemd timers
systemd restart behavior
No central Docker daemon
Better fit with host service management

A server admin can reason about:

systemctl status app.service
journalctl -u app.service -f
systemctl restart app.service

That is the normal Linux service workflow.

Security Model

Docker Compose Security Notes

Docker Compose usually talks to the Docker daemon. On a normal Linux Docker install, access to the Docker socket is powerful. A user who can control Docker can often mount host paths, run privileged containers, or otherwise gain broad host control. For installation options including rootless Docker on Ubuntu, see Install Docker on Ubuntu.

Practical advice:

Do not casually expose /var/run/docker.sock.
Treat the docker group as privileged.
Avoid privileged containers.
Avoid host mounts unless needed.
Keep secrets out of Git.
Use explicit image tags for important services.
Review published ports.

Quadlet Security Notes

Podman Quadlet pairs well with rootless containers and user systemd services. This can reduce risk, especially on shared hosts or personal servers where services should not require a root daemon.

Practical advice:

Prefer rootless services when they fit.
Use user units for user-owned services.
Use system units only when host-level privileges are needed.
Avoid unnecessary privileged containers.
Keep environment files locked down.
Think carefully about bind mounts.

Rootless does not mean risk-free. It means the default blast radius can be smaller.

Performance

For most web services, internal tools, and self-hosted apps, performance is not the deciding factor. The main differences are operational, not raw speed. Choose based on lifecycle model, security model, host integration, team familiarity, update process, networking needs, and debugging workflow.

If you are choosing between Compose and Quadlet because of performance alone, you are probably optimizing the wrong layer.

Failure Modes

Docker Compose Failure Modes

Common problems:

Docker daemon not running
Compose plugin missing
Wrong project directory
.env not loaded as expected
Old docker-compose binary used by accident
Containers not recreated after config changes
Orphan containers left after service rename
Volumes deleted with down -v
Docker logs filling the disk
Docker socket permission errors

Best fixes:

docker compose config
docker compose ps
docker compose logs -f
docker compose up -d --remove-orphans

Podman Quadlet Failure Modes

Common problems:

Unit file in the wrong directory
Forgot systemctl daemon-reload
Using system units when user units were intended
Forgot loginctl enable-linger for rootless services
Image pull takes longer than systemd startup timeout
cgroup v2 not available
SELinux labels or volume permissions
Service name differs from file expectations
Network or volume unit not enabled or referenced correctly

Best fixes:

systemctl status app.service
journalctl -u app.service -f
systemctl daemon-reload
podman ps -a
podman logs container-name

For rootless:

systemctl --user status app.service
journalctl --user -u app.service -f

Migration Example: Compose to Quadlet

Start with this Compose service:

services:
  whoami:
    image: traefik/whoami:v1.10
    restart: unless-stopped
    ports:
      - "8080:80"
    environment:
      WHOAMI_NAME: compose-demo

Run with Compose:

docker compose up -d

A rough Quadlet equivalent:

[Unit]
Description=Whoami demo container
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/traefik/whoami:v1.10
PublishPort=8080:80
Environment=WHOAMI_NAME=quadlet-demo

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

Save as /etc/containers/systemd/whoami.container, then:

sudo systemctl daemon-reload
sudo systemctl enable --now whoami.service
sudo systemctl status whoami.service

The example is easy because it is one container. A larger Compose stack with databases, networks, volumes, and build steps needs more careful translation.

Migration Checklist

Before moving from Compose to Quadlet, ask:

[ ] Is this stack really a set of long-running host services?
[ ] Are the images already built and published?
[ ] Do I need rootless services?
[ ] Do I want systemd dependencies and timers?
[ ] Are volumes and bind mounts clearly understood?
[ ] Are ports documented?
[ ] Are secrets handled outside Git?
[ ] Is there a backup and restore process?
[ ] Can I monitor logs through journalctl?
[ ] Do I have a rollback path?

If most answers are yes, Quadlet may be a good fit. If the stack is mostly for local development, Compose is probably still better.

When to Stay with Docker Compose

Stay with Compose when:

The project already ships a good Compose file.
You need the easiest onboarding path.
Developers run the same stack locally.
You build images during development.
You want one YAML file for services, volumes, and networks.
You want maximum tutorial and community compatibility.
Your current systemd wrapper works fine.

There is no prize for migrating a working Compose stack to Quadlet just because Quadlet is cleaner in theory. If Compose is boring and reliable for your use case, keep it.

When to Move to Podman Quadlet

Move to Quadlet when:

The stack is really a host service.
You want rootless service management.
You prefer Podman over Docker.
You want systemd to own lifecycle.
You want journalctl logs.
You want service dependencies.
You want user services that survive logout.
You want less Docker daemon exposure.
You are building a self-hosting host around systemd.

Quadlet is not "Compose but better." It is a different design center.

Recommended Patterns

Pattern 1: Local Development

Use Docker Compose. Fast, familiar, portable, easy to rebuild, easy for teams.

Pattern 2: Single-Host Self-Hosting

Use either. Choose Compose if the project already provides a Compose file. Choose Quadlet if you want systemd-native service management. Compose gives a better app bundle; Quadlet gives a better Linux service.

Pattern 3: User-Owned Rootless Service

Use Podman Quadlet. Rootless workflow, user-level service management, no central Docker daemon.

~/.config/containers/systemd/app.container
systemctl --user enable --now app.service
loginctl enable-linger

Pattern 4: Production-Like Single Server

Use Docker Compose with a disciplined systemd wrapper, or use Quadlet if your team is comfortable with Podman. Do not choose based on fashion. Choose based on who will operate it at 2 AM.

Pattern 5: Multi-Node Platform

Use neither as the final orchestration layer. Consider Kubernetes, Nomad, Swarm, or a managed platform. Compose and Quadlet are excellent single-host tools. They are not cluster schedulers.

Practical Decision Tree

Is this mainly for local development?
  yes:
    use Docker Compose
  no:
    continue

Does the project already provide a maintained compose.yaml?
  yes:
    use Docker Compose unless you have a strong reason to migrate
  no:
    continue

Do you want rootless long-running services managed by systemd?
  yes:
    use Podman Quadlet
  no:
    continue

Do you want the easiest multi-container app definition?
  yes:
    use Docker Compose
  no:
    continue

Do you want containers to behave like normal Linux services?
  yes:
    use Podman Quadlet
  no:
    use Docker Compose

Side-by-Side Commands

Task	Docker Compose	Podman Quadlet
Start	`docker compose up -d`	`systemctl start app.service`
Stop	`docker compose down`	`systemctl stop app.service`
Restart	`docker compose restart`	`systemctl restart app.service`
Apply changes	`docker compose up -d`	`systemctl daemon-reload && systemctl restart app.service`
Logs	`docker compose logs -f`	`journalctl -u app.service -f`
Status	`docker compose ps`	`systemctl status app.service`
Enable on boot	systemd wrapper or restart policy	`systemctl enable app.service`
Pull update	`docker compose pull`	`podman pull image`
Rootless service	possible	natural with `systemctl --user`

Common Misunderstandings

"Quadlet Replaces Docker Compose"

Not exactly. Quadlet replaces some Compose use cases, especially long-running Linux services. It does not replace Compose as the easiest application-stack format for developers.

"Docker Compose Is Not Production Ready"

Too broad. Compose can be perfectly reasonable for small production systems if you understand backups, updates, logging, restart behavior, and host security. The problem is not Compose. The problem is pretending a single-host Compose deployment has the same properties as a cluster orchestrator.

"Podman Is Just Docker Without the Daemon"

Too simple. Podman has Docker-compatible commands, but its design center is different: daemonless operation, rootless workflows, pods, and Linux integration.

"Rootless Means Secure"

No. Rootless reduces some risks. It does not make bad images, exposed secrets, unsafe bind mounts, or vulnerable apps safe.

"systemd Is Too Heavy for Containers"

systemd is already the service manager on most mainstream Linux servers. Using it to manage long-running containers is not strange. It is often the boring and correct thing to do.

Final Recommendation

Use Docker Compose when the application stack is the main thing. Use Podman Quadlet when the Linux service is the main thing.

That distinction is more useful than arguing which tool is better. For developer workflows, Compose is hard to beat. It is popular, readable, portable, and supported by countless projects. For long-running Linux services, Quadlet is quietly excellent. It makes containers feel like native systemd services, works naturally with rootless Podman, and fits the operational model of a serious Linux host.

The preferred split:

Local development: Docker Compose
Portable app examples: Docker Compose
Small self-hosted stacks: Docker Compose or Quadlet
Rootless user services: Podman Quadlet
Long-running host services: Podman Quadlet
Multi-node orchestration: neither; use a real orchestrator

Do not migrate just to be modern. Migrate when the lifecycle model is better. Compose is a great stack tool. Quadlet is a great service tool. The smart choice is to use each where its mental model matches the job.

References

Podman Quadlet Documentation — official Podman Quadlet reference
Docker Compose Documentation — official Docker Compose reference
Run Docker Compose as a Linux Service with systemd — detailed Compose-as-systemd walkthrough on this site
Docker Compose Cheatsheet — Compose commands and patterns reference
Docker Cheatsheet — Docker commands reference
Install Docker on Ubuntu — Docker installation guide with rootless alternatives
Run any Executable as a Service in Linux — general systemd service patterns
Developer Tools: The Complete Guide to Modern Development Workflows — cluster home

Hermes Agent: Headless Server + Remote Desktop Setup

Rost — Tue, 14 Jul 2026 13:07:27 +0000

Running Hermes Agent on a headless server while connecting from a desktop client on another machine requires two server processes and a single client connection.

The architecture separates the Hermes backend into two server-side processes and one client-side surface. The hermes serve backend handles the API and dashboard connections, while the hermes gateway run process manages messaging channels independently. The desktop client connects to the serve backend, not the gateway.

flowchart LR
    subgraph Server["HEADLESS SERVER"]
        serve["hermes serve<br/>--host 0.0.0.0<br/>:9119"]
        gateway["hermes gateway run<br/>Telegram, Discord, Slack"]
    end

    subgraph Client["DESKTOP PC"]
        desktop["hermes desktop<br/>(WebSocket connection)"]
    end

    desktop <--->|WebSocket| serve
    gateway -.->|Shares ~/.hermes/| serve

Two processes on the server, one app on the client. Both server processes share the same ~/.hermes/ config, skills, memory, and sessions. Cron jobs execute on the server where the gateway runs.

For installation, provider setup, and initial configuration, start with the Hermes AI Assistant — Install, Setup, Workflow, and Troubleshooting.

Set up the headless server

1. Configure authentication

Basic auth provides sufficient protection for a trusted LAN. Add credentials to the environment file:

cat >> ~/.hermes/.env << 'EOF'
HERMES_DASHBOARD_BASIC_AUTH_USERNAME=admin
HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=choose-a-strong-password
HERMES_DASHBOARD_BASIC_AUTH_SECRET=$(openssl rand -base64 32)
EOF
chmod 600 ~/.hermes/.env

The .env file lives alongside config.yaml under ~/.hermes/. Hermes resolves configuration with CLI overrides first, then config.yaml, then .env, then built-in defaults. Secrets belong in .env.

2. Start the backend

Run the serve backend on all interfaces:

hermes serve --host 0.0.0.0 --port 9119

The backend listens on port 9119 and accepts WebSocket connections from the desktop client. Verify it is running with a quick status check:

curl -s http://localhost:9119/api/status | jq '.auth_required, .auth_providers'
# Expected: true  "basic"

3. Run as a systemd service

For a persistent server that survives reboots, install a user-level systemd service. Create the unit file at /etc/systemd/user/hermes-serve.service:

[Unit]
Description=Hermes Agent Serve Backend

[Service]
EnvironmentFile=%h/.hermes/.env
ExecStart=/home/rg/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main serve --host 0.0.0.0 --port 9119
Restart=on-failure

[Install]
WantedBy=default.target

Reload the daemon, enable the service, and start it:

systemctl --user daemon-reload
systemctl --user enable hermes-serve
systemctl --user start hermes-serve

Check the service status:

systemctl --user status hermes-serve

4. Start the gateway

The gateway is a separate process that handles messaging channels — Telegram, Discord, Slack, and others. Start it independently:

hermes gateway run

The gateway and the serve backend are two separate processes. The gateway manages sessions, runs cron jobs, and routes messages. The serve backend provides the API surface for desktop and web dashboard connections. They share the same home directory but run independently.

For the full list of gateway commands and subcommands, see the Hermes Agent CLI cheat sheet.
If your primary interface is mobile messaging, pair this setup with Hermes Voice Control from Your Phone for voice-first workflows on top of the same gateway process.

5. Verify the backend

Confirm the backend is responding and authentication is active:

curl -s http://localhost:9119/api/status | jq '.auth_required, .auth_providers'

Expected output:

true
"basic"

If authentication is not enabled, the response will show false for auth_required. Check that the .env file contains the correct variables and that the service has restarted after configuration changes.

Connect from the desktop client

Option A: Hermes Desktop App

Install Hermes Desktop from the official site. Launch the app, navigate to Settings → Gateway → Remote gateway, and enter the server address:

http://<server-ip>:9119

Alternatively, set the remote URL via environment variable:

HERMES_DESKTOP_REMOTE_URL=http://<server-ip>:9119 hermes desktop

Option B: Web dashboard

Open http://<server-ip>:9119 in a browser and sign in with the basic auth credentials. The web dashboard provides a browser-based interface to the Hermes backend without requiring a desktop installation.

Option C: CLI

From the desktop PC's terminal, configure the remote URL via the desktop app settings or the HERMES_DESKTOP_REMOTE_URL environment variable. The CLI surface connects through the same WebSocket channel as the desktop app.

Network and security considerations

Scenario	Recommendation
Trusted LAN	`--host 0.0.0.0` + basic auth
Exposed to internet	Use Tailscale (`--host <tailscale-ip>`) or OAuth provider
Firewall	Open port 9119 TCP on the server

For a trusted local network, basic auth on 0.0.0.0 is adequate. If the server is exposed to the internet, use Tailscale to bind to the Tailscale IP instead of 0.0.0.0, or configure an OAuth provider for stronger authentication. Always open port 9119 TCP in the server's firewall when the backend needs to accept external connections.

Important process distinctions

Understanding the separation between the two server processes prevents common configuration mistakes:

hermes serve — The backend that the desktop app and web dashboard connect to. Handles the API surface and WebSocket connections.
hermes gateway run — The process that handles Telegram, Discord, Slack, and other messaging channels. Manages sessions, runs cron jobs, and routes messages.
These are two separate processes on the server. They share ~/.hermes/ config, skills, memory, and sessions, but run independently.
Cron jobs execute on the server where the gateway runs.
Profiles, skills, and memory are configured on the server side. The client connects to the already-configured backend.

For profile-first configuration and skills tuned to different production roles, see Hermes AI Assistant Skills for Real Production Setups.

Troubleshooting

Desktop cannot connect to the server

Verify the backend is running: systemctl --user status hermes-serve
Check the port is open: ss -tlnp | grep 9119
Test from the server: curl -s http://localhost:9119/api/status
Test from the client machine: curl -s http://<server-ip>:9119/api/status
If the client test fails, check firewall rules: sudo ufw status or sudo iptables -L

Authentication fails

Confirm .env file has correct permissions: ls -la ~/.hermes/.env (should be 600)
Verify the secret was generated: grep HERMES_DASHBOARD_BASIC_AUTH_SECRET ~/.hermes/.env
Restart the service after config changes: systemctl --user restart hermes-serve

Gateway does not respond to messages

The gateway is a separate process from the backend. If the desktop connects but messaging platforms do not work:

Check gateway status: hermes gateway status
Start the gateway if stopped: hermes gateway start
Review logs: hermes logs gateway -f

References

This article is part of the AI Systems cluster, which covers self-hosted assistants, retrieval architecture, local LLM infrastructure, and observability.

GPUs for AI in 2026: NVIDIA, AMD, Intel Compared

Rost — Tue, 14 Jul 2026 00:14:41 +0000

The AI hardware landscape has shifted significantly in 2026, with NVIDIA, AMD, and Intel all competing for developers who need GPUs capable of running local large language models and AI inference workloads.

Choosing the right GPU for AI workloads requires looking beyond marketing numbers and focusing on the specifications that actually affect real-world performance. Memory capacity, memory bandwidth, and software ecosystem maturity consistently matter more than theoretical compute peaks when running transformer models locally.

This comparison covers the most relevant workstation and prosumer GPUs available in mid-2026, including NVIDIA's Blackwell architecture (RTX 50-series), AMD's Radeon AI Pro R9700, and Intel's Arc Pro B70. The goal is to provide a practical reference for developers deciding which hardware best fits their model sizes, software stack, and budget constraints.

Which GPU specifications matter for AI workloads

Marketing materials from GPU vendors emphasise AI TOPS and tensor performance, but these metrics rarely tell the complete story for local inference. The specifications below are ranked by their actual impact on running large language models.

VRAM capacity

VRAM is typically the first limiting factor when running LLMs locally. A model cannot execute entirely on the GPU if it does not fit into available memory. Once model weights spill into system RAM, inference performance drops dramatically.

Approximate VRAM requirements for common model sizes:

Model Size	Recommended VRAM
7B	8-12 GB
14B	16 GB
32B	24-32 GB
70B	48-64 GB
120B+	Multiple GPUs

For most homelab users, moving from 16 GB to 32 GB of VRAM provides a substantially larger practical benefit than increasing raw compute performance. A 32 GB GPU capable of running an entire model will often outperform a theoretically faster 16 GB GPU forced to offload tensors into system memory.

Memory bandwidth

Memory bandwidth determines how quickly model weights can be streamed into compute units. Large transformer models continuously move massive amounts of data between VRAM and processing cores during inference.

As models grow, bandwidth often becomes the dominant performance bottleneck. A card with higher bandwidth can outperform another GPU with significantly higher theoretical compute performance, particularly during prompt processing phases where the model reads through the entire context window.

FP32 compute

FP32 throughput remains useful for scientific computing, simulation, rendering, and some AI preprocessing workloads. Modern inference engines rarely execute entirely in FP32 precision, relying instead on quantised formats like Q4_K_M or Q8_0. FP32 should be considered a secondary metric for AI inference.

AI TOPS and tensor performance

Every GPU vendor promotes AI TOPS as a headline number. These values are not directly comparable across vendors. NVIDIA, AMD, and Intel measure AI throughput differently, use different tensor hardware, and apply different assumptions regarding sparsity and numerical precision.

AI TOPS should be viewed as an indication of peak theoretical capability rather than an expected LLM inference speed. Real-world token generation rates depend on model architecture, quantisation level, context length, and software optimisation — factors that TOPS numbers do not capture.

Software ecosystem maturity

Software support often determines whether hardware reaches its full potential. The current ecosystem landscape is approximately:

Vendor	Primary AI Stack	Maturity
NVIDIA	CUDA, TensorRT	Industry standard
AMD	ROCm, HIP, Vulkan	Solid for PyTorch, llama.cpp, Ollama
Intel	oneAPI, SYCL, OpenVINO	Improving rapidly, trailing peers

CUDA remains the industry standard with the broadest library support. ROCm has matured significantly over the past two years and now provides a functional experience for PyTorch, llama.cpp, and Ollama on Linux. Intel's oneAPI ecosystem continues to improve but still trails both NVIDIA and AMD in overall software maturity and community adoption.

For a deeper look at NVIDIA-specific GPU analysis, see Comparing NVIDIA GPU Suitability for AI.

Complete GPU comparison table

The table below compares the most relevant workstation and enthusiast GPUs for AI workloads in 2026.

GPU	VRAM	Bandwidth	FP32 (TFLOPS)	AI TOPS (INT8)	TBP	MSRP
NVIDIA RTX 5090	32 GB	1792 GB/s	104.6	3352	575 W	$1799
NVIDIA RTX 5080	16 GB	960 GB/s	56.3	1801	360 W	$999
NVIDIA RTX 5070 Ti	16 GB	896 GB/s	43.9	1406	300 W	$649
NVIDIA RTX 5070	12 GB	672 GB/s	30.9	494	250 W	$549
NVIDIA RTX 5060 Ti 16GB	16 GB	448 GB/s	23.7	614	180 W	$399
NVIDIA RTX PRO 6000	96 GB	1792 GB/s	125.0	4000	600 W	$4999
NVIDIA RTX PRO 5000	48 GB	1344 GB/s	73.7	2064	300 W	$2499
NVIDIA RTX PRO 4500	32 GB	896 GB/s	54.9	1577	200 W	$2500
NVIDIA RTX PRO 4000	24 GB	672 GB/s	46.9	1178	145 W	$1500
NVIDIA RTX PRO 4000 SFF	24 GB	432 GB/s	46.9	770	125 W	$1500
NVIDIA RTX PRO 2000	16 GB	288 GB/s	18.4	592	70 W	$700
AMD Radeon AI Pro R9700	32 GB	640 GB/s	47.8	766	300 W	$1299
Intel Arc Pro B70	32 GB	608 GB/s	22.94	367	230 W	$949

Key observations by segment

Consumer GPUs

The RTX 5090 remains the fastest single-GPU solution for local AI development, combining exceptional memory bandwidth with the mature CUDA ecosystem. For users running large quantised models, it currently represents the highest-performance consumer option.

The RTX 5080 and RTX 5070 Ti both offer 16 GB of VRAM, which is sufficient for most 7B-14B models but limits you when working with larger checkpoints. The RTX 5060 Ti 16GB variant is an interesting budget option — 16 GB of VRAM at $399 is compelling for entry-level AI workloads, though the narrower memory bus will impact throughput.

Workstation GPUs

Within the workstation segment, AMD's Radeon AI Pro R9700 occupies an attractive middle ground. It delivers 32 GB of VRAM, competitive memory bandwidth, and a significantly lower purchase price than NVIDIA's professional offerings. For developers already comfortable with ROCm on Linux, it provides one of the strongest value propositions in 2026.

Intel's Arc Pro B70 is particularly interesting because of its pricing. Although it offers lower compute performance than both NVIDIA and AMD, it provides the same 32 GB memory capacity while consuming less power. For users building cost-effective multi-GPU inference servers, the B70 deserves consideration — especially if the oneAPI ecosystem meets your software requirements.

Professional GPUs

NVIDIA's RTX PRO series dominates the professional segment, with the RTX PRO 6000 offering 96 GB of VRAM — unmatched by any competitor. For teams running very large models or multiple concurrent inference workloads, the RTX PRO 6000 and RTX PRO 5000 remain the safest choices, though at a premium price.

For a real-world performance comparison across different hardware platforms, see NVIDIA DGX Spark vs Mac Studio vs RTX-4080.

Practical hardware considerations

Physical dimensions and form factor

GPU size varies significantly across product lines and affects compatibility with your case and cooling solution.

GPU	Approx. Length	Slots	Cooler Type
RTX 5090	333 mm	2.7×	Triple-fan, blower or open
RTX 5080	303 mm	2.5×	Dual/triple-fan
RTX 5070 Ti	280 mm	2.4×	Dual-fan
RTX 5070	245 mm	2.1×	Dual-fan
RTX 5060 Ti	200 mm	1.8×	Dual-fan
AMD R9700	300 mm	2.5×	Dual-fan
Intel Arc Pro B70	267 mm	2.1×	Single/dual-fan
RTX PRO 6000	438 mm	3.5×	Blower, full-height
RTX PRO 5000	438 mm	3.5×	Blower, full-height
RTX PRO 4000	267 mm	2.1×	Blower, low-profile option
RTX PRO 4000 SFF	178 mm	1.5×	Blower, half-height

The RTX PRO 6000 and 5000 are significantly longer than consumer cards and require full-height tower cases. The RTX PRO 4000 SFF is one of the few GPUs under 180 mm, making it suitable for compact workstation builds and rack-mounted servers.

Consumer GPUs (RTX 50-series) use open-air coolers that exhaust heat into the case — adequate case airflow is essential. Workstation GPUs use blower-style coolers that exhaust heat directly out the rear, which is better for multi-GPU configurations and enclosed server environments.

Power delivery and PSU requirements

TBP (Total Board Power) is the GPU's maximum power draw, but actual system requirements depend on transient spikes and CPU overhead.

GPU	TBP	Recommended PSU	Power Connectors
RTX 5090	575 W	1000 W+	12V-2x6 (20-pin)
RTX 5080	360 W	750 W	12V-2x6
RTX 5070 Ti	300 W	650 W	8-pin + 8-pin
RTX 5070	250 W	600 W	8-pin
RTX 5060 Ti	180 W	550 W	8-pin
AMD R9700	300 W	650 W	8-pin + 8-pin
Intel Arc Pro B70	230 W	550 W	8-pin
RTX PRO 6000	600 W	1000 W+	12V-2x6
RTX PRO 5000	300 W	650 W	8-pin + 8-pin
RTX PRO 4000	145 W	500 W	8-pin
RTX PRO 4000 SFF	125 W	450 W	8-pin
RTX PRO 2000	70 W	400 W	PCIe slot only

The RTX 5090 and RTX PRO 6000 both exceed 575W TBP and require the newer 12V-2x6 connector (20-pin). Ensure your PSU supports this connector natively — adapter cables from multiple 8-pin connectors are not recommended for cards above 450W due to transient power spikes that can exceed rated capacity momentarily.

Thermal characteristics and sustained workloads

AI inference workloads keep the GPU under sustained load, unlike gaming which has variable utilisation. This affects thermal behaviour significantly.

RTX 5090 at 575W: Expect GPU temperatures of 72-78°C under sustained inference. The higher TBP means more heat dissipation is required — a case with positive static pressure and quality filters is recommended.
RTX 5080 at 360W: Runs cooler, typically 65-72°C. More manageable for standard mid-tower cases.
Workstation GPUs (blower): RTX PRO series exhaust heat directly out the case, keeping case temperatures lower. GPU temperatures may read higher (75-82°C) but this is by design — the blower cooler trades GPU temperature for lower case temperature.
Low-power options: RTX PRO 2000 at 70W and RTX PRO 4000 SFF at 125W are suitable for passive or low-fan-speed cooling, making them ideal for always-on inference servers where noise matters.

For multi-GPU setups, blower-style coolers (workstation GPUs) are strongly preferred over open-air consumer coolers, as the second GPU would otherwise pull hot air from the first.

PCIe lanes and bandwidth

GPU performance can be limited by PCIe lane count. A GPU plugged into a x8 or x4 slot will experience reduced memory bandwidth compared to a full x16 connection. For multi-GPU setups, understand how PCIe lanes are distributed across your motherboard. See LLM Performance and PCIe Lanes for detailed analysis.

Multi-GPU setups

When a single GPU cannot fit your model, multi-GPU configurations become necessary. NVIDIA NVLink (where supported) and PCIe-based model parallelism are the primary approaches. The AI Infrastructure on Consumer Hardware guide covers multi-GPU deployment strategies in depth.

Note that AMD and Intel GPUs have limited multi-GPU inference support in most frameworks. If you plan to scale with multiple GPUs, NVIDIA is currently the only practical option.

Conclusion

There is no universally best GPU for AI workloads. The right choice depends on your software stack, budget, and the size of the models you intend to run.

NVIDIA's Blackwell family remains the benchmark for inference performance, thanks to outstanding memory bandwidth and the maturity of CUDA and TensorRT. AMD's Radeon AI Pro R9700 has established itself as a compelling workstation option, offering an excellent balance between price, memory capacity, and compute performance. Intel's Arc Pro B70 proves that affordable 32 GB workstation GPUs are now a reality, though its software ecosystem continues to mature.

The most important lesson from 2026 is that AI hardware should no longer be evaluated using gaming benchmarks. For modern LLM inference, VRAM capacity, memory bandwidth, and software support consistently have a greater impact on real-world performance than theoretical AI TOPS alone.

References

Comparing NVIDIA GPU Suitability for AI — NVIDIA-specific GPU analysis with detailed CUDA core and tensor core comparisons
AI Infrastructure on Consumer Hardware — Full-stack guide to deploying self-hosted AI with consumer GPUs
NVIDIA DGX Spark vs Mac Studio vs RTX-4080 — Real-world Ollama performance benchmarks across hardware platforms
LLM Performance and PCIe Lanes — How PCIe configuration affects LLM inference performance
Ollama Cheatsheet — Command reference and tips for Ollama model serving
Quadro RTX 5880 Ada Review — Review of the 48GB workstation GPU alternative
Best LLM on 16 GB VRAM GPU — llama.cpp benchmarks for models on 16 GB VRAM