DEV Community: Rost

Dead Letter Queues: Handling Poison Messages in Distributed Systems

Rost — Fri, 31 Jul 2026 13:26:50 +0000

A dead-letter queue is the safety net that catches messages your consumers cannot process, so one broken payload does not block or silently drop everything behind it in the queue.

Every message-driven system eventually receives a message it cannot handle: a malformed payload, a schema that changed underneath the consumer, or a downstream call that fails no matter how many times you retry it. Without a dead-letter queue, that message either blocks the head of the queue forever or gets silently discarded, and both outcomes are worse than knowing about the failure.

A DLQ turns an invisible failure into a visible, inspectable one. It gives you a place to quarantine the message, alert on it, and decide — deliberately, not by accident — whether to fix and replay it or discard it for good.

The mechanics differ across brokers, but the underlying pattern is the same everywhere: a delivery-attempt counter, a threshold, and a destination for messages that cross it. This guide covers what a DLQ actually does, how to tell a poison message from a transient failure, when to retry versus discard, and how to replay safely once you have fixed the root cause. For the broader integration-patterns context this pattern sits inside, see the App Architecture.

What Is a Dead Letter Queue

A dead-letter queue is a separate, ordinary queue that a broker or consumer routes a message to after that message fails processing too many times. It is not a special construct — RabbitMQ's dead-letter queue is a regular queue bound to a regular exchange, and an SQS DLQ is a regular standard or FIFO queue. What makes a queue a "DLQ" is purely that something else points failed messages at it.

flowchart LR
    P[Producer] --> Q[Main Queue]
    Q --> C[Consumer]
    C -- ack: success --> Done[Message deleted]
    C -- fail / nack / timeout --> Q
    Q -- retry budget exhausted --> DLQ[Dead Letter Queue]
    DLQ --> I[Inspect / alert]
    I -- fix root cause --> R[Replay to main queue]
    I -- unrecoverable --> D[Archive / discard]

Each broker implements the redirect differently:

Amazon SQS uses a redrive policy with a maxReceiveCount. Once a message has been received that many times without being deleted, SQS moves it to the configured deadLetterTargetArn. AWS explicitly recommends keeping the DLQ's message retention period longer than the source queue's, because the original enqueue timestamp — not the move time — still governs expiry.
RabbitMQ dead-letters a message when it is rejected with requeue=false, its per-message TTL expires, the queue hits a length limit, or a quorum queue exceeds its delivery-limit. You configure this with the x-dead-letter-exchange (and optionally x-dead-letter-routing-key) queue arguments, and RabbitMQ attaches x-death headers recording the reason, the origin queue, and how many times it happened.
Apache Kafka has no broker-native DLQ. Kafka only tracks offsets; it has no concept of a "failed" message. The dead-letter topic pattern is something you build in the consumer, in a Kafka Streams topology, or in a Kafka Connect connector — commonly paired with a retry-topic tier before the terminal DLT, as Spring Kafka's @RetryableTopic and DeadLetterPublishingRecoverer do.
Azure Service Bus dead-letters automatically once a message's delivery count exceeds MaxDeliveryCount (default 10), and also for a handful of system reasons such as TTLExpiredException, HeaderSizeExceeded, and MaxTransferHopCountExceeded, each recorded in the message's DeadLetterReason property.

For a broader view of how brokers and streaming platforms fit together operationally rather than as a reliability pattern, Apache Kafka Quickstart and RabbitMQ on AWS EKS vs SQS cover the infrastructure side of running these brokers.

Poison Messages

A poison message is one that will never succeed no matter how many times a consumer retries it — a malformed JSON payload, a schema field that a producer renamed, a business rule violation, or a bug that throws on a specific input every single time. That is different from a transient failure, where the message is fine but the environment briefly is not: a downstream timeout, a database connection blip, a rate limit response.

Treating both failure types the same way is the most common DLQ mistake. If you dead-letter on the first failure, you punish transient errors that would have succeeded on retry. If you retry poison messages dozens of times before giving up, you waste compute, delay unrelated messages behind them (on ordered queues and partitions), and flood your logs with the same stack trace.

A few detection signals help separate the two:

Exception type. Deserialization errors, validation errors, and ClassCastException-style failures are almost always permanent. Spring Kafka's DefaultErrorHandler explicitly treats certain exceptions as fatal and skips retries for them rather than exhausting the retry budget first.
Repeat count with no variance. RabbitMQ's x-death header array lets you see exactly how many times a message has been dead-lettered and why; a message with a growing count and an identical x-first-death-reason on every cycle is poison, not unlucky.
Consistent failure across replicas. If every consumer instance fails on the same message while succeeding on everything around it, the message itself is the problem, not the infrastructure.

For distinguishing retryable from non-retryable failures at the code level — the same classification a DLQ policy depends on — see Go Error Handling Architecture: Boundaries and Patterns.

Retry vs Discard

The core policy decision behind every DLQ is the retry threshold: how many delivery attempts a message gets before it is quarantined. Get this too low and you dead-letter messages that would have succeeded after a brief downstream hiccup. Get it too high and a poison message sits in the main queue for a long time, consuming worker capacity and — on ordered systems — blocking everything queued behind it.

Current guidance across the major brokers converges on similar numbers:

Broker	Mechanism	Typical threshold
Amazon SQS	`maxReceiveCount` in redrive policy	3–5 for mixed transient/permanent workloads
RabbitMQ (quorum queues)	`delivery-limit` policy argument	3–5, tuned per queue
Azure Service Bus	`MaxDeliveryCount`	Default 10, often reduced for latency-sensitive queues
Kafka (via retry topics)	Retry-count header + retry-topic tier	3–4 retry-topic hops before the terminal DLT

A practical middle ground many teams land on is: start conservative (2–3 attempts), watch the actual failure mix in production, and raise the threshold only for queues where you can show most failures resolve within a few retries. Pair the retry count with exponential backoff and jitter between attempts so a downstream outage does not turn into a retry storm — the same discipline covered in backoff and circuit-breaker design. A circuit breaker at the integration boundary complements this: it stops sending requests to an unhealthy dependency instead of letting every message in the queue individually discover the outage and dead-letter one by one.

Once a message is in the DLQ, "discard" should still be a deliberate action, not neglect. Set a retention period on the DLQ itself — long enough to investigate (AWS recommends the DLQ retention exceed the source queue's; a week is a common floor for RabbitMQ DLQs) — and alert on DLQ depth and age so failures get triaged instead of silently expiring. A message that ages out of the DLQ unexamined is a message you decided to lose without deciding to lose it.

Idempotency matters just as much here as it does anywhere else duplicates can occur: a message that gets redriven from a DLQ back to the main queue is, functionally, a duplicate delivery. If your consumer is not safe to run twice on the same message, redriving from a DLQ can create the exact duplicate-side-effect bug you were trying to avoid. See Idempotency in Distributed Systems That Actually Works for the consumer-side patterns that make redrive safe.

Replay Strategies

Getting a message out of the DLQ correctly is its own discipline, separate from getting it in.

Fix the root cause first. Deploying the consumer fix before replaying is the difference between a clean recovery and re-poisoning the queue with the same failure a second time.
Redrive deliberately, not automatically. SQS supports a redrive-to-source feature that moves messages back to their original queue (or another destination) on demand; RabbitMQ and Kafka require you to build the equivalent consumer or tooling yourself. Either way, treat replay as an operator-triggered action with a record of what was replayed and when.
Preserve ordering where it matters. For Kafka, the dead-letter topic should have at least as many partitions as the source topic and should retain the original message key, so that replayed messages land back on the correct partition and preserve per-key ordering.
Cap replay attempts. A message that fails again after a fix-and-replay cycle is not transient — route it to a permanent archive (a database table, an object-storage bucket) instead of looping it through the DLQ indefinitely. RabbitMQ's own docs warn that a dead-lettered message can be routed between queues only a limited number of times (16) before further TTL-based dead-lettering is disabled.
Never let a DLQ dead-letter into itself. If your DLQ has its own x-dead-letter-exchange (RabbitMQ) or its own redrive policy (SQS) pointed back at the same chain, a replay failure can create an infinite loop. Keep the DLQ's own dead-letter configuration empty, or point it at a strictly terminal archive.
Alert on volume, not just presence. A single message in a DLQ is a data point; a sudden spike is an incident. Wire DLQ depth and message age into the same alerting pipeline you use for everything else — see Modern Alerting Systems Design for Observability Teams for routing and noise-reduction practices that apply directly to DLQ alerts.

If your workflow involves multi-step, long-running processes rather than single messages, the same dead-letter thinking applies at the workflow layer — a saga's compensation logic needs the same "quarantine, inspect, decide" discipline when a step fails permanently instead of transiently. And when the events themselves originate from a database write, the transactional outbox pattern already builds dead-letter handling into the relay worker, so the pattern shows up one layer earlier than the broker.

Where DLQs Fit in the Bigger Picture

A dead-letter queue does not make failures go away — it makes them survivable and reviewable instead of silent. It works best alongside retries with backoff for the transient case, idempotent consumers so redrive is safe, and a circuit breaker so a struggling dependency does not flood the main queue (and, eventually, the DLQ) with the same failure thousands of times over. Treat the DLQ threshold, retention, and alerting as first-class configuration decisions, not defaults you leave untouched, and dead letters become a diagnostic tool instead of a place where data quietly disappears.

Useful Links

LLM Wiki Maintenance: Drift, Contradictions and Review

Rost — Mon, 20 Jul 2026 08:29:50 +0000

An LLM Wiki fails when old facts remain plausible, contradictions become polished, and generated summaries drift from their sources.

Maintenance is the real product of any compiled knowledge system. Creating wiki pages is straightforward compared with keeping them trustworthy across months of ingest, edits, rewrites, and new sources.

This article covers the operational side of LLM Wiki systems: drift detection, contradiction checks, citation discipline, linting, Git review, and maintenance workflows. It assumes you already understand the basic pattern described in LLM Wiki - Compiled Knowledge That RAG Cannot Replace: raw sources are compiled into durable Markdown pages that humans and agents can query later. The calm but opinionated view is that an LLM Wiki without maintenance is just a nicer-looking knowledge graveyard.

What Makes LLM Wiki Maintenance Different

Traditional wikis rot because people stop updating them. RAG systems drift because the corpus changes, chunks get stale, metadata is weak, and retrieval keeps finding plausible but outdated fragments. The deeper reason is that retrieval and representation solve different problems: retrieval can be re-run against fresh data, but a representation you already compiled has to be actively kept honest.

An LLM Wiki has a different failure mode. It can look clean even when it is wrong.

The pages may be well formatted. The links may work. The summaries may sound balanced. But underneath that neat surface, the system may have dropped critical facts, merged incompatible concepts, cited summaries instead of sources, or preserved an old decision as if it still applied.

That is why LLM Wiki maintenance must check both structure and meaning.

The Maintenance Goal

The goal is not to make every page perfect. That is too expensive, and it usually leads to abandoned systems.

The goal is to keep the wiki useful, inspectable, and recoverable.

A maintained LLM Wiki should make it easy to answer these questions:

What sources support this claim?
When was this page last reviewed?
Has this concept changed?
Are there conflicting pages?
Is this summary still current?
Did the agent rewrite more than it should?
Can we roll back a bad update?
Can a human understand why the page says what it says?

If the answer is no, the problem is not only content quality. It is system design.

A Simple Maintenance Loop

A useful LLM Wiki needs a repeatable loop. The loop should be simple enough to run often and strict enough to catch drift before it becomes invisible.

flowchart TD
    A[Add or update source] --> B[Compile into wiki pages]
    B --> C[Update links and indexes]
    C --> D[Run structural lint checks]
    D --> E[Run semantic review checks]
    E --> F[Human review of risky changes]
    F --> G[Commit approved changes]
    G --> H[Schedule stale-page review]
    H --> A

This loop is not glamorous. That is the point.

A knowledge system becomes durable through boring maintenance: source preservation, explicit review, predictable structure, and small safe updates. The same instinct drives evergreen notes: a note or a wiki page only compounds in value if someone keeps refining it instead of leaving it to rot after the first draft.

Types of Drift in an LLM Wiki

Knowledge drift is not one thing. Different drift types require different checks. A good maintenance system should name them clearly.

Source Drift

Source drift happens when the underlying source material changes.

For example, a tool releases a new version, a policy is updated, an API changes, or a vendor deprecates an old feature. The old wiki page may still be accurate for the previous version, but wrong for current use.

Source drift is dangerous because the old claim may still be true in historical context. The problem is not that the claim is fake. The problem is that it no longer answers the current question.

Maintenance response:

Record source dates
Record last reviewed dates
Mark version-specific pages clearly
Link old pages to superseding pages
Avoid mixing old and new versions without labels

Concept Drift

Concept drift happens when the meaning of a term changes over time.

This is common in AI and software architecture. Terms like "agent", "memory", "RAG", "workflow", "tool use", "structured output", and "context engineering" can shift meaning quickly.

A wiki can accidentally preserve several meanings of the same term without explaining the difference. That creates confusing pages that sound coherent but combine incompatible ideas.

Maintenance response:

Maintain glossary pages
Add "meaning in this wiki" sections
Separate overloaded concepts into distinct pages
Link related meanings explicitly
Avoid letting the model silently merge terms

Terminology Drift

Terminology drift is smaller than concept drift but still harmful.

It happens when the wiki uses multiple names for the same thing: "LLM Wiki", "compiled knowledge base", "AI-maintained wiki", "agent-maintained wiki", and "Markdown knowledge base".

Some variation is fine. Too much variation breaks search, linking, and review.

Maintenance response:

Define canonical page names
Keep aliases in front matter or page metadata
Redirect duplicate pages
Lint for near-duplicate titles
Use consistent anchor text in internal links

Decision Drift

Decision drift happens when a past decision remains documented but no longer reflects current practice.

For example, a page might say that the project uses vector RAG for all document search, while newer pages describe an LLM Wiki workflow. Both statements may be historically true, but the wiki must show which one is current.

This matters for architecture notes, engineering processes, content strategy, and internal tooling. The same discipline applies to decision records in AI-driven development: a decision page is only trustworthy if superseded choices are marked as superseded rather than silently overwritten.

Maintenance response:

Mark decisions as proposed, accepted, superseded, or rejected
Keep decision dates
Link superseded decisions to replacements
Preserve historical context
Avoid deleting old decisions without trace

Citation Drift

Citation drift happens when a page cites a source, but the claim no longer matches what the source says.

This can happen after a rewrite. The citation remains in place, but the sentence around it changes. The page still looks sourced, yet the citation no longer supports the claim.

This is one of the most serious LLM Wiki failure modes because it creates false confidence.

Maintenance response:

Check claim-level citation support
Avoid citing only summaries
Keep raw sources
Require citations for important claims
Flag paragraphs with sources but no direct support
Review citation changes in Git diffs

Structure Drift

Structure drift happens when the wiki slowly loses navigability.

New pages are created instead of updating old ones. Index pages fall behind. Duplicate pages appear. Orphan pages accumulate. Related pages stop linking to each other.

The wiki still contains useful knowledge, but finding and trusting it becomes harder.

Maintenance response:

Lint for orphan pages
Maintain index pages
Detect duplicate topics
Require backlinks for canonical pages
Archive rather than scatter
Keep folder roles clear

The Core Maintenance Files

An LLM Wiki should not rely on memory, vibes, or repeated prompting. It needs operating files that tell humans and agents how to maintain it.

A practical structure can look like this:

llm-wiki/
  raw/
    sources/
    transcripts/
    documents/
  wiki/
    index.md
    concepts/
    entities/
    projects/
    decisions/
  maintenance/
    review-log.md
    lint-report.md
    stale-pages.md
    contradiction-report.md
  rules/
    AGENTS.md
    schema.md
    style-guide.md
    citation-policy.md
    source-policy.md

The exact names do not matter. The roles do.

The wiki needs source storage, compiled pages, review artifacts, and rules that survive across sessions.

AGENTS.md

AGENTS.md should explain how the agent is allowed to work.

It should answer questions like:

Should the agent update existing pages before creating new ones?
When should it ask for review?
How should it cite sources?
Can it reorganize folders?
Can it rewrite old pages?
What should it do with contradictions?
What should it never delete?

A useful rule is:

Prefer small, source-backed updates over broad rewrites.

That one sentence prevents many bad maintenance habits. It also draws a useful boundary relative to agent memory: memory shapes how an agent behaves in the moment, while the wiki's rules file shapes what the agent is allowed to change in the shared knowledge base.

schema.md

schema.md should define the page structure.

For example:

# Page Title

## Summary
Short current summary.

## Key Claims
- Claim with source reference.
- Claim with source reference.

## Current Status
Current, historical, superseded, draft, or uncertain.

## Details
Main compiled explanation.

## Related Pages
Internal links.

## Sources
Raw source references.

## Review Notes
Last reviewed date and reviewer.

A schema does not need to be rigid. But without a schema, the wiki becomes a pile of inconsistent essays. The schema gives the agent something to preserve.

citation-policy.md

The citation policy should say what requires a source. At minimum, require citations for:

Technical claims
Comparisons
Product behavior
Version-specific statements
Benchmark results
Pricing
Legal or compliance claims
Current status claims
Claims copied or derived from a source

A good citation policy also says what not to do. Do not cite a compiled page as if it were the original source. Do not attach citations to paragraphs they do not support. Do not keep a citation after rewriting a claim unless the source still supports it.

Citation discipline is the difference between a useful LLM Wiki and decorative confidence.

source-policy.md

The source policy should define how raw sources are stored.

For example:

Keep original files when possible
Preserve URLs and access dates
Store copied notes separately from generated summaries
Record source type
Mark low-confidence sources
Avoid overwriting raw material
Keep superseded sources when they explain historical decisions

This policy protects the wiki from a common mistake: summarizing a source, discarding the original, and then treating the summary as evidence.

That is not knowledge management. That is lossy compression.

What an LLM Wiki Linter Should Check

A linter should check structure first. Structural problems are easier to automate and often reveal deeper semantic problems.

Useful structural checks include:

Broken internal links
Orphan pages
Duplicate titles
Missing source sections
Missing review dates
Pages without backlinks
Empty or placeholder sections
Very long pages without section structure
Pages with no incoming links from an index
Inconsistent naming conventions
Invalid front matter
Stale "current" pages older than a review threshold

These checks do not prove the knowledge is true. They prove the wiki is still maintainable. That is a necessary starting point.

Semantic Checks

Semantic checks are harder but more valuable.

Useful semantic checks include:

Claims without direct source support
Two pages making incompatible claims
Old decisions presented as current
Duplicate concepts with different names
One concept page mixing several meanings
Summaries that omit known constraints
"Best" or "recommended" claims without criteria
Version-specific claims without version labels
Pages that contradict newer sources

These checks should not automatically rewrite the wiki. They should usually produce a report for review.

The safer pattern is:

Detect automatically.
Explain clearly.
Update deliberately.
Review risky changes.

Contradiction Detection

Contradiction detection is not just asking an LLM whether two pages contradict each other.

That can help, but it is too vague. A better approach is to compare claims.

A contradiction workflow can look like this:

flowchart TD
    A[Extract claims from page] --> B[Find related pages and sources]
    B --> C[Extract claims from related material]
    C --> D[Group claims by subject]
    D --> E[Compare status, date, version, and scope]
    E --> F{Conflict found?}
    F -->|No| G[No action]
    F -->|Yes| H[Classify conflict]
    H --> I[Create contradiction report]
    I --> J[Human or agent-assisted resolution]

This works better because many apparent contradictions are not real contradictions.

One page may describe version 1.0 and another version 2.0. One page may describe personal use and another enterprise use. One page may describe the design goal and another the implementation reality.

The contradiction report should include:

Conflicting claims
Pages involved
Source references
Date or version context
Likely explanation
Suggested resolution
Whether human review is required

Do not let the agent silently resolve contradictions by blending both claims into a vague compromise. That creates smooth nonsense.

How to Resolve Contradictions

There are several valid ways to resolve a contradiction.

If one claim is outdated, mark it as superseded and link to the newer page.
If both claims are true in different contexts, split the context clearly.
If the sources disagree, preserve the disagreement and explain it.
If the wiki page invented or overgeneralized a claim, remove or narrow the claim.
If the contradiction reflects a real unresolved decision, create or update a decision page.

The goal is not to remove all tension. Some tension is useful. A good wiki shows where knowledge is settled and where it is still uncertain.

Stale Page Review

Every compiled page should have a review status.

A simple metadata block can be enough:

status: current
last_reviewed: 2026-07-10
review_after: 2026-10-10
source_confidence: medium

For fast-moving topics, use short review windows. For stable concepts, use longer windows.

Page type	Review interval
Tool version pages	30 to 90 days
Pricing or availability pages	7 to 30 days
Architecture principles	6 to 18 months
Historical decision records	Only when superseded
Glossary pages	3 to 12 months
Source summaries	When source changes
Comparison pages	30 to 180 days

The interval is less important than the habit. A page without a review date is a page that will eventually lie quietly.

Source-Aware Updates

When a new source arrives, the agent should not simply write a new summary page.

It should ask:

Does this update an existing concept?
Does it supersede a previous source?
Does it contradict an existing page?
Does it add a new entity or term?
Does it change a recommendation?
Does it require an index update?
Does it affect decision pages?

A good ingest workflow updates the wiki as a graph, not as a stack of isolated summaries.

This is why one source may touch many pages. That is normal. The maintenance challenge is making those updates small, reviewable, and traceable.

Review with Git Diffs

Git is one of the best maintenance tools for an LLM Wiki.

Not because Git is fashionable, but because generated knowledge needs reviewable change history. A Git diff shows what the agent changed, deleted, moved, or rephrased.

Use Git to review:

New pages
Deleted sections
Changed claims
Changed citations
Renamed pages
Index updates
Link changes
Status changes
Broad rewrites

The most important review habit is to inspect deletions. LLMs often remove details while making prose cleaner.

Clean prose is not always better knowledge.

Safe Commit Strategy

Avoid giant commits like:

Update wiki

Use commits that explain the operation:

ingest: add source notes for Qwen embedding release
maint: mark old vector store comparison as superseded
lint: fix broken internal links in RAG pages
review: update LLM Wiki page citations
refactor: split agent memory concept from long-term memory

This gives future humans and agents a useful history.

Commit messages are part of the knowledge system. They explain why the wiki changed.

Review Levels

Not every page needs the same review level. Use risk-based review.

Low-risk changes:

Broken link fixes
Formatting cleanup
Adding backlinks
Adding index entries
Correcting typos
Adding aliases

Medium-risk changes:

Adding source summaries
Creating new concept pages
Updating recommendations
Merging duplicate pages
Changing page status

High-risk changes:

Deleting claims
Rewriting canonical pages
Changing decision records
Updating security, legal, pricing, or benchmark claims
Resolving contradictions
Marking content as current or superseded

High-risk changes should get human review. That may be you, a maintainer, or a domain owner.

The agent can prepare the change, but it should not always be the final authority.

A Maintenance Dashboard

A simple maintenance dashboard can live in Markdown.

Example:

# LLM Wiki Maintenance Dashboard

## Needs Review
- wiki/concepts/agent-memory.md - review overdue
- wiki/projects/search-index.md - conflicting claims found

## Stale Pages
- wiki/tools/ollama.md - last reviewed 120 days ago
- wiki/concepts/rag.md - fast-moving topic, review due

## Broken Links
- wiki/index.md -> wiki/concepts/old-page.md

## Orphan Pages
- wiki/entities/vendor-x.md

## Contradiction Reports
- reports/contradictions/2026-07-10-agent-memory.md

## Recent Ingests
- raw/sources/2026-07-09-new-paper.md

This is not a fancy admin panel. It is a working surface.

For many solo or small-team systems, a Markdown dashboard is enough.

Metrics for a Healthy LLM Wiki

You can track wiki health without overengineering it.

Useful metrics include:

Number of pages
Pages without sources
Pages without review dates
Pages past review date
Broken internal links
Orphan pages
Duplicate titles or aliases
Contradiction reports open
Contradiction reports resolved
Average page age by cluster
Raw sources not yet ingested
Pages changed without human review
Canonical pages changed in the last 30 days

These metrics are not vanity numbers. They tell you whether the system is becoming easier or harder to trust.

A growing wiki with rising orphan pages and stale reviews is not compounding knowledge. It is compounding maintenance debt.

Maintenance Prompts

Maintenance prompts should be narrow. Broad prompts produce broad rewrites.

Structural Lint Prompt

Review the wiki structure.

Check for:
- broken internal links
- orphan pages
- duplicate or near-duplicate page titles
- pages missing source sections
- pages missing last_reviewed metadata
- index pages that do not link to new pages

Do not rewrite content.
Produce a Markdown report with findings and suggested fixes.

Citation Review Prompt

Review citations for the selected page.

For each important claim:
- identify the supporting source
- check whether the source directly supports the claim
- flag claims with weak, missing, or mismatched support
- do not rewrite the page yet

Return a table with:
claim, current citation, support level, issue, suggested action.

Contradiction Check Prompt

Check this page for contradictions against related pages.

Steps:
1. Extract the main claims from the page.
2. Find related pages through links, backlinks, aliases, and search.
3. Extract potentially conflicting claims.
4. Classify each conflict as:
   - real contradiction
   - version difference
   - scope difference
   - terminology difference
   - unresolved uncertainty
5. Recommend a resolution.

Do not edit files until the contradiction report is reviewed.

Stale Page Review Prompt

Review this page for staleness.

Check:
- dates
- version-specific claims
- tool behavior
- recommendations
- links to superseded pages
- sources newer than the page review date
- claims marked current without recent support

Return:
- keep current
- update needed
- mark historical
- mark superseded
- split into versioned pages

Safe Update Prompt

Update the selected wiki page using the provided source.

Rules:
- preserve existing useful structure
- do not remove sourced claims unless the source is superseded
- add citations for new claims
- mark uncertainty explicitly
- update related pages only when necessary
- add a review note explaining what changed
- keep the diff small

These prompts are not magic. They are guardrails. The real value comes from making the maintenance action explicit.

Common Maintenance Mistakes

The first mistake is letting the agent rewrite too much.

Large rewrites feel productive because they make pages smoother. They also make it harder to see what changed and easier to lose sharp details.

The second mistake is treating generated summaries as sources.

A summary can be useful, but it is not the evidence. Keep raw sources and cite them when claims matter.

The third mistake is creating new pages instead of updating old ones.

This produces duplicate concepts, conflicting recommendations, and search confusion. The agent should search first, update second, and create new pages only when the concept is genuinely new.

The fourth mistake is deleting historical context.

Old decisions, failed experiments, and superseded recommendations can be valuable. Mark them clearly instead of erasing them.

The fifth mistake is forgetting indexes.

A page that is not linked from the right place is half-lost. Index maintenance is not clerical work. It is navigation architecture.

The sixth mistake is using the same review policy for every page.

A stable conceptual page and a fast-moving tool comparison should not have the same review schedule.

Archive, Supersede, or Delete

Most stale pages should not be deleted immediately.

Outcome	Use when
Archive	The page is historical but still useful
Supersede	A newer page replaces the old answer
Delete	The page is duplicate, empty, wrong, or unrecoverable

A superseded page should link to the replacement.

For example:

Status: superseded
Superseded by: wiki/concepts/agent-memory-architecture.md
Reason: This page used an older definition of agent memory before the project separated session memory, user memory, and compiled knowledge.

This preserves context and reduces confusion.

Deletion should be rare and reviewable. Knowledge systems need pruning, but invisible pruning is dangerous.

Canonical Pages Need Extra Care

Every LLM Wiki eventually develops canonical pages.

These are pages that define core concepts, architecture choices, workflows, or project vocabulary. They are more important than ordinary notes because many other pages depend on them.

Examples:

What is LLM Wiki
RAG vs compiled knowledge
Agent memory
Source policy
Review workflow
Project architecture
Glossary
Index

Changes to canonical pages should require stricter review.

A small error on an obscure page is local. A small error on a canonical page can distort the whole wiki.

How Often Should You Run Maintenance

For an active LLM Wiki, run small maintenance frequently and deep maintenance occasionally.

Frequency	Maintenance task
Every ingest	Update links, sources, indexes, and review notes
Weekly	Run structural lint checks
Monthly	Review stale pages and orphan pages
Monthly	Check contradictions in active clusters
Quarterly	Review canonical pages
Quarterly	Archive or supersede old pages
Before major use	Run citation checks on pages used for reports, articles, or decisions

The schedule should match risk. A personal research wiki can be lighter. A wiki used for technical publishing, customer support, or engineering decisions needs more discipline.

Using an LLM Wiki for Technical Publishing

For a technical blog or documentation site, the LLM Wiki can become the knowledge layer behind article planning.

It can track:

Canonical explanations
Repeated definitions
Internal link opportunities
Stale comparisons
Article gaps
Topic clusters
Source notes
Decisions about structure
Claims that need verification

This is useful, but it raises the maintenance bar.

If the wiki feeds published articles, then stale wiki pages can become stale public content. If the wiki contains contradictions, those contradictions may leak into articles. If citations are weak, generated drafts will inherit that weakness. The scoped summaries, schema-based extraction, and human review loops described in AI for Knowledge Management: Real Workflows That Hold Up are a good template for keeping that editorial pipeline honest.

For publishing workflows, treat the LLM Wiki as an editorial system, not just a private notebook. A digital garden makes this explicit by showing readers which pages are still growing and which are mature, which is exactly the kind of status signal a well-maintained LLM Wiki should also expose.

Maintenance Is Not Optional

An LLM Wiki is attractive because it promises compounding knowledge.

But compounding only works if old knowledge remains useful. Otherwise, the system compounds noise, not insight.

The maintenance burden does not disappear because an agent is involved. It changes shape.

Humans should not have to do every tedious task. Agents can check links, find stale pages, compare claims, draft reports, and propose updates. But humans still need to set policy, review risky changes, and decide what the wiki is allowed to mean.

That is the real bargain.

Conclusion

LLM Wiki maintenance is about keeping compiled knowledge honest.

The important practices are not complicated: preserve sources, cite claims, keep indexes current, detect contradictions, review Git diffs, mark stale pages, and avoid broad unreviewed rewrites.

A good LLM Wiki does not pretend to be magically self-correcting. It makes correction easier.

That is the standard worth aiming for: not perfect knowledge, but inspectable knowledge that can be reviewed, repaired, and trusted over time.

Syncthing File Sync for Self-Hosted Knowledge Systems

Rost — Sun, 19 Jul 2026 12:19:06 +0000

Syncthing keeps files synchronized across devices you control, making it one of the most practical tools for a self-hosted knowledge infrastructure that avoids cloud lock-in.

Unlike cloud storage platforms, Syncthing uses a peer-to-peer model where each device holds its own copy of synced folders and exchanges changes directly with trusted peers. There is no central server that owns your data, no subscription account, and no vendor lock-in. The project is open-source and community-driven, with more details at syncthing.net.

This architecture makes Syncthing especially useful for knowledge workers who manage markdown notes, research documents, PDFs, and project files across a desktop, laptop, home server, and possibly a phone. The tool is simple in concept but requires careful setup to avoid common pitfalls like treating sync as backup or syncing folders that should remain isolated.

What Syncthing Is and Is Not

Syncthing synchronizes files between two or more devices. Each device maintains its own copy of a folder, and changes propagate between trusted peers. Discovery and relay services may help devices find each other across networks, but the storage model remains local-first. The Syncthing documentation covers installation and configuration in detail.

The calm but important opinion is this: Syncthing is excellent when treated as sync infrastructure. It becomes dangerous when treated as backup.

It is not a cloud drive. It is not a complete backup system. It is not a collaboration suite. It is a private, peer-to-peer file synchronization tool.

Why Syncthing Matters for Knowledge Management

Knowledge management is not only about note-taking. It is also about where knowledge lives, how it moves, and whether it remains accessible over time — see the knowledge management guide for the broader picture of tools, methods, and self-hosted platforms this fits into.

A useful personal or team knowledge system often contains:

markdown notes
PDFs and papers
diagrams and screenshots
exported web pages
source snippets and configuration files
meeting notes and project documents
scanned documents and plain text logs

Many of these are just files. That is good. Files are durable, portable, searchable, scriptable, and easy to back up.

Syncthing gives those files movement without forcing them into a vendor platform. You can write notes on one machine, read them on another, keep a copy on a home server, and still use normal tools like grep, ripgrep, Obsidian, VS Code, DokuWiki imports, static site generators, or custom scripts.

For a self-hosted knowledge system, that is a strong architectural property. See personal knowledge management foundations for a broader view of PKM goals and methods, and PKM vs RAG vs Wiki vs Memory Systems for how different knowledge systems operate at different layers.

Distinct System Roles

graph LR
    S["Syncthing<br/>moves files"] --> B["Backup<br/>preserves history"]
    S --> K["Knowledge tools<br/>create and edit"]
    B --> R["Recovery<br/>when things go wrong"]
    K --> S

Keep those roles separate.

Syncthing should not be your only copy of important data. It should not be the only thing standing between you and accidental deletion. It should not be the only recovery mechanism after corruption, ransomware, filesystem failure, or a bad script.

Used correctly, Syncthing is part of a resilient file workflow. Used alone, it can replicate mistakes very efficiently.

Good Use Cases for Syncthing

Syncthing works best when the folder has a clear owner, a predictable structure, and a small number of trusted devices.

Good use cases include:

Syncing an Obsidian or markdown notes vault
Syncing project notes between desktop and laptop
Syncing documents to an always-on home server
Syncing exported PDFs and research material
Syncing configuration files across personal machines
Syncing scanned documents from one machine to another
Syncing selected phone folders to a desktop or NAS
Syncing static-site source notes before publishing
Syncing knowledge archives between a workstation and server

These are not exotic use cases. They are exactly the kind of boring file movement that knowledge workers need every day.

Syncthing is strongest when the folder remains understandable without Syncthing. If you can open the folder in a file manager and understand what it is, you are probably using the tool well.

Risky Use Cases for Syncthing

Syncthing becomes risky when users expect it to behave like backup, collaboration software, or managed cloud storage.

Be careful with:

Using Syncthing as your only backup
Syncing huge folders without thinking about deletion risk
Syncing the same files while multiple apps edit them
Syncing application databases that expect exclusive local access
Syncing browser profiles
Syncing mail stores
Syncing build directories or cache folders
Syncing folders with frequent generated files
Syncing very large photo libraries to low-storage devices
Relying on mobile background sync without testing it

The issue is not that Syncthing is unreliable. The issue is that sync is powerful. It does what you ask, including syncing deletions, conflicts, corrupted files, and accidental edits.

That is why a knowledge system should combine Syncthing with versioning, snapshots, and real backup.

Syncthing Is Not Backup

This point deserves its own section.

Backup is about recovery. Sync is about convergence. These are related, but they are not the same.

If you delete a file on one device, a sync tool may delete it everywhere. If a script corrupts a folder, the corrupted version may sync to other devices. If ransomware encrypts local files, the encrypted versions may be treated as changed files.

File versioning can reduce this risk, but it does not turn Syncthing into a complete backup system. Syncthing versioning is configured per folder and per device, and it mainly protects old versions when changes are received from other devices. It does not magically preserve every local edit before it happens.

A safer design is:

Syncthing for active file movement.
File versioning for short-term mistake recovery.
Filesystem snapshots for local rollback.
Restic, Borg, ZFS send, Btrfs snapshots, or another backup system for real recovery.

That is more boring than pretending sync is backup. It is also much safer.

Recommended Knowledge Workflow

A practical self-hosted knowledge setup can look like this:

graph TD
    subgraph "Desktop Workstation"
        D["Main editing device<br/>Full notes and documents"]
    end
    subgraph "Laptop"
        L["Mobile editing device<br/>Same knowledge folders"]
    end
    subgraph "Home Server or NAS"
        H["Always-on sync target<br/>Receive-side versioning<br/>Filesystem snapshots<br/>Separate backup job"]
    end
    subgraph "Phone"
        P["Selected folders only<br/>Camera scans or quick capture<br/>Avoid full archive sync"]
    end
    D <--> H
    L <--> H
    P --> H

This shape works because the always-on server becomes the stable point in the system. It does not need to be a central cloud server, but it gives your sync topology a reliable anchor.

For knowledge management, that anchor matters. Laptops sleep. Phones throttle background services. Desktops are not always on. A small home server or NAS gives the system somewhere steady to converge.

Folder Design Matters

Do not create one giant "sync everything" folder.

Create separate folders for separate purposes:

knowledge-notes
knowledge-documents
research-papers
project-notes
scans-inbox
static-site-drafts
configs

This gives you better control over:

Which devices receive which files
Which folders need versioning
Which folders need snapshots
Which folders can be send-only
Which folders should avoid mobile devices
Which folders have privacy or size concerns

Folder boundaries are architecture. They define trust, storage, recovery, and operational behavior.

For example, an Obsidian vault might sync to laptop, desktop, and server. A large PDF archive might sync only to desktop and server. A phone scan inbox might be send-only from the phone to the server.

That is much cleaner than treating every device as equal.

Understanding Syncthing Folder Types

Syncthing supports different folder behavior depending on how you want changes to flow.

Send and receive is the default mode. A device both sends local changes and receives remote changes. This is the normal choice for active editing across trusted devices.

Send-only sends local changes to other devices but does not accept remote changes as authoritative. This can be useful for source folders where one device should be treated as the origin.

Receive-only receives changes but does not publish local changes to the rest of the cluster. This can be useful for mirrors, replication targets, and backup-adjacent destinations where local edits should not be sent back.

These modes are powerful, but they can also make the system harder to reason about. Use them intentionally. For most everyday folders, send and receive is simpler.

A Good Folder Layout for Notes

For markdown-based notes, keep the structure predictable.

Example:

knowledge-notes/
  inbox/
  projects/
  areas/
  references/
  archive/
  attachments/

This works well with tools like Obsidian, VS Code, ripgrep, static site generators, and command-line scripts.

Avoid putting temporary exports, generated indexes, and application caches into the same folder unless you really want them synced. Use ignore patterns for files that are noisy or machine-specific.

Examples of files you may want to ignore:

.DS_Store
Thumbs.db
*.tmp
*.swp
.cache/
node_modules/
dist/
build/

For Obsidian specifically, think carefully before syncing all plugin state. Some settings are useful across devices, but workspace state can be annoying if it constantly changes window layouts or active panes.

Syncthing and Obsidian

Syncthing is often used to sync Obsidian vaults because Obsidian stores notes as local markdown files.

This is a good pairing. Obsidian gives you the writing and linking interface. Syncthing moves the files. Your notes remain plain files.

The main risks are conflicts and mobile behavior.

If you edit the same note on two devices before they sync, Syncthing may create conflict files. That is better than silent data loss, but it still requires cleanup. The practical habit is to let devices sync before editing the same active note elsewhere.

For mobile devices, do not assume background sync behaves exactly like desktop sync. Test it. Open the app, let sync complete, edit a note, and verify the change appears on your other devices.

See the Obsidian vs Logseq comparison for a broader discussion of sync considerations in PKM tools.

Syncthing and DokuWiki or Static Knowledge Sites

Syncthing also works well around file-based or file-friendly knowledge systems.

For DokuWiki, you might use Syncthing to move exported documents, media files, or staging content between machines. Be more careful with live server data, permissions, and concurrent edits. See DokuWiki and self-hosted wiki alternatives for platform options.

For Hugo or other static site generators, Syncthing can be useful for drafts, research notes, and content source files. It should not replace Git for source control, but it can complement Git for non-code knowledge material.

A good rule is:

Use Git for history and collaboration.
Use Syncthing for private file movement.
Use backup for recovery.

Each tool has a job.

Syncthing vs Nextcloud

Syncthing and Nextcloud are often compared, but they solve different problems.

Feature	Syncthing	Nextcloud
Architecture	Peer-to-peer sync	Central server
Best for	Private device-to-device folders	Web-accessible file platform
Storage model	Local copies on devices	Server-first storage
Web UI	Minimal local admin UI	Full web file interface
Sharing links	Not the main purpose	Built in
Users and permissions	Device trust model	User and group model
Calendars and contacts	No	Yes
Office collaboration	No	Possible with add-ons
Knowledge notes	Good for local-first notes	Good for server-centered sharing
Operational style	Lightweight but manual	Heavier but more complete

Use Syncthing when you want private file sync between your own devices.

Use Nextcloud when you want a self-hosted cloud platform with users, web access, sharing, calendars, contacts, and broader collaboration features. See the Nextcloud self-hosting guide for a detailed setup walkthrough.

Use both if you have both needs. For example, Syncthing can handle your local-first notes, while Nextcloud handles family file sharing or browser-based access.

Syncthing vs rsync

Rsync is excellent for one-off or scheduled file copying. It is simple, scriptable, and widely available.

Syncthing is better when you want continuous synchronization across multiple devices without writing your own scheduling, conflict detection, and device discovery logic.

Use rsync for:

Scripted deployments
One-way copies
Server maintenance
Simple backup jobs
Predictable batch transfers

Use Syncthing for:

Continuous multi-device sync
Local-first notes
Personal document movement
Always-on folder convergence
Cross-platform device sync

Rsync is a tool. Syncthing is a small synchronization system.

Syncthing vs Seafile

Seafile is closer to a file sync platform. It can be a good fit when you want a central service, clients, libraries, and a more managed team file sync experience.

Syncthing is more decentralized and simpler in concept. There is no main server that owns the truth. Devices share folders with each other.

For personal knowledge management, Syncthing is usually easier to reason about if you already like local files. For team file sharing, Seafile or Nextcloud may be more appropriate.

The question is not which tool is better. The question is whether you want local-first peer sync or a central file platform.

Android in 2026

The Android story needs caution.

The original Syncthing Android app was discontinued after the December 2024 release. Community forks and alternative approaches may exist, but Android should not be treated as the most stable part of a Syncthing knowledge system.

This does not mean Syncthing is useless on Android. It means you should design mobile sync as a convenience layer, not the only reliable copy.

For Android, prefer narrower use cases:

Scan inbox
Camera import folder
Quick notes folder
Read-only reference folder
Selected documents

Avoid making your phone responsible for the only complete sync path of your knowledge archive. Phones are battery-managed, storage-limited, and increasingly restrictive about background file access.

The boring recommendation is best: keep the authoritative knowledge set on desktop, laptop, and server. Let the phone participate selectively.

Versioning Strategy

Syncthing supports file versioning, and you should usually enable it on at least one stable device.

For a knowledge-management setup, the home server or NAS is often the best place for versioning. It is always on, has more storage, and is easier to include in backup jobs.

Common versioning approaches include:

Trash can versioning — moved files go to a trash folder
Simple versioning — keeps a fixed number of old versions
Staggered versioning — keeps more versions near the present, fewer as files age
External versioning — delegates versioning to an external tool

For most personal knowledge folders, staggered versioning is a reasonable starting point. It keeps more versions near the present and fewer versions as files get older.

Versioning is not free. It consumes storage and needs occasional review. But storage is cheaper than losing a year of notes.

Snapshots and Backup

If your Syncthing target is a NAS or Linux server, add snapshots.

Good options include:

ZFS snapshots
Btrfs snapshots
LVM snapshots
restic
BorgBackup
Kopia
Filesystem-level backup tools from your NAS platform

A strong setup might be:

Syncthing syncs files to the server.
The server keeps Syncthing file versions.
The filesystem keeps snapshots.
A backup tool copies encrypted backups off-device.

This gives you multiple recovery layers.

If you accidentally delete a note, Syncthing versioning may help. If a folder is corrupted, filesystem snapshots may help. If the server disk dies, off-device backup may help.

That is the difference between a sync setup and a recovery strategy.

Privacy and Trust

Syncthing is attractive because it lets you avoid placing all knowledge files into a third-party cloud account.

However, privacy is not automatic. You still need to think about device trust.

Every normal trusted device that participates in a folder can read that folder. If you sync your notes to an old laptop, that laptop is now part of your security boundary. If you sync documents to a VPS, that server matters too.

Syncthing also has an untrusted encrypted device feature, but treat it carefully. It can be useful when you want an encrypted replica on a device that should not see plaintext file content, but it is more advanced and should be tested before relying on it.

For most people, the simpler model is better:

Only sync sensitive folders to devices you actually trust.
Encrypt disks on laptops and servers.
Back up important data separately.
Do not sync private archives everywhere.

Self-hosted does not automatically mean secure. It means you own the responsibility.

Conflict Handling

Conflicts happen when different devices change the same file before synchronization converges.

For notes, this can happen if you edit the same markdown file on a laptop and desktop while one of them is offline. Syncthing will preserve conflict copies rather than silently choosing one version.

That is the right behavior, but it still leaves you with cleanup work.

To reduce conflicts:

Avoid editing the same note on two offline devices
Let sync finish before switching machines
Keep frequently edited inbox notes small
Avoid syncing application state files unnecessarily
Use ignore patterns for volatile files
Review conflict files periodically

Conflict files are not a sign that Syncthing is broken. They are a sign that two devices changed reality at the same time.

Ignore Patterns

Ignore patterns are important for knowledge folders.

They keep generated, temporary, or machine-specific files out of the sync set. This reduces conflicts, saves bandwidth, and avoids polluting other devices.

Common examples:

.DS_Store
Thumbs.db
*.tmp
*.swp
*.bak
.cache/
node_modules/
dist/
build/
__pycache__/

For a notes vault, consider whether plugin caches, workspace layout files, or generated indexes should really sync. Some should. Some should not.

The principle is simple: sync source material, not noise.

Suggested Syncthing Topologies

For one person, a star-like topology around a home server is often easiest.

desktop <-> home-server
laptop  <-> home-server
phone   -> home-server

The devices can still connect directly, but the server gives them a stable meeting point.

For a small household, keep personal folders separate. Do not create one shared mega-folder unless everyone understands the consequences.

For a small technical team, be careful. Syncthing can work for shared files among trusted peers, but it does not replace proper collaboration tools, permissions, review workflows, or version control.

The more people you add, the more attractive a server-centered platform like Nextcloud, Seafile, Git, or a document system becomes.

Practical Setup Example

Here is a reasonable self-hosted knowledge setup.

Folders:
  knowledge-notes
  knowledge-documents
  research-papers
  scans-inbox

Devices:
  desktop
  laptop
  home-server
  phone

Rules:
  desktop and laptop use send-receive for notes
  home-server uses send-receive with versioning enabled
  phone sends scans-inbox only
  large PDFs do not sync to phone
  home-server snapshots all synced folders
  home-server backup runs nightly

This setup is not fancy, but it is robust.

It supports local-first work, mobile capture, server-side recovery, and off-device backup. It avoids making the phone responsible for the whole archive. It also avoids confusing sync with backup.

Operational Checklist

Before trusting Syncthing with important knowledge files, check the basics.

Are all important folders synced to at least two real devices?
Is one device always on or frequently online?
Is file versioning enabled on at least one stable device?
Are filesystem snapshots enabled on the server or NAS?
Is there an off-device backup?
Are noisy files ignored?
Are mobile folders limited?
Have you tested restore?
Have you tested conflict behavior?
Do you understand which folders are send-only or receive-only?

The most important item is restore testing.

A backup strategy you have never restored from is a theory. A sync strategy you have never tested under conflict is also a theory.

Common Mistakes

The most common mistake is syncing too much.

People start with one useful folder, then add every document, every photo, every export, every cache, and every application directory. The system becomes noisy and hard to reason about.

The second mistake is treating receive-only folders as magical backup. They are useful, but they are not a full historical recovery system.

The third mistake is ignoring deletion behavior. If deletion syncs everywhere, then deletion is part of the design.

The fourth mistake is trusting mobile sync too much. Mobile operating systems are not friendly to long-running background file synchronization.

The fifth mistake is not using versioning. If the data matters, keep versions somewhere.

The sixth mistake is not having a real backup. Syncthing can help populate a backup target, but it should not be the only protection.

Where Syncthing Fits in a Self-Hosted Knowledge Stack

In a self-hosted knowledge system, Syncthing is infrastructure.

It sits below note-taking apps, wikis, search tools, static sites, scripts, and AI-assisted workflows. It moves the files those systems use.

A simple stack might look like this:

graph TB
    subgraph "Capture"
        C1["Phone scanner"]
        C2["Browser save"]
        C3["Quick notes"]
    end
    subgraph "Sync"
        S["Syncthing"]
    end
    subgraph "Storage"
        ST1["Desktop"]
        ST2["Laptop"]
        ST3["Home server / NAS"]
    end
    subgraph "Authoring"
        A1["Obsidian"]
        A2["VS Code"]
        A3["Vim"]
        A4["DokuWiki"]
        A5["Static site generator"]
    end
    subgraph "Search"
        SR1["ripgrep"]
        SR2["Desktop search"]
        SR3["Local index"]
        SR4["RAG pipeline"]
    end
    subgraph "Recovery"
        R1["Versioning"]
        R2["Snapshots"]
        R3["Encrypted backup"]
    end
    C1 --> S
    C2 --> S
    C3 --> S
    S --> ST1
    S --> ST2
    S --> ST3
    ST1 --> A1
    ST1 --> A2
    ST2 --> A3
    ST3 --> A4
    ST3 --> A5
    A1 --> SR1
    A2 --> SR2
    A3 --> SR3
    A4 --> SR4
    ST3 --> R1
    ST3 --> R2
    ST3 --> R3

This separation is healthy.

You can replace the editor without replacing the storage. You can replace the backup tool without replacing the note format. You can stop Syncthing and still have ordinary files.

That is the value of local-first knowledge infrastructure.

When Syncthing Is the Right Tool

Syncthing is a strong choice when the problem is private file synchronization between devices you control. It is less suitable when the real problem is collaboration, web access, permissions, or long-term recovery.

This distinction matters because many self-hosted file workflows fail from unclear expectations. A sync tool can move files very well, but it should not be asked to behave like a cloud suite, a backup archive, and a team collaboration platform at the same time.

Need	Syncthing fit	Better fit when Syncthing is not enough
Private sync between your own devices	Excellent	Usually none needed
Local-first markdown notes	Excellent	Obsidian Sync if you prefer managed sync
Personal document folders	Good	Nextcloud if you need browser access
Family file sharing	Possible, but awkward	Nextcloud
Team document collaboration	Limited	Nextcloud, Seafile, Google Drive, Microsoft 365
One-way server copy	Good	rsync may be simpler
Full backup and restore	Not enough alone	restic, BorgBackup, Kopia
Web file access	Weak	Nextcloud or Seafile
Calendar and contacts	Not supported	Nextcloud
Mobile photo backup	Possible with caution	Immich, Nextcloud, platform photo tools
Encrypted offsite backup	Not the main role	restic, BorgBackup, Kopia
Encrypted sync to untrusted device	Advanced use case	Syncthing untrusted devices, tested carefully

This is the main reason Syncthing remains useful. It does not try to become a full cloud platform. It works best when the job is clear: keep selected folders synchronized across trusted devices.

Final Recommendation

Use Syncthing for self-hosted knowledge management when you want private, local-first file sync across trusted devices.

Use it for notes, documents, research folders, scans, and personal knowledge archives. Pair it with markdown, plain files, local search, and self-hosted services. Keep the structure simple enough that you can understand it without a dashboard.

But do not confuse sync with backup.

The best setup is not Syncthing alone. The best setup is Syncthing plus file versioning, filesystem snapshots, and real backup.

That combination gives you the thing self-hosted knowledge systems should aim for: local control, practical convenience, and recoverable data.

Circuit Breaker Pattern in Go: Stop Cascading Failures

Rost — Sat, 18 Jul 2026 07:49:30 +0000

A circuit breaker stops your Go service from hammering a failing dependency,
preventing cascading failures that consume goroutines, sockets, and memory until the entire system collapses.

The hard part is not the state machine. It is deciding where the breaker belongs, what counts as failure, how it interacts with timeouts and retries, and what your service should do when the circuit is open.

In Go, the circuit breaker pattern is especially useful around outbound calls: HTTP APIs, payment gateways, search services, email providers, LLM gateways, internal microservices, and other dependencies that can become slow, overloaded, or partially unavailable. Used well, a circuit breaker reduces cascading failures. Used badly, it becomes another obscure failure mode.

What Problem Does a Circuit Breaker Solve?

Distributed systems rarely fail cleanly.

A dependency might not be fully down. It might be:

returning 500 errors
returning 429 rate limit responses
accepting TCP connections but never replying
responding in 30 seconds instead of 300 milliseconds
failing only for some requests
overloaded because every client is retrying at once

The worst case is often not a hard failure. It is a slow dependency.

Slow calls consume goroutines, sockets, database connections, memory, and worker capacity. If your service keeps waiting on a dependency that is already unhealthy, your service can become unhealthy too.

A circuit breaker prevents that by failing fast after the dependency crosses a failure threshold.

Instead of doing this forever:

request -> call dependency -> wait -> timeout -> retry -> wait -> fail

the service eventually does this:

request -> circuit open -> return fallback or error immediately

That fast failure is not always pleasant, but it is predictable. Predictable failure is easier to operate than a slow collapse.

The Three Circuit Breaker States

Most circuit breakers use three states.

Closed

The circuit is closed during normal operation.

Requests are allowed through. The breaker records successes and failures. If the number or ratio of failures crosses a threshold, the breaker opens.

Closed does not mean "safe forever." It means "traffic is currently allowed."

Open

The circuit is open when the dependency is considered unhealthy.

Requests are rejected immediately. The service should return a fallback, cached response, degraded response, or a clear upstream error.

Open does not fix the dependency. It gives the dependency time to recover and protects the caller from wasting resources.

Half-Open

After a cool-down period, the breaker enters a half-open state.

Only a limited number of trial requests are allowed through. If they succeed, the breaker closes. If they fail, the breaker opens again.

Half-open is important because it avoids two bad extremes:

never trying the dependency again
sending full traffic back too quickly

The state transitions look like this:

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: Failure threshold reached
    Open --> HalfOpen: Timeout elapsed
    HalfOpen --> Closed: Trial succeeds
    HalfOpen --> Open: Trial fails

Circuit Breaker vs Timeout vs Retry

A common mistake is treating circuit breakers, retries, and timeouts as interchangeable. They are related, but they solve different problems.

Timeout

A timeout limits how long one operation can run.

In Go, this usually means passing a context.Context with a deadline or timeout into the outbound call.

A timeout answers this question:

How long am I willing to wait for this one call?

Retry

A retry repeats an operation when the failure might be temporary.

Retries are useful for short network glitches, temporary 503 responses, connection resets, and other transient failures.

A retry answers this question:

Should I try this call again?

Circuit Breaker

A circuit breaker stops calls when the dependency is probably unhealthy.

It answers this question:

Should I call this dependency at all right now?

Rate Limiter

A rate limiter controls how much traffic is allowed over time.

It answers this question:

How much traffic should this caller send?

Bulkhead

A bulkhead isolates resources so one dependency cannot consume everything.

It answers this question:

How much of my service can this dependency damage?

These patterns are strongest when used together. A circuit breaker without timeouts is weak. Retries without jitter can create retry storms. A fallback without metrics can hide an outage.

When to Use a Circuit Breaker in Go

Use a circuit breaker when your service calls a dependency that can fail independently from your service.

Good candidates include:

external HTTP APIs
payment processors
email and SMS providers
search services
recommendation services
LLM inference gateways
internal microservice endpoints
third-party SaaS APIs
slow or overloaded read-side services

Circuit breakers are especially useful when the caller can degrade gracefully.

For example:

return cached product data
skip a recommendation block
mark a payment provider as temporarily unavailable
queue work for later
return a partial response
fail fast with a clear temporary error

The important question is not "can this call fail?" Everything can fail. The better question is:

If this dependency is failing, should we continue sending full traffic to it?

If the answer is no, a circuit breaker may help.

When Not to Use a Circuit Breaker

Do not add a circuit breaker to every function just because the pattern sounds responsible.

A circuit breaker is usually not useful for:

local in-process function calls
simple CRUD inside a monolith
validation logic
deterministic business rules
CPU-only local operations
code paths where no useful fallback exists
write operations that are not idempotent
dependencies already protected by a stronger workflow layer

A circuit breaker also does not replace basic hygiene:

set timeouts
propagate context
use connection pools correctly
handle errors explicitly
make retries safe
observe failure rates

A bad circuit breaker can make a system harder to reason about. It can hide the real problem, reject traffic too aggressively, or create confusing behavior during recovery.

The slightly opinionated rule is simple:

Add circuit breakers at dependency boundaries, not everywhere.

Choosing a Go Circuit Breaker Library

You can implement a basic circuit breaker yourself, but most production Go services should use a library.

The most common simple choice is sony/gobreaker.

It gives you:

closed, open, and half-open states
configurable failure thresholds
configurable open-state timeout
state change callbacks
request counters
generic support in v2
a small API surface

For larger resilience pipelines, you may also look at libraries that compose multiple policies, such as retry, timeout, fallback, rate limiting, bulkhead isolation, and circuit breaking. That can be useful when you want a single resilience layer around an operation.

For many Go services, though, gobreaker is enough.

Go Circuit Breaker Packages Compared

Go does not include a built-in circuit breaker in the standard library. In practice, you usually choose between a small circuit breaker library, a larger resilience framework, or an older Hystrix-style package.

For most new Go services, the decision is simple:

use sony/gobreaker if you want a small, focused circuit breaker
use failsafe-go if you want circuit breakers composed with retries, timeouts, fallbacks, bulkheads, rate limits, and other resilience policies
avoid starting new projects on hystrix-go unless you already have legacy code using it

Package	Best for	Strengths	Tradeoffs
`sony/gobreaker/v2`	Simple circuit breakers around HTTP/RPC clients	Small API, generic v2 support, clear state model, easy to wrap dependency clients	Only solves circuit breaking; retries, timeouts, and fallbacks must be composed separately
`failsafe-go`	Full resilience policy composition	Retry, fallback, circuit breaker, timeout, bulkhead, rate limiter, cache, hedge, adaptive limiter, and adaptive throttler policies	More concepts to learn; heavier than needed if you only want a basic breaker
`afex/hystrix-go`	Legacy Hystrix-style systems	Familiar Hystrix concepts, command-style execution, historical usage	Older design; not the best default for new Go services
`go-kit/kit/circuitbreaker`	Go kit endpoint-based services	Fits Go kit middleware style and endpoint architecture	Mostly useful if your service already uses Go kit
`cep21/circuit`	Hystrix-like circuit breaker behavior	More featureful Hystrix-style approach	Less common as the simple default; may be more than needed for small services

My default recommendation is boring on purpose: start with sony/gobreaker/v2 when you only need a circuit breaker. Reach for failsafe-go when you want to express a complete resilience policy in one place.

That split keeps the architecture clean. A small service client does not need a full resilience framework just to stop calling a failing dependency. But a gateway, aggregator, API client SDK, or high-traffic integration layer may benefit from composed policies.

Installing gobreaker

Use the v2 package for new code:

go get github.com/sony/gobreaker/v2

Then import it:

import "github.com/sony/gobreaker/v2"

A Basic Circuit Breaker in Go

Here is a small example around an HTTP call.

package main

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "time"

    "github.com/sony/gobreaker/v2"
)

var ErrTemporaryUnavailable = errors.New("dependency temporarily unavailable")

type UserClient struct {
    baseURL string
    http    *http.Client
    cb      *gobreaker.CircuitBreaker[[]byte]
}

func NewUserClient(baseURL string) *UserClient {
    settings := gobreaker.Settings{
        Name:        "user-service",
        MaxRequests: 3,
        Interval:    30 * time.Second,
        Timeout:     10 * time.Second,
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            return counts.ConsecutiveFailures >= 5
        },
        OnStateChange: func(name string, from gobreaker.State, to gobreaker.State) {
            fmt.Printf("circuit breaker %s changed from %s to %s\n", name, from, to)
        },
    }

    return &UserClient{
        baseURL: baseURL,
        http: &http.Client{
            Timeout: 3 * time.Second,
        },
        cb: gobreaker.NewCircuitBreaker[[]byte](settings),
    }
}

func (c *UserClient) GetUser(ctx context.Context, userID string) ([]byte, error) {
    result, err := c.cb.Execute(func() ([]byte, error) {
        req, err := http.NewRequestWithContext(
            ctx,
            http.MethodGet,
            c.baseURL+"/users/"+userID,
            nil,
        )
        if err != nil {
            return nil, err
        }

        resp, err := c.http.Do(req)
        if err != nil {
            return nil, err
        }
        defer resp.Body.Close()

        if resp.StatusCode >= 500 {
            return nil, fmt.Errorf("user service returned %d", resp.StatusCode)
        }

        if resp.StatusCode == http.StatusNotFound {
            return nil, fmt.Errorf("user not found")
        }

        if resp.StatusCode >= 400 {
            return nil, fmt.Errorf("user service client error: %d", resp.StatusCode)
        }

        return io.ReadAll(resp.Body)
    })

    if errors.Is(err, gobreaker.ErrOpenState) {
        return nil, ErrTemporaryUnavailable
    }

    if errors.Is(err, gobreaker.ErrTooManyRequests) {
        return nil, ErrTemporaryUnavailable
    }

    return result, err
}

This is not a complete production client, but it shows the shape:

the breaker wraps the outbound call
the HTTP request receives a context
the HTTP client has a timeout
server-side failures count as breaker failures
open circuit errors are mapped into an application error

Configuring gobreaker Settings

The key settings are worth understanding.

Name

Name identifies the breaker.

Use a stable, specific name:

payment-api
search-service
llm-gateway
user-service

Avoid vague names like:

http-client
external-call
default

You will want this name in logs and metrics.

MaxRequests

MaxRequests controls how many requests are allowed while the breaker is half-open.

A small number is usually safer. The purpose of half-open is to test recovery, not to send full traffic immediately.

Interval

Interval controls when internal counts are cleared while the breaker is closed.

If it is zero, counts are not cleared automatically. A non-zero interval gives the breaker a rolling-ish memory window, although it is not the same as a full sliding window implementation.

Timeout

Timeout controls how long the breaker stays open before moving to half-open.

If the timeout is too short, your service will keep probing a dependency that has not recovered. If it is too long, recovery will be delayed.

Start with something conservative, such as 10 to 30 seconds, then tune from production metrics.

ReadyToTrip

ReadyToTrip decides when the breaker should open.

A simple rule is consecutive failures:

ReadyToTrip: func(counts gobreaker.Counts) bool {
    return counts.ConsecutiveFailures >= 5
}

That is easy to reason about, but it may not be right for high-volume services.

Another option is failure ratio after a minimum number of requests:

ReadyToTrip: func(counts gobreaker.Counts) bool {
    total := counts.Requests
    failures := counts.TotalFailures

    if total < 20 {
        return false
    }

    return float64(failures)/float64(total) >= 0.5
}

This avoids opening the circuit after a tiny sample size.

OnStateChange

OnStateChange is where you should emit logs or metrics.

At minimum, record:

breaker name
old state
new state
timestamp

For production systems, expose breaker state as a metric. Logs are useful for debugging, but metrics are better for alerting and dashboards.

IsSuccessful

IsSuccessful lets you decide which errors count as failures.

This is important.

Not every error should open the breaker. For example, a 404 Not Found from a user service may be a valid business result. A 400 Bad Request might be the caller's fault, not the dependency's fault.

A 503 Service Unavailable, timeout, connection reset, or 429 Too Many Requests may be a real dependency health signal.

Be careful here. Counting the wrong errors is one of the easiest ways to build a noisy circuit breaker.

What Should Count as Failure?

This is where engineering judgement matters.

Usually count these as failures:

network timeouts
connection refused
connection reset
HTTP 500
HTTP 502
HTTP 503
HTTP 504
repeated 429 responses
malformed responses from the dependency
context deadline exceeded during the outbound call

Usually do not count these as dependency failures:

validation errors
local serialization errors
expected 404 responses
caller-side authorization failures
business rule rejections
user input errors

The breaker should represent dependency health, not general application failure.

Circuit Breakers and context.Context

In Go, circuit breakers should not replace context.Context.

A circuit breaker decides whether to attempt a call. A context controls how long that call may run and whether it should stop when the caller is gone.

A good outbound call should usually have both:

ctx, cancel := context.WithTimeout(parentCtx, 2*time.Second)
defer cancel()

data, err := client.GetUser(ctx, userID)

The context should flow through the call chain:

incoming request context
-> service method
-> client method
-> HTTP request
-> dependency

Avoid creating detached background contexts inside request-scoped code. If the user request is canceled, the downstream work should usually stop too.

The calm rule is:

The breaker protects the system. The context protects the request.

You normally need both.

Circuit Breakers and Retries

Retries and circuit breakers can work well together, but the order matters.

The safest default is:

timeout per attempt
retry with backoff and jitter
circuit breaker around the dependency call

But there is no universal answer. Think about what you want to count.

If each retry attempt passes through the breaker, one user request can contribute multiple failures. That may open the breaker faster, which can be good or bad.

If the breaker wraps the whole retry operation, the breaker sees one final success or failure per user request. That is calmer, but it may hide the number of failed attempts.

For many application services, this shape is reasonable:

user request
-> circuit breaker
   -> retry policy
      -> one HTTP attempt with timeout

That means the breaker tracks whether the dependency operation ultimately worked for the caller.

For lower-level clients, this shape can also make sense:

user request
-> retry policy
   -> circuit breaker
      -> one HTTP attempt with timeout

That means the breaker protects each attempt.

The more important rule is this:

Do not retry blindly.

Use:

a small maximum retry count
exponential backoff
jitter
per-attempt timeouts
an overall request deadline
idempotency for writes
metrics for retry attempts

Without those, retries can turn a small outage into a larger one. For a deeper treatment of retry safety, see Idempotency in Distributed Systems That Actually Works.

Circuit Breakers and Idempotency

Circuit breakers often appear next to retries, and retries raise the question of idempotency.

For read operations, retrying is usually safe.

For write operations, retrying can be dangerous.

Consider this payment call:

POST /charge

If the request times out, did the payment fail? Maybe. Did it succeed but the response was lost? Also maybe.

If you retry without an idempotency key, you might charge twice.

For write operations, use one or more of these:

idempotency keys
request IDs
operation IDs
unique constraints
transactional outbox
workflow orchestration
explicit reconciliation

A circuit breaker can stop you from continuing to call a failing payment provider, but it cannot make unsafe retries safe.

Circuit Breakers and Fallbacks

When the circuit is open, your service needs a plan.

Possible fallback strategies include:

return cached data
return stale data with a warning
omit a non-critical section
queue work for later
switch to another provider
return a temporary error
show degraded functionality
fail the request quickly

A fallback should be honest.

For example, this is usually good:

{
  "status": "temporary_unavailable",
  "message": "Recommendations are temporarily unavailable"
}

This is risky:

{
  "recommendations": []
}

An empty list may look like a valid result. It can hide an outage, confuse users, and make debugging harder.

Silent fallbacks are tempting. They are also dangerous.

Circuit Breakers and Observability

A circuit breaker without observability is mostly a surprise generator.

Track at least these metrics:

current breaker state
state changes
calls allowed
calls rejected
successes
failures
timeouts
fallback responses
retry attempts
downstream latency
downstream status codes

Useful labels include:

breaker name
dependency name
operation name
status class
error category

Avoid high-cardinality labels such as user ID, full URL, request ID, or raw error messages.

You should be able to answer these questions from dashboards:

Which circuit breakers are open right now?
How often do they open?
Which dependency caused the opening?
Are users seeing fallback responses?
Did latency improve after the breaker opened?
Did retry volume spike before the breaker opened?
Did the dependency recover?

If you cannot observe the breaker, you cannot tune it. For structured logging that pairs well with metrics, see Structured Logging in Go with slog.

A More Production-Friendly HTTP Client Shape

For real services, avoid scattering circuit breaker logic across handlers.

Create a small client package around the dependency.

Example structure:

internal/
  userservice/
    client.go
    errors.go
    metrics.go

The handler should not know the details of gobreaker. It should depend on a domain-level client method:

type UserService interface {
    GetUser(ctx context.Context, userID string) (*User, error)
}

Then the implementation can contain:

HTTP request creation
context propagation
breaker execution
status code handling
response decoding
metrics
error mapping

This keeps the resilience policy close to the dependency boundary. For more on error classification at boundaries, see Go Error Handling Architecture: Boundaries and Patterns.

Where Circuit Breakers Fit in Application Architecture

The circuit breaker pattern belongs at integration boundaries.

In a Go application, that usually means:

graph LR
    A[Handler] --> B[Application Service]
    B --> C[Dependency Client]
    C --> D[Circuit Breaker]
    D --> E[HTTP / RPC / DB / Queue]

Keep the breaker out of business logic when possible.

The business layer should understand domain errors like:

payment provider unavailable
recommendations unavailable
profile service timeout

It should not need to understand gobreaker states.

This separation keeps the architecture clean:

transport concerns stay in clients
resilience policy stays near dependencies
domain logic stays readable
handlers stay thin
tests are easier to write

Common Mistakes

Mistake 1: No Timeout

A circuit breaker does not magically stop slow calls unless the calls return.

If the outbound operation can hang forever, the breaker may not see a failure quickly enough.

Always use timeouts.

Mistake 2: One Global Breaker for Everything

Do not use one breaker for all dependencies.

A failing email provider should not open the circuit for your payment provider. A slow search endpoint should not block user profile calls.

Use separate breakers for separate dependency operations when their failure modes differ.

Mistake 3: Counting Caller Errors as Dependency Failures

If your service sends bad input and receives 400 Bad Request, that is usually not a downstream outage.

Do not train the breaker on your own bugs.

Mistake 4: Retrying Non-Idempotent Writes

Retries are not free. They can duplicate writes, payments, messages, or side effects.

Make writes idempotent before retrying them.

Mistake 5: Hiding Outages Behind Fallbacks

Fallbacks should degrade gracefully, not falsify reality.

If a dependency is down, your metrics and logs should make that obvious.

Mistake 6: Tuning Without Production Data

Thresholds copied from examples are only starting points.

Tune based on:

request volume
normal error rate
dependency latency
user impact
recovery time
fallback quality

Mistake 7: Using Circuit Breakers Instead of Capacity Management

A circuit breaker is not a substitute for:

load shedding
rate limiting
queue limits
autoscaling
database tuning
connection pool limits
upstream quotas

It is one part of a resilience strategy.

Practical Defaults

For a typical Go service calling an internal HTTP dependency, a reasonable starting point might be:

HTTP client timeout: 2 to 5 seconds
per-request context timeout: based on caller SLA
breaker failure rule: 5 consecutive failures or 50 percent failure after 20 requests
open timeout: 10 to 30 seconds
half-open requests: 1 to 5
retry count: 1 to 3 attempts
retry backoff: exponential with jitter

These are not universal values. They are safe-ish starting points.

For user-facing APIs, keep total latency budgets tight. For background jobs, you may tolerate longer waits. For payment providers, be much more careful with retries and idempotency.

Circuit Breaker Checklist

Before adding a circuit breaker, answer these questions:

What dependency is being protected?
What operation is being protected?
What errors count as dependency failure?
What errors should be ignored by the breaker?
What timeout applies to each call?
Are retries allowed?
Are writes idempotent?
What happens when the circuit is open?
Is there a fallback?
Is the fallback visible in metrics?
Who gets alerted if the circuit keeps opening?
How will the breaker be tuned after deployment?

If you cannot answer these, adding a breaker may create more confusion than resilience.

Testing Circuit Breakers in Go

Test behavior, not the internal state machine of the library.

Useful tests include:

dependency succeeds and response is returned
dependency fails repeatedly and circuit opens
open circuit returns a temporary error
client-side validation errors do not trip the breaker
context timeout is respected
fallback response is returned when expected
metrics are emitted on state changes

Use fake HTTP servers for integration-style tests:

server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    http.Error(w, "unavailable", http.StatusServiceUnavailable)
}))
defer server.Close()

For unit tests, hide the dependency behind an interface and inject a fake implementation.

Keep tests deterministic. Avoid sleeping for long real durations. Configure short breaker timeouts in tests. For more on testing concurrent Go code with fake time and isolated bubbles, see Testing Concurrent Go Code with testing/synctest.

Should You Build Your Own Circuit Breaker?

Building a small circuit breaker is a good learning exercise. It helps you understand the state machine.

For production code, prefer a maintained library unless your needs are very specific.

A production breaker needs to handle:

concurrency
state transitions
counters
half-open probes
callbacks
custom failure classification
race-free behavior
predictable error handling

That is not impossible, but it is easy to get subtly wrong.

The boring library is usually the better choice.

Conclusion

The circuit breaker pattern is not magic reliability dust.

In Go, it works best when it is part of a small, explicit resilience stack:

context timeout
+ retry with backoff and jitter
+ circuit breaker
+ fallback
+ metrics

The pattern is most useful at dependency boundaries, especially around remote services that can become slow or partially unavailable.

Use it to stop cascading failures. Use it to fail fast when a dependency is clearly unhealthy. Use it to give overloaded systems room to recover.

But do not use it as an excuse to ignore timeouts, idempotency, observability, or clean architecture.

A good circuit breaker makes failure clearer and cheaper. A bad one just makes failure more mysterious.

References

Go Microservices for AI/ML Orchestration — broader orchestration context where circuit breakers fit
Saga Pattern in Distributed Transactions — distributed transaction patterns that pair with circuit breakers
Idempotency in Distributed Systems — retry safety and idempotent operations
Transactional Outbox Pattern in Go — reliable event delivery alongside resilience patterns
Go Error Handling Architecture — error classification at dependency boundaries
Testing Concurrent Go Code with synctest — testing async behavior with circuit breakers
Structured Logging in Go with slog — observability alongside circuit breakers
github.com/sony/gobreaker/v2 — official gobreaker v2 package
Go Context Cancellation and Timeouts — context patterns that pair with circuit breakers

Podman Quadlet vs Docker Compose for Linux Services

Rost — Thu, 16 Jul 2026 10:13:30 +0000

Docker Compose and Podman Quadlet solve overlapping problems but come from different design centers, and choosing between them depends on whether you think in application stacks or Linux services.

The distinction matters for anyone running containers on a Linux host beyond a single afternoon of experimentation. Compose describes services, networks, and volumes in a YAML file and starts them with docker compose up. Quadlet describes containers in systemd-style unit files and lets the system service manager own the lifecycle.

Both approaches work for self-hosted services, internal tools, and small servers. The right choice comes down to your operational model, team familiarity, and whether you prefer a developer-friendly stack format or a systemd-native service model. This comparison covers the practical differences: file formats, lifecycle ownership, rootless containers, logging, updates, networking, security, and migration paths. It is part of Developer Tools: The Complete Guide to Modern Development Workflows.

Quick Recommendation

Use Docker Compose when:

You want the fastest multi-container workflow.
You already use Docker Engine.
You share stacks with developers.
You need a familiar compose.yaml for local development.
You deploy small services with Docker on one host.
You use existing Compose examples from projects.

Use Podman Quadlet when:

You want systemd-native container services.
You prefer rootless containers.
You do not want a central Docker daemon.
You run long-lived services on a Linux host.
You want systemctl, journalctl, timers, dependencies, and auto-start.
You are building a self-hosted or homelab server around systemd.
You want containers to fit into the normal Linux service model.

The practical rule:

Docker Compose is better for application stacks.
Podman Quadlet is better for Linux services.

That is not a law. It is a useful default.

Comparison Table

Area	Docker Compose	Podman Quadlet
Primary model	Multi-container application	systemd-managed container service
File format	YAML	systemd-like unit files
Runtime	Docker Engine	Podman
Daemon	Uses Docker daemon	Daemonless Podman model
Service manager	Compose manages stack lifecycle	systemd manages lifecycle
Best fit	Dev stacks, app bundles, simple deployments	Long-running Linux services
Rootless support	Possible, but not the default mental model	Strong fit
Logs	`docker compose logs`	`journalctl` and `podman logs`
Startup on boot	Usually via systemd wrapper or restart policy	Native systemd unit
Updates	`docker compose pull && docker compose up -d`	Podman auto-update or systemd workflows
Portability	Very high across Docker environments	Best on Linux with systemd
Learning curve	Easier for most developers	Easier for systemd users
Ecosystem examples	Huge	Smaller, but growing

Neither one is Kubernetes. Most small services do not need a cluster. They need a boring, understandable way to start, stop, update, log, and recover.

What Docker Compose Is Good At

Docker Compose is a tool for defining and running multi-container applications. A typical Compose file describes services, images, build contexts, ports, volumes, networks, environment variables, health checks, dependencies, and profiles.

services:
  web:
    image: nginx:stable
    ports:
      - "8080:80"
    restart: unless-stopped
    volumes:
      - ./html:/usr/share/nginx/html:ro

  redis:
    image: redis:7
    restart: unless-stopped

Run it:

docker compose up -d

Check status:

docker compose ps

Read logs:

docker compose logs -f

Stop it:

docker compose down

Compose is direct and productive. It is especially good when the unit of thought is "this application has several containers." For a comprehensive reference of Compose commands and patterns, see the Docker Compose Cheatsheet. For Docker commands beyond Compose — images, volumes, networks, and cleanup — see the Docker Cheatsheet.

What Podman Quadlet Is Good At

Podman Quadlet is a way to define Podman containers using systemd-style files. Instead of writing a full generated systemd service by hand, you write a declarative file:

[Unit]
Description=Example web container
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/library/nginx:stable
PublishPort=8080:80
Volume=/opt/example/html:/usr/share/nginx/html:ro

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

Save it as /etc/containers/systemd/example.container, then reload systemd:

sudo systemctl daemon-reload

Start it:

sudo systemctl enable --now example.service

Check it:

systemctl status example.service
journalctl -u example.service -f

The core appeal of Quadlet: the container becomes a normal Linux service.

The Philosophical Difference

Docker Compose Thinks in Stacks

Compose asks: What services make up this application?

A Compose project usually lives near application code:

myapp/
  compose.yaml
  .env
  app/
  db/

You start the project as a unit:

docker compose up -d

You update the project as a unit:

docker compose pull
docker compose up -d

This is simple, visible, and portable.

Podman Quadlet Thinks in Services

Quadlet asks: What containers should this Linux host run as services?

Quadlet files live in systemd-related container paths:

/etc/containers/systemd/
~/.config/containers/systemd/

You manage generated services with systemd:

systemctl status myapp.service
systemctl restart myapp.service
journalctl -u myapp.service -f

This feels more native on a Linux server. For general systemd service patterns, see Run any Executable as a Service in Linux.

The Important Difference: Who Owns Lifecycle?

With Docker Compose, Compose owns the application lifecycle. With Quadlet, systemd owns the service lifecycle.

This affects boot behavior, shutdown behavior, restart policy, dependency ordering, logs, health visibility, user services, updates, integration with timers, and integration with other host services.

If you already use systemd to manage everything else on the host, Quadlet fits neatly. If you think mainly in terms of application stacks, Compose is usually more comfortable.

Docker Compose Under systemd vs Quadlet

You can run Docker Compose as a systemd service. That is often a good pattern. Example systemd unit:

[Unit]
Description=MyApp Docker Compose stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecReload=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

This works well. But it is still a wrapper around Compose. systemd starts the Compose command, while Docker and Compose handle containers behind it.

With Quadlet, the unit generation is designed for Podman and systemd directly. You write container-oriented unit files, and systemd manages the generated services.

The distinction is subtle but important:

Docker Compose under systemd:
  systemd manages a Compose command.

Podman Quadlet:
  systemd manages generated container services.

For a detailed walkthrough of this pattern, see Run Docker Compose as a Linux Service with systemd.

File Format Comparison

Docker Compose YAML

Compose uses YAML. It is compact, popular, and easy to share. It is also indentation-sensitive and can grow messy when a stack becomes large.

services:
  app:
    image: ghcr.io/example/app:1.0.0
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      APP_ENV: production
    volumes:
      - app-data:/data

volumes:
  app-data:

Quadlet Unit Files

Quadlet uses systemd-like files. They are more verbose when you have many services, but readable if you already understand systemd.

[Unit]
Description=Example app container
After=network-online.target
Wants=network-online.target

[Container]
Image=ghcr.io/example/app:1.0.0
PublishPort=8080:8080
Environment=APP_ENV=production
Volume=app-data.volume:/data

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

And a volume file:

[Volume]
VolumeName=app-data

Saved as app.container and app-data.volume in the appropriate systemd container directory.

Mapping Docker Compose Concepts to Quadlet

Compose concept	Quadlet equivalent
`services`	`.container` files or `.pod` plus `.container`
`volumes`	`.volume` files or bind mounts
`networks`	`.network` files
`ports`	`PublishPort=`
`environment`	`Environment=` or `EnvironmentFile=`
`restart`	`[Service] Restart=`
`depends_on`	systemd `After=`, `Wants=`, `Requires=`
`healthcheck`	Podman healthcheck options
`profiles`	systemd enablement and separate units
`docker compose logs`	`journalctl -u service` and `podman logs`
`docker compose up -d`	`systemctl start service`
`docker compose down`	`systemctl stop service`
project directory	systemd container unit directory

The migration is conceptually simple but not mechanical. Compose describes a stack. Quadlet describes services.

Rootless Containers

Rootless containers are one of the strongest reasons to look at Podman and Quadlet.

With Docker, many users add themselves to the docker group. That is convenient, but access to the Docker daemon is effectively powerful host access. On a personal workstation, that may be acceptable. On shared servers, it deserves more caution.

Podman was designed with rootless usage as a first-class workflow. A rootless Quadlet lives under the user's config directory:

~/.config/containers/systemd/whoami.container

Then manage it with user systemd:

systemctl --user daemon-reload
systemctl --user enable --now whoami.service
systemctl --user status whoami.service
journalctl --user -u whoami.service -f

To allow the user service to keep running after logout:

sudo loginctl enable-linger "$USER"

This is a clean model for user-owned services.

Rootless Comparison

Area	Docker Compose	Podman Quadlet
Default common setup	Rootful Docker daemon	Rootless-friendly Podman
User service model	Possible, but less native	Native with `systemctl --user`
Daemon access risk	Docker socket is powerful	No central root daemon by default
Low port binding	Simple as rootful Docker	Needs extra setup when rootless
Host integration	Very common	More Linux-native
Shared server fit	Needs care	Strong fit

Rootless is not magic. It has tradeoffs around networking, privileged behavior, and low ports. But for long-running user-owned services, Quadlet is a very elegant model.

Startup and Boot Behavior

Docker Compose

Compose by itself does not create a boot service. You usually rely on Docker restart policies, a systemd wrapper around docker compose up -d, a deployment script, or a higher-level tool.

Example Compose restart policy:

services:
  app:
    image: example/app:stable
    restart: unless-stopped

A systemd wrapper gives you a host-level service. That is good, but it is still an extra wrapper. See Run Docker Compose as a Linux Service with systemd for the full pattern.

Podman Quadlet

Quadlet is already systemd-oriented. Enable on boot:

sudo systemctl enable myapp.service

For rootless:

systemctl --user enable myapp.service
sudo loginctl enable-linger "$USER"

Boot behavior is not an add-on. It is the model.

Restart Behavior

Compose commonly uses restart: unless-stopped in YAML. Quadlet commonly uses systemd restart behavior:

[Service]
Restart=always

Or:

[Service]
Restart=on-failure

This moves restart logic into the service manager. The preference: use Compose/Docker restart policies for Compose stacks, use systemd restart policies for Quadlet, and do not stack too many supervisors. Keep one clear owner of restart behavior.

Logging Comparison

Docker Compose Logs

docker compose logs -f
docker compose logs -f app

This is excellent for developers.

Quadlet Logs

Quadlet services use systemd logs:

journalctl -u app.service -f

For rootless units:

journalctl --user -u app.service -f

You can still use Podman logs:

podman logs -f container-name

For server operations, journalctl integration is a major advantage. Your containers fit into the same log workflow as other Linux services.

Updates

Updating Docker Compose

A common update flow:

cd /opt/myapp
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f

Easy to wrap in a script:

#!/usr/bin/env bash
set -euo pipefail

cd /opt/myapp

docker compose config --quiet
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f
docker compose ps

Updating Podman Quadlet

A simplified manual flow:

sudo podman pull ghcr.io/example/app:1.0.1
sudo systemctl restart app.service

Or for rootless:

podman pull ghcr.io/example/app:1.0.1
systemctl --user restart app.service

Podman can support auto-update workflows when containers are configured with the right labels and image policy. Quadlet's advantage is not that updates are always simpler. The advantage is that updates are service-manager-native.

Auto-Update Philosophy

Auto-updates are convenient. They are also a risk. For low-risk homelab services, automatic container updates can be fine. For databases, stateful apps, or business services, the preferred flow is:

Back up.
Pull.
Recreate or restart.
Check health.
Prune later.

Compose makes this explicit. Quadlet and Podman can make it systemd-native. Neither tool removes the need for a rollback plan.

Volumes and Persistent Data

Compose Volumes

Compose supports named volumes:

services:
  db:
    image: postgres:16
    volumes:
      - db-data:/var/lib/postgresql/data

volumes:
  db-data:

And bind mounts:

services:
  app:
    image: example/app
    volumes:
      - ./config:/config:ro
      - ./data:/data

Quadlet Volumes

Quadlet can use bind mounts:

[Container]
Volume=/opt/app/config:/config:ro
Volume=/opt/app/data:/data

Or a .volume file:

[Volume]
VolumeName=app-data

Then reference it:

[Container]
Volume=app-data.volume:/data

Compose is more compact for stack-level storage. Quadlet is more aligned with independently managed service units.

Secrets and Environment Files

Compose

Compose often uses env_file or environment in YAML. For a small private service, .env is common. For serious systems, treat .env as sensitive and keep it out of Git.

Quadlet

Quadlet can use:

[Container]
Environment=APP_ENV=production
EnvironmentFile=/opt/app/app.env

Restrict permissions:

chmod 600 /opt/app/app.env

Neither Compose nor Quadlet is a complete secret-management system by itself. Do not confuse "not in the command line" with "secure".

Networking

Compose Networking

Compose creates a default project network and gives services DNS names based on service names:

services:
  app:
    image: example/app
    depends_on:
      - db

  db:
    image: postgres:16

The app can usually reach the database at db. Multi-container app networking feels natural. This is one of Compose's strongest features.

Quadlet Networking

Quadlet can define networks separately with .network files or use Podman network options:

[Network]
NetworkName=appnet

Container file:

[Container]
Image=example/app:stable
Network=appnet.network

This is more explicit and systemd-like. For one or two containers, it is fine. For a large app stack, Compose is often easier to read.

Pods

Podman has a native pod concept. That matters if you like the Kubernetes mental model where multiple containers share a network namespace and lifecycle boundary. Quadlet supports .pod files:

[Pod]
PodName=myapp
PublishPort=8080:8080

A container can join that pod:

[Container]
Image=ghcr.io/example/app:stable
Pod=myapp.pod

Compose does not have the same pod model. It has services on networks. For most simple web apps, Compose networks are enough. For Podman users who like pod-style grouping, Quadlet is a better match.

Build Workflows

Compose is usually better when you build images as part of the local application workflow:

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"

Then:

docker compose up --build

This is extremely convenient for development. Quadlet is usually better when you run already-built images as services. Build images with Podman separately:

podman build -t localhost/myapp:latest .

Then reference the image:

[Container]
Image=localhost/myapp:latest

If your workflow is "edit code, rebuild, restart stack", Compose wins. If your workflow is "deploy a known image as a Linux service", Quadlet wins.

Portability

Compose files are widely shared. Many open-source projects provide a compose.yaml or docker-compose.yml. If a project says "run this with Docker Compose", you can usually start quickly with docker compose up -d. This is a major practical advantage.

Quadlet is portable across systems that have Podman, systemd, and compatible Quadlet support. That is a narrower target, but a very good one for modern Linux servers. Quadlet is not the best format for sharing an application with every possible developer. It is a good format for describing how a specific Linux host should run a containerized service.

Developer Experience

Docker Compose usually wins developer experience. More examples, more tutorials, more project templates, easier local builds, easy one-file stack, familiar docker compose up, and strong fit for dev dependencies. A developer can read this quickly:

services:
  db:
    image: postgres:16
  redis:
    image: redis:7
  app:
    build: .

Quadlet can do similar things, but it is more operations-shaped. For local development, I would rarely start with Quadlet unless the application itself is specifically about Podman or systemd.

Operations Experience

Quadlet often wins operations experience on a Linux host. Reasons:

Native systemctl
Native journalctl
Rootless user services
systemd dependencies
systemd timers
systemd restart behavior
No central Docker daemon
Better fit with host service management

A server admin can reason about:

systemctl status app.service
journalctl -u app.service -f
systemctl restart app.service

That is the normal Linux service workflow.

Security Model

Docker Compose Security Notes

Docker Compose usually talks to the Docker daemon. On a normal Linux Docker install, access to the Docker socket is powerful. A user who can control Docker can often mount host paths, run privileged containers, or otherwise gain broad host control. For installation options including rootless Docker on Ubuntu, see Install Docker on Ubuntu.

Practical advice:

Do not casually expose /var/run/docker.sock.
Treat the docker group as privileged.
Avoid privileged containers.
Avoid host mounts unless needed.
Keep secrets out of Git.
Use explicit image tags for important services.
Review published ports.

Quadlet Security Notes

Podman Quadlet pairs well with rootless containers and user systemd services. This can reduce risk, especially on shared hosts or personal servers where services should not require a root daemon.

Practical advice:

Prefer rootless services when they fit.
Use user units for user-owned services.
Use system units only when host-level privileges are needed.
Avoid unnecessary privileged containers.
Keep environment files locked down.
Think carefully about bind mounts.

Rootless does not mean risk-free. It means the default blast radius can be smaller.

Performance

For most web services, internal tools, and self-hosted apps, performance is not the deciding factor. The main differences are operational, not raw speed. Choose based on lifecycle model, security model, host integration, team familiarity, update process, networking needs, and debugging workflow.

If you are choosing between Compose and Quadlet because of performance alone, you are probably optimizing the wrong layer.

Failure Modes

Docker Compose Failure Modes

Common problems:

Docker daemon not running
Compose plugin missing
Wrong project directory
.env not loaded as expected
Old docker-compose binary used by accident
Containers not recreated after config changes
Orphan containers left after service rename
Volumes deleted with down -v
Docker logs filling the disk
Docker socket permission errors

Best fixes:

docker compose config
docker compose ps
docker compose logs -f
docker compose up -d --remove-orphans

Podman Quadlet Failure Modes

Common problems:

Unit file in the wrong directory
Forgot systemctl daemon-reload
Using system units when user units were intended
Forgot loginctl enable-linger for rootless services
Image pull takes longer than systemd startup timeout
cgroup v2 not available
SELinux labels or volume permissions
Service name differs from file expectations
Network or volume unit not enabled or referenced correctly

Best fixes:

systemctl status app.service
journalctl -u app.service -f
systemctl daemon-reload
podman ps -a
podman logs container-name

For rootless:

systemctl --user status app.service
journalctl --user -u app.service -f

Migration Example: Compose to Quadlet

Start with this Compose service:

services:
  whoami:
    image: traefik/whoami:v1.10
    restart: unless-stopped
    ports:
      - "8080:80"
    environment:
      WHOAMI_NAME: compose-demo

Run with Compose:

docker compose up -d

A rough Quadlet equivalent:

[Unit]
Description=Whoami demo container
After=network-online.target
Wants=network-online.target

[Container]
Image=docker.io/traefik/whoami:v1.10
PublishPort=8080:80
Environment=WHOAMI_NAME=quadlet-demo

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

Save as /etc/containers/systemd/whoami.container, then:

sudo systemctl daemon-reload
sudo systemctl enable --now whoami.service
sudo systemctl status whoami.service

The example is easy because it is one container. A larger Compose stack with databases, networks, volumes, and build steps needs more careful translation.

Migration Checklist

Before moving from Compose to Quadlet, ask:

[ ] Is this stack really a set of long-running host services?
[ ] Are the images already built and published?
[ ] Do I need rootless services?
[ ] Do I want systemd dependencies and timers?
[ ] Are volumes and bind mounts clearly understood?
[ ] Are ports documented?
[ ] Are secrets handled outside Git?
[ ] Is there a backup and restore process?
[ ] Can I monitor logs through journalctl?
[ ] Do I have a rollback path?

If most answers are yes, Quadlet may be a good fit. If the stack is mostly for local development, Compose is probably still better.

When to Stay with Docker Compose

Stay with Compose when:

The project already ships a good Compose file.
You need the easiest onboarding path.
Developers run the same stack locally.
You build images during development.
You want one YAML file for services, volumes, and networks.
You want maximum tutorial and community compatibility.
Your current systemd wrapper works fine.

There is no prize for migrating a working Compose stack to Quadlet just because Quadlet is cleaner in theory. If Compose is boring and reliable for your use case, keep it.

When to Move to Podman Quadlet

Move to Quadlet when:

The stack is really a host service.
You want rootless service management.
You prefer Podman over Docker.
You want systemd to own lifecycle.
You want journalctl logs.
You want service dependencies.
You want user services that survive logout.
You want less Docker daemon exposure.
You are building a self-hosting host around systemd.

Quadlet is not "Compose but better." It is a different design center.

Recommended Patterns

Pattern 1: Local Development

Use Docker Compose. Fast, familiar, portable, easy to rebuild, easy for teams.

Pattern 2: Single-Host Self-Hosting

Use either. Choose Compose if the project already provides a Compose file. Choose Quadlet if you want systemd-native service management. Compose gives a better app bundle; Quadlet gives a better Linux service.

Pattern 3: User-Owned Rootless Service

Use Podman Quadlet. Rootless workflow, user-level service management, no central Docker daemon.

~/.config/containers/systemd/app.container
systemctl --user enable --now app.service
loginctl enable-linger

Pattern 4: Production-Like Single Server

Use Docker Compose with a disciplined systemd wrapper, or use Quadlet if your team is comfortable with Podman. Do not choose based on fashion. Choose based on who will operate it at 2 AM.

Pattern 5: Multi-Node Platform

Use neither as the final orchestration layer. Consider Kubernetes, Nomad, Swarm, or a managed platform. Compose and Quadlet are excellent single-host tools. They are not cluster schedulers.

Practical Decision Tree

Is this mainly for local development?
  yes:
    use Docker Compose
  no:
    continue

Does the project already provide a maintained compose.yaml?
  yes:
    use Docker Compose unless you have a strong reason to migrate
  no:
    continue

Do you want rootless long-running services managed by systemd?
  yes:
    use Podman Quadlet
  no:
    continue

Do you want the easiest multi-container app definition?
  yes:
    use Docker Compose
  no:
    continue

Do you want containers to behave like normal Linux services?
  yes:
    use Podman Quadlet
  no:
    use Docker Compose

Side-by-Side Commands

Task	Docker Compose	Podman Quadlet
Start	`docker compose up -d`	`systemctl start app.service`
Stop	`docker compose down`	`systemctl stop app.service`
Restart	`docker compose restart`	`systemctl restart app.service`
Apply changes	`docker compose up -d`	`systemctl daemon-reload && systemctl restart app.service`
Logs	`docker compose logs -f`	`journalctl -u app.service -f`
Status	`docker compose ps`	`systemctl status app.service`
Enable on boot	systemd wrapper or restart policy	`systemctl enable app.service`
Pull update	`docker compose pull`	`podman pull image`
Rootless service	possible	natural with `systemctl --user`

Common Misunderstandings

"Quadlet Replaces Docker Compose"

Not exactly. Quadlet replaces some Compose use cases, especially long-running Linux services. It does not replace Compose as the easiest application-stack format for developers.

"Docker Compose Is Not Production Ready"

Too broad. Compose can be perfectly reasonable for small production systems if you understand backups, updates, logging, restart behavior, and host security. The problem is not Compose. The problem is pretending a single-host Compose deployment has the same properties as a cluster orchestrator.

"Podman Is Just Docker Without the Daemon"

Too simple. Podman has Docker-compatible commands, but its design center is different: daemonless operation, rootless workflows, pods, and Linux integration.

"Rootless Means Secure"

No. Rootless reduces some risks. It does not make bad images, exposed secrets, unsafe bind mounts, or vulnerable apps safe.

"systemd Is Too Heavy for Containers"

systemd is already the service manager on most mainstream Linux servers. Using it to manage long-running containers is not strange. It is often the boring and correct thing to do.

Final Recommendation

Use Docker Compose when the application stack is the main thing. Use Podman Quadlet when the Linux service is the main thing.

That distinction is more useful than arguing which tool is better. For developer workflows, Compose is hard to beat. It is popular, readable, portable, and supported by countless projects. For long-running Linux services, Quadlet is quietly excellent. It makes containers feel like native systemd services, works naturally with rootless Podman, and fits the operational model of a serious Linux host.

The preferred split:

Local development: Docker Compose
Portable app examples: Docker Compose
Small self-hosted stacks: Docker Compose or Quadlet
Rootless user services: Podman Quadlet
Long-running host services: Podman Quadlet
Multi-node orchestration: neither; use a real orchestrator

Do not migrate just to be modern. Migrate when the lifecycle model is better. Compose is a great stack tool. Quadlet is a great service tool. The smart choice is to use each where its mental model matches the job.

References

Podman Quadlet Documentation — official Podman Quadlet reference
Docker Compose Documentation — official Docker Compose reference
Run Docker Compose as a Linux Service with systemd — detailed Compose-as-systemd walkthrough on this site
Docker Compose Cheatsheet — Compose commands and patterns reference
Docker Cheatsheet — Docker commands reference
Install Docker on Ubuntu — Docker installation guide with rootless alternatives
Run any Executable as a Service in Linux — general systemd service patterns
Developer Tools: The Complete Guide to Modern Development Workflows — cluster home

Hermes Agent: Headless Server + Remote Desktop Setup

Rost — Tue, 14 Jul 2026 13:07:27 +0000

Running Hermes Agent on a headless server while connecting from a desktop client on another machine requires two server processes and a single client connection.

The architecture separates the Hermes backend into two server-side processes and one client-side surface. The hermes serve backend handles the API and dashboard connections, while the hermes gateway run process manages messaging channels independently. The desktop client connects to the serve backend, not the gateway.

flowchart LR
    subgraph Server["HEADLESS SERVER"]
        serve["hermes serve<br/>--host 0.0.0.0<br/>:9119"]
        gateway["hermes gateway run<br/>Telegram, Discord, Slack"]
    end

    subgraph Client["DESKTOP PC"]
        desktop["hermes desktop<br/>(WebSocket connection)"]
    end

    desktop <--->|WebSocket| serve
    gateway -.->|Shares ~/.hermes/| serve

Two processes on the server, one app on the client. Both server processes share the same ~/.hermes/ config, skills, memory, and sessions. Cron jobs execute on the server where the gateway runs.

For installation, provider setup, and initial configuration, start with the Hermes AI Assistant — Install, Setup, Workflow, and Troubleshooting.

Set up the headless server

1. Configure authentication

Basic auth provides sufficient protection for a trusted LAN. Add credentials to the environment file:

cat >> ~/.hermes/.env << 'EOF'
HERMES_DASHBOARD_BASIC_AUTH_USERNAME=admin
HERMES_DASHBOARD_BASIC_AUTH_PASSWORD=choose-a-strong-password
HERMES_DASHBOARD_BASIC_AUTH_SECRET=$(openssl rand -base64 32)
EOF
chmod 600 ~/.hermes/.env

The .env file lives alongside config.yaml under ~/.hermes/. Hermes resolves configuration with CLI overrides first, then config.yaml, then .env, then built-in defaults. Secrets belong in .env.

2. Start the backend

Run the serve backend on all interfaces:

hermes serve --host 0.0.0.0 --port 9119

The backend listens on port 9119 and accepts WebSocket connections from the desktop client. Verify it is running with a quick status check:

curl -s http://localhost:9119/api/status | jq '.auth_required, .auth_providers'
# Expected: true  "basic"

3. Run as a systemd service

For a persistent server that survives reboots, install a user-level systemd service. Create the unit file at /etc/systemd/user/hermes-serve.service:

[Unit]
Description=Hermes Agent Serve Backend

[Service]
EnvironmentFile=%h/.hermes/.env
ExecStart=/home/rg/.hermes/hermes-agent/venv/bin/python -m hermes_cli.main serve --host 0.0.0.0 --port 9119
Restart=on-failure

[Install]
WantedBy=default.target

Reload the daemon, enable the service, and start it:

systemctl --user daemon-reload
systemctl --user enable hermes-serve
systemctl --user start hermes-serve

Check the service status:

systemctl --user status hermes-serve

4. Start the gateway

The gateway is a separate process that handles messaging channels — Telegram, Discord, Slack, and others. Start it independently:

hermes gateway run

The gateway and the serve backend are two separate processes. The gateway manages sessions, runs cron jobs, and routes messages. The serve backend provides the API surface for desktop and web dashboard connections. They share the same home directory but run independently.

For the full list of gateway commands and subcommands, see the Hermes Agent CLI cheat sheet.
If your primary interface is mobile messaging, pair this setup with Hermes Voice Control from Your Phone for voice-first workflows on top of the same gateway process.

5. Verify the backend

Confirm the backend is responding and authentication is active:

curl -s http://localhost:9119/api/status | jq '.auth_required, .auth_providers'

Expected output:

true
"basic"

If authentication is not enabled, the response will show false for auth_required. Check that the .env file contains the correct variables and that the service has restarted after configuration changes.

Connect from the desktop client

Option A: Hermes Desktop App

Install Hermes Desktop from the official site. Launch the app, navigate to Settings → Gateway → Remote gateway, and enter the server address:

http://<server-ip>:9119

Alternatively, set the remote URL via environment variable:

HERMES_DESKTOP_REMOTE_URL=http://<server-ip>:9119 hermes desktop

Option B: Web dashboard

Open http://<server-ip>:9119 in a browser and sign in with the basic auth credentials. The web dashboard provides a browser-based interface to the Hermes backend without requiring a desktop installation.

Option C: CLI

From the desktop PC's terminal, configure the remote URL via the desktop app settings or the HERMES_DESKTOP_REMOTE_URL environment variable. The CLI surface connects through the same WebSocket channel as the desktop app.

Network and security considerations

Scenario	Recommendation
Trusted LAN	`--host 0.0.0.0` + basic auth
Exposed to internet	Use Tailscale (`--host <tailscale-ip>`) or OAuth provider
Firewall	Open port 9119 TCP on the server

For a trusted local network, basic auth on 0.0.0.0 is adequate. If the server is exposed to the internet, use Tailscale to bind to the Tailscale IP instead of 0.0.0.0, or configure an OAuth provider for stronger authentication. Always open port 9119 TCP in the server's firewall when the backend needs to accept external connections.

Important process distinctions

Understanding the separation between the two server processes prevents common configuration mistakes:

hermes serve — The backend that the desktop app and web dashboard connect to. Handles the API surface and WebSocket connections.
hermes gateway run — The process that handles Telegram, Discord, Slack, and other messaging channels. Manages sessions, runs cron jobs, and routes messages.
These are two separate processes on the server. They share ~/.hermes/ config, skills, memory, and sessions, but run independently.
Cron jobs execute on the server where the gateway runs.
Profiles, skills, and memory are configured on the server side. The client connects to the already-configured backend.

For profile-first configuration and skills tuned to different production roles, see Hermes AI Assistant Skills for Real Production Setups.

Troubleshooting

Desktop cannot connect to the server

Verify the backend is running: systemctl --user status hermes-serve
Check the port is open: ss -tlnp | grep 9119
Test from the server: curl -s http://localhost:9119/api/status
Test from the client machine: curl -s http://<server-ip>:9119/api/status
If the client test fails, check firewall rules: sudo ufw status or sudo iptables -L

Authentication fails

Confirm .env file has correct permissions: ls -la ~/.hermes/.env (should be 600)
Verify the secret was generated: grep HERMES_DASHBOARD_BASIC_AUTH_SECRET ~/.hermes/.env
Restart the service after config changes: systemctl --user restart hermes-serve

Gateway does not respond to messages

The gateway is a separate process from the backend. If the desktop connects but messaging platforms do not work:

Check gateway status: hermes gateway status
Start the gateway if stopped: hermes gateway start
Review logs: hermes logs gateway -f

References

This article is part of the AI Systems cluster, which covers self-hosted assistants, retrieval architecture, local LLM infrastructure, and observability.

GPUs for AI in 2026: NVIDIA, AMD, Intel Compared

Rost — Tue, 14 Jul 2026 00:14:41 +0000

The AI hardware landscape has shifted significantly in 2026, with NVIDIA, AMD, and Intel all competing for developers who need GPUs capable of running local large language models and AI inference workloads.

Choosing the right GPU for AI workloads requires looking beyond marketing numbers and focusing on the specifications that actually affect real-world performance. Memory capacity, memory bandwidth, and software ecosystem maturity consistently matter more than theoretical compute peaks when running transformer models locally.

This comparison covers the most relevant workstation and prosumer GPUs available in mid-2026, including NVIDIA's Blackwell architecture (RTX 50-series), AMD's Radeon AI Pro R9700, and Intel's Arc Pro B70. The goal is to provide a practical reference for developers deciding which hardware best fits their model sizes, software stack, and budget constraints.

Which GPU specifications matter for AI workloads

Marketing materials from GPU vendors emphasise AI TOPS and tensor performance, but these metrics rarely tell the complete story for local inference. The specifications below are ranked by their actual impact on running large language models.

VRAM capacity

VRAM is typically the first limiting factor when running LLMs locally. A model cannot execute entirely on the GPU if it does not fit into available memory. Once model weights spill into system RAM, inference performance drops dramatically.

Approximate VRAM requirements for common model sizes:

Model Size	Recommended VRAM
7B	8-12 GB
14B	16 GB
32B	24-32 GB
70B	48-64 GB
120B+	Multiple GPUs

For most homelab users, moving from 16 GB to 32 GB of VRAM provides a substantially larger practical benefit than increasing raw compute performance. A 32 GB GPU capable of running an entire model will often outperform a theoretically faster 16 GB GPU forced to offload tensors into system memory.

Memory bandwidth

Memory bandwidth determines how quickly model weights can be streamed into compute units. Large transformer models continuously move massive amounts of data between VRAM and processing cores during inference.

As models grow, bandwidth often becomes the dominant performance bottleneck. A card with higher bandwidth can outperform another GPU with significantly higher theoretical compute performance, particularly during prompt processing phases where the model reads through the entire context window.

FP32 compute

FP32 throughput remains useful for scientific computing, simulation, rendering, and some AI preprocessing workloads. Modern inference engines rarely execute entirely in FP32 precision, relying instead on quantised formats like Q4_K_M or Q8_0. FP32 should be considered a secondary metric for AI inference.

AI TOPS and tensor performance

Every GPU vendor promotes AI TOPS as a headline number. These values are not directly comparable across vendors. NVIDIA, AMD, and Intel measure AI throughput differently, use different tensor hardware, and apply different assumptions regarding sparsity and numerical precision.

AI TOPS should be viewed as an indication of peak theoretical capability rather than an expected LLM inference speed. Real-world token generation rates depend on model architecture, quantisation level, context length, and software optimisation — factors that TOPS numbers do not capture.

Software ecosystem maturity

Software support often determines whether hardware reaches its full potential. The current ecosystem landscape is approximately:

Vendor	Primary AI Stack	Maturity
NVIDIA	CUDA, TensorRT	Industry standard
AMD	ROCm, HIP, Vulkan	Solid for PyTorch, llama.cpp, Ollama
Intel	oneAPI, SYCL, OpenVINO	Improving rapidly, trailing peers

CUDA remains the industry standard with the broadest library support. ROCm has matured significantly over the past two years and now provides a functional experience for PyTorch, llama.cpp, and Ollama on Linux. Intel's oneAPI ecosystem continues to improve but still trails both NVIDIA and AMD in overall software maturity and community adoption.

For a deeper look at NVIDIA-specific GPU analysis, see Comparing NVIDIA GPU Suitability for AI.

Complete GPU comparison table

The table below compares the most relevant workstation and enthusiast GPUs for AI workloads in 2026.

GPU	VRAM	Bandwidth	FP32 (TFLOPS)	AI TOPS (INT8)	TBP	MSRP
NVIDIA RTX 5090	32 GB	1792 GB/s	104.6	3352	575 W	$1799
NVIDIA RTX 5080	16 GB	960 GB/s	56.3	1801	360 W	$999
NVIDIA RTX 5070 Ti	16 GB	896 GB/s	43.9	1406	300 W	$649
NVIDIA RTX 5070	12 GB	672 GB/s	30.9	494	250 W	$549
NVIDIA RTX 5060 Ti 16GB	16 GB	448 GB/s	23.7	614	180 W	$399
NVIDIA RTX PRO 6000	96 GB	1792 GB/s	125.0	4000	600 W	$4999
NVIDIA RTX PRO 5000	48 GB	1344 GB/s	73.7	2064	300 W	$2499
NVIDIA RTX PRO 4500	32 GB	896 GB/s	54.9	1577	200 W	$2500
NVIDIA RTX PRO 4000	24 GB	672 GB/s	46.9	1178	145 W	$1500
NVIDIA RTX PRO 4000 SFF	24 GB	432 GB/s	46.9	770	125 W	$1500
NVIDIA RTX PRO 2000	16 GB	288 GB/s	18.4	592	70 W	$700
AMD Radeon AI Pro R9700	32 GB	640 GB/s	47.8	766	300 W	$1299
Intel Arc Pro B70	32 GB	608 GB/s	22.94	367	230 W	$949

Key observations by segment

Consumer GPUs

The RTX 5090 remains the fastest single-GPU solution for local AI development, combining exceptional memory bandwidth with the mature CUDA ecosystem. For users running large quantised models, it currently represents the highest-performance consumer option.

The RTX 5080 and RTX 5070 Ti both offer 16 GB of VRAM, which is sufficient for most 7B-14B models but limits you when working with larger checkpoints. The RTX 5060 Ti 16GB variant is an interesting budget option — 16 GB of VRAM at $399 is compelling for entry-level AI workloads, though the narrower memory bus will impact throughput.

Workstation GPUs

Within the workstation segment, AMD's Radeon AI Pro R9700 occupies an attractive middle ground. It delivers 32 GB of VRAM, competitive memory bandwidth, and a significantly lower purchase price than NVIDIA's professional offerings. For developers already comfortable with ROCm on Linux, it provides one of the strongest value propositions in 2026.

Intel's Arc Pro B70 is particularly interesting because of its pricing. Although it offers lower compute performance than both NVIDIA and AMD, it provides the same 32 GB memory capacity while consuming less power. For users building cost-effective multi-GPU inference servers, the B70 deserves consideration — especially if the oneAPI ecosystem meets your software requirements.

Professional GPUs

NVIDIA's RTX PRO series dominates the professional segment, with the RTX PRO 6000 offering 96 GB of VRAM — unmatched by any competitor. For teams running very large models or multiple concurrent inference workloads, the RTX PRO 6000 and RTX PRO 5000 remain the safest choices, though at a premium price.

For a real-world performance comparison across different hardware platforms, see NVIDIA DGX Spark vs Mac Studio vs RTX-4080.

Practical hardware considerations

Physical dimensions and form factor

GPU size varies significantly across product lines and affects compatibility with your case and cooling solution.

GPU	Approx. Length	Slots	Cooler Type
RTX 5090	333 mm	2.7×	Triple-fan, blower or open
RTX 5080	303 mm	2.5×	Dual/triple-fan
RTX 5070 Ti	280 mm	2.4×	Dual-fan
RTX 5070	245 mm	2.1×	Dual-fan
RTX 5060 Ti	200 mm	1.8×	Dual-fan
AMD R9700	300 mm	2.5×	Dual-fan
Intel Arc Pro B70	267 mm	2.1×	Single/dual-fan
RTX PRO 6000	438 mm	3.5×	Blower, full-height
RTX PRO 5000	438 mm	3.5×	Blower, full-height
RTX PRO 4000	267 mm	2.1×	Blower, low-profile option
RTX PRO 4000 SFF	178 mm	1.5×	Blower, half-height

The RTX PRO 6000 and 5000 are significantly longer than consumer cards and require full-height tower cases. The RTX PRO 4000 SFF is one of the few GPUs under 180 mm, making it suitable for compact workstation builds and rack-mounted servers.

Consumer GPUs (RTX 50-series) use open-air coolers that exhaust heat into the case — adequate case airflow is essential. Workstation GPUs use blower-style coolers that exhaust heat directly out the rear, which is better for multi-GPU configurations and enclosed server environments.

Power delivery and PSU requirements

TBP (Total Board Power) is the GPU's maximum power draw, but actual system requirements depend on transient spikes and CPU overhead.

GPU	TBP	Recommended PSU	Power Connectors
RTX 5090	575 W	1000 W+	12V-2x6 (20-pin)
RTX 5080	360 W	750 W	12V-2x6
RTX 5070 Ti	300 W	650 W	8-pin + 8-pin
RTX 5070	250 W	600 W	8-pin
RTX 5060 Ti	180 W	550 W	8-pin
AMD R9700	300 W	650 W	8-pin + 8-pin
Intel Arc Pro B70	230 W	550 W	8-pin
RTX PRO 6000	600 W	1000 W+	12V-2x6
RTX PRO 5000	300 W	650 W	8-pin + 8-pin
RTX PRO 4000	145 W	500 W	8-pin
RTX PRO 4000 SFF	125 W	450 W	8-pin
RTX PRO 2000	70 W	400 W	PCIe slot only

The RTX 5090 and RTX PRO 6000 both exceed 575W TBP and require the newer 12V-2x6 connector (20-pin). Ensure your PSU supports this connector natively — adapter cables from multiple 8-pin connectors are not recommended for cards above 450W due to transient power spikes that can exceed rated capacity momentarily.

Thermal characteristics and sustained workloads

AI inference workloads keep the GPU under sustained load, unlike gaming which has variable utilisation. This affects thermal behaviour significantly.

RTX 5090 at 575W: Expect GPU temperatures of 72-78°C under sustained inference. The higher TBP means more heat dissipation is required — a case with positive static pressure and quality filters is recommended.
RTX 5080 at 360W: Runs cooler, typically 65-72°C. More manageable for standard mid-tower cases.
Workstation GPUs (blower): RTX PRO series exhaust heat directly out the case, keeping case temperatures lower. GPU temperatures may read higher (75-82°C) but this is by design — the blower cooler trades GPU temperature for lower case temperature.
Low-power options: RTX PRO 2000 at 70W and RTX PRO 4000 SFF at 125W are suitable for passive or low-fan-speed cooling, making them ideal for always-on inference servers where noise matters.

For multi-GPU setups, blower-style coolers (workstation GPUs) are strongly preferred over open-air consumer coolers, as the second GPU would otherwise pull hot air from the first.

PCIe lanes and bandwidth

GPU performance can be limited by PCIe lane count. A GPU plugged into a x8 or x4 slot will experience reduced memory bandwidth compared to a full x16 connection. For multi-GPU setups, understand how PCIe lanes are distributed across your motherboard. See LLM Performance and PCIe Lanes for detailed analysis.

Multi-GPU setups

When a single GPU cannot fit your model, multi-GPU configurations become necessary. NVIDIA NVLink (where supported) and PCIe-based model parallelism are the primary approaches. The AI Infrastructure on Consumer Hardware guide covers multi-GPU deployment strategies in depth.

Note that AMD and Intel GPUs have limited multi-GPU inference support in most frameworks. If you plan to scale with multiple GPUs, NVIDIA is currently the only practical option.

Conclusion

There is no universally best GPU for AI workloads. The right choice depends on your software stack, budget, and the size of the models you intend to run.

NVIDIA's Blackwell family remains the benchmark for inference performance, thanks to outstanding memory bandwidth and the maturity of CUDA and TensorRT. AMD's Radeon AI Pro R9700 has established itself as a compelling workstation option, offering an excellent balance between price, memory capacity, and compute performance. Intel's Arc Pro B70 proves that affordable 32 GB workstation GPUs are now a reality, though its software ecosystem continues to mature.

The most important lesson from 2026 is that AI hardware should no longer be evaluated using gaming benchmarks. For modern LLM inference, VRAM capacity, memory bandwidth, and software support consistently have a greater impact on real-world performance than theoretical AI TOPS alone.

References

Comparing NVIDIA GPU Suitability for AI — NVIDIA-specific GPU analysis with detailed CUDA core and tensor core comparisons
AI Infrastructure on Consumer Hardware — Full-stack guide to deploying self-hosted AI with consumer GPUs
NVIDIA DGX Spark vs Mac Studio vs RTX-4080 — Real-world Ollama performance benchmarks across hardware platforms
LLM Performance and PCIe Lanes — How PCIe configuration affects LLM inference performance
Ollama Cheatsheet — Command reference and tips for Ollama model serving
Quadro RTX 5880 Ada Review — Review of the 48GB workstation GPU alternative
Best LLM on 16 GB VRAM GPU — llama.cpp benchmarks for models on 16 GB VRAM

Spec-Driven Development Workflow From Requirements to Code

Rost — Mon, 13 Jul 2026 04:21:33 +0000

Spec-Driven Development works when the specification is a workflow, not a document you file away after kickoff. The point is not to produce a large product requirements document.

The point is to move through a sequence of reviewable artifacts that each reduce ambiguity before anyone -- human or AI agent -- changes production code.

If you do not know what SDD is conceptually, start with What Is Spec-Driven Development? for definitions, comparisons with TDD and BDD, and the case for treating the spec as source of truth. This article in the App Architecture documentation cluster is the operational guide. It walks through the five phases, shows what each artifact should contain, explains where AI agents fit, and gives reusable templates you can copy into your repository today.

SDD Is a Workflow, Not a Document

The most common failure mode in spec-driven development is treating the spec as paperwork. A team writes a long requirements document, stores it in a wiki, and then codes from memory and chat threads. The spec exists, but it does not drive anything. That is documentation theater, and it is worse than no spec because it creates false confidence.

A working SDD workflow produces a chain of artifacts, each reviewed before the next phase begins. Requirements reduce product ambiguity. Design reduces technical ambiguity. Tasks reduce execution ambiguity. Implementation produces code against a known target. Validation proves the chain held. When any phase reveals a mistake, you fix the artifact and re-run from that point -- not after three thousand lines of drift have landed in main.

flowchart LR
  A[Specify] --> B[Plan]
  B --> C[Tasks]
  C --> D[Implement]
  D --> E[Validate]
  E -->|drift found| A
  E -->|ship| F[Done]

The workflow is tool-neutral. You can run it with markdown files in Git, with GitHub Spec Kit, with Cursor plans, or with a plain text editor and a disciplined reviewer. What matters is the sequence and the review gates, not the brand on the tooling.

Phase 1 -- Specify the Requirements

The specify phase answers what problem you are solving and what done looks like. It deliberately avoids how to build it. The moment your requirements spec says "use Redis sorted sets," you have stopped specifying and started designing in the wrong document. Keep implementation out of requirements. Put it in the plan.

Problem statement and users

Start with one paragraph that states the problem in plain language. Name the users affected and the situation that makes the problem painful. A good problem statement lets a reviewer who was not in the planning meeting decide whether a proposed solution actually addresses the pain.

Example for an API rate-limiting feature:

API consumers on the free tier can send unlimited requests, which causes cost spikes and noisy-neighbor impact on paid tenants. Platform operators need a enforceable per-key limit without manual intervention.

Goals, non-goals, and acceptance criteria

Goals describe outcomes you will deliver. Non-goals describe tempting adjacent work you will explicitly not do. Together they bound the agent's creativity, which is essential when AI tools otherwise "helpfully" expand scope.

Section	Good example	Weak example
Goal	Reject requests over the per-key limit with HTTP 429	Make the API faster
Non-goal	Per-tenant billing dashboards	Improve all API performance
Acceptance criterion	Unauthenticated requests receive 401 before rate check runs	The endpoint is secure

Acceptance criteria should be precise enough that each one maps to at least one test. "The endpoint is secure" is not an acceptance criterion. "Unauthenticated requests receive HTTP 401" is. If you cannot write a concrete criterion, the requirement is still too vague to implement.

Open questions

List every decision that is not yet settled. Unclear questions are not a sign of failure. They are the specify phase doing its job. Resolve them before you write the design plan, or you will pay for the ambiguity in implementation rework.

A minimal requirements template:

## Problem
[One paragraph: who hurts, why, and what triggers the pain.]

## Users
- [Primary user role]
- [Secondary user role]

## Goals
1. [Measurable outcome]
2. [Measurable outcome]

## Non-goals
- [Explicitly out of scope]
- [Explicitly out of scope]

## Acceptance criteria
- [ ] [Verifiable behavior]
- [ ] [Verifiable behavior]

## Open questions
- [ ] [Question that blocks planning]

Phase 2 -- Plan the Design

The plan phase translates intent into technical decisions. This is where Redis sorted sets belong, along with module boundaries, schema changes, API contracts, migration steps, security constraints, and the test strategy. The plan is derived from the requirements spec plus your project's existing constraints -- stack choices, decision records, and conventions stored in files like AGENTS.md or a project constitution.

Architecture and affected modules

Name the modules, services, or packages that will change and summarize the integration pattern. If the feature crosses a service boundary, document the contract on both sides. Agents hallucinate APIs when contracts are implicit. Making them explicit in the plan prevents invented endpoints and wrong response shapes.

Data model, API contracts, and migrations

Document schema changes, new tables or fields, index requirements, and backward-compatibility rules. For HTTP APIs, write method, path, request shape, response shape, and error codes. For events, write topic names, payload schemas, and delivery semantics. Include migration steps and rollback notes when the data model changes.

Security, observability, and test strategy

Security constraints belong in the plan, not as afterthoughts in code review. Note authentication requirements, authorization rules, input validation boundaries, and data that must not appear in logs. Observability should cover metrics, logs, or traces needed to confirm the feature works in production.

The test strategy connects back to acceptance criteria. Identify which criteria need unit tests, which need integration tests, and which need manual verification. If you use unit testing in Go or unit testing in Python, name the packages and test files you expect to add. A plan without a test strategy is a plan that will ship with gaps you discover in production.

flowchart TB
  subgraph plan [Design plan contents]
    R[Requirements spec]
    C[Project constitution / ADRs]
    R --> D[Architecture decisions]
    C --> D
    D --> M[Data model and migrations]
    D --> A[API contracts]
    D --> S[Security constraints]
    D --> T[Test strategy]
  end

Phase 3 -- Break Down Implementation Tasks

The task phase decomposes the plan into slices small enough to implement, review, and validate independently. This is what makes agent-assisted development reviewable. Instead of one enormous diff, you get a sequence of focused changes that each map back to a named requirement.

Task sizing and dependencies

A good task touches a bounded set of files, completes in one agent session, and ends with a verification step. Tasks should declare dependencies explicitly. Migration tasks run before code that reads the new schema. Shared library changes run before consumers. Authentication middleware changes run before endpoints that depend on the new behavior.

flowchart TD
  T1[Task 1 -- schema migration] --> T2[Task 2 -- repository layer]
  T2 --> T3[Task 3 -- HTTP handler]
  T2 --> T4[Task 4 -- metrics instrumentation]
  T3 --> T5[Task 5 -- integration tests]
  T4 --> T5

Files, validation, and review checkpoints

Each task should list the files likely to change, the acceptance criteria it satisfies, and how to validate completion. Validation might be a test command, a curl example, or a manual check described in copy-pasteable steps. Every task ends at a human review checkpoint. The reviewer confirms the diff matches the task description before the next task starts.

A minimal task entry:

### Task 3 -- Add rate-limit middleware

**Depends on:** Task 1 (schema), Task 2 (repository)
**Files:** middleware/ratelimit.go, middleware/ratelimit_test.go, server.go
**Satisfies:** AC-2 (429 over limit), AC-3 (limit headers in response)
**Validate:** `go test ./middleware/...` passes; curl over limit returns 429 with Retry-After
**Review checkpoint:** Confirm middleware runs after auth, before handler

Watch for generated task explosions. AI agents can produce fifty-task plans in seconds. Most of those tasks will be redundant or too granular to review efficiently. A useful task list for a medium feature often has five to fifteen items, not fifty.

Phase 4 -- Implement One Task at a Time

Implementation is deliberately narrow. Pick one task, give the agent only the context it needs for that task, and stop when validation passes. Context resets between tasks are a feature, not a bug. They prevent earlier assumptions from polluting later work and keep diffs reviewable.

Apply constraints from the spec stack

The implementing agent should read the requirements spec, the design plan, the current task description, and project-level constraints. Constraints are the highest-ROI section most teams skip. They tell the agent what not to do -- do not refactor unrelated modules, do not change public API signatures outside this feature, do not introduce new dependencies without updating the plan.

Update the plan when reality differs

Implementation will surface surprises. A library does not support the assumed behavior. A migration takes longer than expected. An edge case was missing from acceptance criteria. When that happens, update the spec before continuing. Fix the requirements or plan, get a quick review, then resume implementation against the corrected artifact. Code that diverges silently from the spec is how drift becomes permanent.

sequenceDiagram
  participant H as Human reviewer
  participant A as AI agent
  participant S as Spec artifacts
  H->>S: Approve task N
  A->>S: Read task + plan + constraints
  A->>A: Implement task N
  A->>A: Run task validation
  A->>H: Submit diff for review
  H->>H: Review diff against task
  alt drift or surprise
    H->>S: Update spec/plan
    H->>A: Re-run with corrected context
  else approved
    H->>S: Mark task N complete
    H->>A: Proceed to task N+1
  end

Phase 5 -- Validate Against the Spec

Validation is where SDD earns its keep. Without it, the spec is a planning exercise. With it, the spec is a contract you can check against the shipped code.

Automated checks

Run the full test suite, lint, and type checks on CI. Wire these into your pipeline using patterns from the GitHub Actions cheatsheet if you need a practical starting point. Automated checks catch regressions. They do not catch wrong features built correctly, which is why acceptance criteria review still matters.

Acceptance criteria and manual review

Walk through each acceptance criterion from the requirements spec. Mark each as satisfied, failed, or deferred with justification. Manual review catches UX issues, security gaps, and wrong behavior that tests missed because the tests were written to match a flawed spec.

Spec-to-code diff

The final validation step compares the implementation against the design plan. Did the files that changed match the files the plan predicted? Did architectural decisions in the code match the recorded decisions? Unexpected files in the diff are a signal -- either the plan was incomplete or the agent wandered. Both deserve attention before merge.

Validation layer	Catches
Unit and integration tests	Regressions and incorrect logic within scope
Lint and type checks	Style issues and type errors
Acceptance criteria walkthrough	Wrong behavior built to spec
Spec-to-code diff	Architectural drift and scope creep

Where AI Agents Fit in the Workflow

AI agents are accelerators on each phase, not replacements for review. The productive pattern is draft, review, refine, then proceed. Ask an agent to draft the requirements spec from a problem description, then edit intent until goals, non-goals, and acceptance criteria are right. Ask an agent to draft the design plan from the approved requirements, then review architecture decisions before any code exists. Ask an agent to implement one task slice at a time, with you approving each diff before the next task starts.

flowchart LR
  subgraph human [Human owns]
    H1[Intent and priorities]
    H2[Architecture approval]
    H3[Diff review at checkpoints]
    H4[Final acceptance]
  end
  subgraph agent [Agent accelerates]
    A1[Draft requirements]
    A2[Draft design plan]
    A3[Generate task list]
    A4[Implement task slices]
    A5[Draft tests]
  end
  H1 --> A1 --> H1
  A1 --> A2 --> H2
  H2 --> A3 --> A4 --> H3
  H3 --> A4
  A4 --> A5 --> H4

Agents are especially useful at producing first drafts and boilerplate tests. Humans are especially useful at catching wrong goals, unsafe architecture, and subtle scope creep. The workflow fails when either side is skipped -- when agents implement without specs, or when humans write specs without ever validating them against code.

This workflow article stays tool-neutral on purpose. Tool-specific execution guides -- editor setup, slash commands, agent configuration -- belong under the AI Developer Tools cluster. The process pillar lives here under documentation practices because the artifacts matter more than the vendor.

Common Mistakes That Kill Spec-Driven Development

Huge specs before any validation. A thirty-page requirements document written before a prototype or spike is waterfall paperwork, not SDD. Write the minimum spec that removes ambiguity for the next phase, then validate assumptions early. Not every feature needs the full five-phase loop -- Spec-Driven Development vs Vibe Coding explains when lighter structure is enough.

Vague acceptance criteria. Adjectives like "fast," "clean," and "user-friendly" are not acceptance criteria. Replace them with measurable behavior. If you cannot test it, you cannot implement it reliably -- especially with an AI agent.

Missing non-goals. Without non-goals, agents expand scope by default. They add caching layers, refactor neighboring modules, and introduce dependencies you did not ask for. Non-goals are how you say no in advance.

No test plan in the design phase. Tests written only after implementation tend to confirm what was built, not what was intended. The plan should name which acceptance criteria map to which test types before the first production file changes.

Skipping review at phase boundaries. The spec reviewed before the plan. The plan reviewed before tasks. Tasks reviewed before implementation. Each gate is cheap. Fixing drift after a large merge is expensive.

Letting generated tasks explode. Treat a fifty-item AI-generated task list as a first draft, not a schedule. Merge redundant items, split oversized ones, and delete tasks that do not map to a requirement.

SDD works when each phase reduces ambiguity. It fails when it creates paperwork.

Reusable Templates

Copy these into your repository and adapt them. Store specs alongside the feature branch, review them in pull requests, and keep them in version control so agents and humans read the same source.

Requirements template

# Feature -- [name]

## Problem
## Users
## Goals
## Non-goals
## Acceptance criteria
## Open questions

Design template

# Design -- [feature name]

## Summary
## Affected modules
## Data model changes
## API contracts
## Migrations
## Security
## Observability
## Test strategy
## Risks and mitigations

Task list template

# Tasks -- [feature name]

## Task 1 -- [title]
Depends on:
Files:
Satisfies:
Validate:
Review checkpoint:

## Task 2 -- [title]
...

Validation checklist

# Validation -- [feature name]

## Automated
- [ ] All tests pass
- [ ] Lint clean
- [ ] Type check clean

## Acceptance criteria
- [ ] AC-1 --
- [ ] AC-2 --

## Spec-to-code
- [ ] Changed files match plan
- [ ] No undocumented architectural changes
- [ ] Spec updated if implementation differed

Conclusion

Spec-driven development is not about writing more documents. It is about moving through specify, plan, task, implement, and validate with a review gate at each step. Each phase should leave the next actor -- human or agent -- with less guesswork than the phase before.

Start small. Run the full workflow on one medium-sized feature. Keep artifacts in markdown in the repository. Update the spec when reality diverges. Validate before merge. When the chain works, you get less drift, smaller reviewable diffs, and a durable record of intent that survives session resets and team handoffs.

When the chain becomes paperwork, cut scope -- not review. A two-page spec that was validated beats a thirty-page spec that nobody read.

Useful Links

GitHub Spec Kit documentation -- open-source toolkit that implements a similar specify-plan-tasks-implement loop
Martin Fowler on Spec-Driven Development tools -- analysis of Kiro, Spec Kit, and Tessl

GitHub Spec Kit vs Kiro vs Claude Code SDD Workflows

Rost — Sun, 12 Jul 2026 08:29:43 +0000

Developers comparing Spec-Driven Development setups in 2026 are usually not asking which model is smartest. They are asking which workflow will keep an AI agent aligned without burying them in ceremony.

GitHub Spec Kit, AWS Kiro, and Claude Code custom workflows all implement the same broad idea -- requirements, design, tasks, implementation, validation -- but they trade off portability, integration depth, and how much process they enforce.

If you need the concepts first, read What Is Spec-Driven Development? and the tool-neutral Spec-Driven Development Workflow guide in the App Architecture documentation cluster. This comparison sits in the AI Developer Tools hub alongside assistant reviews and workflow guides. It covers the three setups developers argue about most on Hacker News and Reddit-style forums, plus the lighter and heavier alternatives orbiting them.

SDD Is Becoming a Tool Category

Spec-Driven Development stopped being a paper exercise sometime in late 2025. Every major AI coding vendor now ships some version of specify-plan-implement, and a growing list of standalone tools competes on how much structure they add around that loop.

Tool / approach	Maintainer	Shape	Typical strength
GitHub Spec Kit	GitHub (open source)	CLI scaffolding, multi-file artifacts, 30+ agents	Portability across editors and agents
Kiro	AWS	Spec-native IDE (VS Code fork) plus CLI	Guided workflow inside one environment
Claude Code skills/commands	Anthropic ecosystem	Lightweight repo-local workflows	Fast to customize, easy to hack
OpenSpec	Fission AI (community)	Change-centric, fewer artifacts	Brownfield iteration with lower overhead
BMAD-METHOD	Community	Multi-agent, role-based ceremony	Large features with explicit role simulation
Tessl	Tessl (commercial, beta)	Spec-as-source code generation	Strong traceability, higher lock-in

The comparison that matters is not "which tool wins." It is process depth versus portability. Kiro is integrated. Spec Kit is portable. Claude Code workflows are hackable. Bad specs make every agent worse regardless of which wrapper you choose. Good specs travel across tools.

flowchart LR
  subgraph portable [Portable]
    SK[Spec Kit]
    CC[Claude Code skills]
    OS[OpenSpec]
  end
  subgraph integrated [Integrated]
    KI[Kiro IDE]
    TE[Tessl]
  end
  portable --> M[Markdown specs in Git]
  integrated --> E[Editor-native loop]

How to Compare SDD Setups

Before picking a tool, name what you are optimizing for. The same feature can feel effortless in one setup and bureaucratic in another depending on team size, codebase age, and how much review you need.

Portability -- Can the specs live as plain markdown in your repository and work with the agent you prefer next quarter? Or are they tied to one IDE, one cloud, or one proprietary format?

Setup friction -- How long from "I want to try SDD" to a working specify-plan-tasks loop? CLI scaffolding, IDE install, or rolling your own slash commands all have different activation energy.

Spec quality -- Does the tool help you write precise requirements and acceptance criteria, or does it mostly generate long documents? Structure is useful. Volume is not.

Task execution -- How does the tool break work into reviewable slices? Can tasks run in parallel? Does it resist fifty-item task explosions?

Review checkpoints -- Are there natural human gates between specify, plan, tasks, and implement? SDD without review is just slower vibe coding.

Repository grounding -- Does the workflow read project conventions, decision records, ADRs, AGENTS.md, and existing code before planning? Agents without grounding reinvent architecture because they never see the reviewed intent behind prior choices.

Team collaboration -- Can multiple people review the same spec artifacts in pull requests? Can you mix agents without rewriting the process?

Lock-in -- What do you lose if you switch editors, models, or cloud vendors in six months?

GitHub Spec Kit

GitHub Spec Kit is an open-source CLI toolkit that scaffolds a spec-driven loop into your repository and hands execution to whichever coding agent you already use. The specify CLI drops templates, slash commands, and a conventional folder layout. Typical commands follow a constitution-specify-clarify-plan-tasks-implement sequence, with an explicit clarify step to resolve ambiguity before architecture work begins.

Spec Kit's defining advantage is agent independence. The official docs position it as tooling that works with Claude Code, GitHub Copilot, Cursor, Gemini CLI, Codex, and dozens of other agents. You write specs once in markdown, commit them like code, and swap the executor without rewriting the process. That makes Spec Kit the default recommendation for teams that want SDD without betting on a single vendor.

The tradeoffs are real. Spec Kit can produce a large artifact tree -- constitution, spec, plan, tasks, contracts -- which pays off on multi-session features but feels heavy for a small CLI tweak. Hacker News threads regularly compare that overhead to waterfall ceremony. Spec Kit is also weaker if you want a fully integrated IDE where specs, tasks, and implementation live in one guided surface. It layers process on top of your existing editor rather than replacing it.

Strength	Limitation
Free, MIT licensed, repo-portable	No built-in IDE integration
Works with 30+ coding agents	Can generate verbose artifact sets
Explicit clarify and review phases	You assemble editor + agent + CLI yourself
Specs are plain markdown in Git	No automatic bidirectional spec sync

Spec Kit fits teams that already have a preferred AI coding assistant and want a standardized SDD scaffold on top. It is especially strong for greenfield features, multi-agent shops, and anyone who refuses editor lock-in.

AWS Kiro

Kiro is AWS's spec-driven IDE, built on a VS Code / Code OSS fork. Where Spec Kit brings SDD to your existing stack, Kiro assumes SDD deserves a purpose-built environment. A prompt generates structured artifacts -- typically requirements.md in EARS-style notation, design.md, and a dependency-sequenced tasks.md -- before agents write production code.

The guided experience is Kiro's main selling point. Requirements, design, and tasks are first-class UI objects beside your code, not files you manage through a separate CLI. Kiro also ships Agent Hooks, event-driven automations that can update tests, docs, or related artifacts when implementation changes. That bidirectional loop is something Spec Kit does not provide out of the box -- Spec Kit specs stay static until a human updates them.

The costs are integration depth traded for portability. Kiro runs inside its editor, uses AWS Bedrock-backed models, and bills through a credit-based pricing model with tiered plans. Enterprise teams already on AWS infrastructure often find that acceptable. Solo developers and multi-editor teams may not. Kiro also has rough edges typical of a newer IDE -- extension compatibility, workflow surprises, and the usual "do I really need another editor?" question.

Strength	Limitation
Tight requirements-design-tasks loop in one IDE	Editor and cloud ecosystem lock-in
EARS-style requirements rigor	Credit-metered pricing surface
Agent Hooks for spec-code sync	Weaker appeal outside AWS-native shops
Strong traceability from requirement to task	Harder to mix arbitrary external agents

Kiro fits developers who want the most guided SDD experience and are comfortable adopting a spec-native IDE. It is a strong option for enterprise teams, AWS-heavy environments, and anyone migrating from Amazon Q Developer who wants spec discipline without assembling the toolchain manually. If you live in standard VS Code today and love your current setup, Kiro asks for a bigger switch than Spec Kit does.

Claude Code Custom Commands and Skills

Claude Code does not ship a single official SDD product the way Spec Kit or Kiro do. If you are new to the tool itself, start with the Claude Code install and config guide for setup, permissions, and local backends. The SDD pattern itself lives in custom commands, skills, and repo-local markdown templates that developers maintain. Anthropic folded older .claude/commands/*.md files into the Skills mechanism, so the durable pattern is a SKILL.md (or equivalent) that defines your specify-plan-implement checklist, loaded on demand.

This approach is the lightest and most hackable. You can port a Kiro-style three-file layout, mirror Spec Kit phases with slash commands, or invent a minimal workflow that fits one repository. Claude Code reads CLAUDE.md for always-on project context and pulls skills when the task matches. That progressive disclosure keeps sessions focused without loading a full constitution on every prompt.

The downside is discipline. Nothing forces you through clarify or review gates unless you build those gates yourself. Reddit and Hacker News threads about "spec-driven development inside Claude Code" are full of developers who copied someone else's skill, ran it once, and went back to unstructured prompting when the skill felt slow. Claude Code SDD works when you treat skills like code -- versioned, reviewed, and maintained -- not like a one-time prompt download.

Strength	Limitation
Fast to customize per repo	No enforced workflow without your own rules
Portable markdown specs in Git	Quality depends entirely on author discipline
Skills reusable across compatible clients	No built-in multi-agent orchestration
Lowest ceremony for solo developers	Easy to drift back to vibe coding

For a serious implementation, read Claude Skills and SKILL.md for Developers and encode your phases as skills with explicit review checkpoints. Claude Code SDD is the right pick when you already live in Claude Code, want maximum flexibility, and will maintain the workflow yourself.

sequenceDiagram
  participant D as Developer
  participant S as Spec artifacts
  participant A as Coding agent
  Note over D,S: Spec Kit / Kiro / Claude skill
  D->>S: Specify requirements
  D->>S: Review and approve plan
  D->>S: Approve task list
  D->>A: Implement one task
  A->>D: Diff for review
  D->>S: Update spec if drift found

BMAD, OpenSpec, and Other Workflows

Not every team wants the Spec Kit artifact tree or the Kiro IDE. Two alternatives show up constantly in 2026 comparisons.

OpenSpec (Fission AI) takes a change-centric approach with fewer generated files than Spec Kit. Community benchmarks report materially lower token usage for comparable tasks, at the cost of less upfront structure. OpenSpec tends to win when you are modifying an existing codebase and want reviewable specs without an 800-line planning phase. It competes with Spec Kit on portability more than with Kiro on IDE integration.

BMAD-METHOD (community) pushes in the opposite direction -- multi-agent, role-based workflows that simulate product owner, architect, developer, and reviewer personas. BMAD can be powerful on large greenfield efforts where explicit role separation helps. It is also heavy. Teams frequently report that the ceremony only pays off when coordination pain is already acute.

Tessl treats the spec as the literal source of generated code, marking output as derived and discouraging hand-edits. That is the strongest "spec-as-source" stance among mainstream tools, but Tessl remains beta and carries the highest product lock-in of the group.

Spec Kitty and other community scaffolds sit between OpenSpec and Spec Kit on weight. They are worth watching if you want templates without adopting the full GitHub toolchain.

The pattern across all of them is the same. More process helps when ambiguity is expensive. More process hurts when feedback speed matters more than alignment. Match tool weight to task size, not to hype.

Which SDD Setup Should You Use?

There is no universal winner. The right setup depends on who you are, what you are building, and how much structure you will actually maintain.

Solo developer, existing codebase, small features. Start with Claude Code skills or OpenSpec. Write a short requirements block, a minimal task list, and one review checkpoint. Do not install a full Spec Kit tree for a fifty-line change.

Solo developer, greenfield feature, multiple sessions. Spec Kit or a well-maintained Claude Code SDD skill. You need durable artifacts more than IDE hand-holding.

Small team, mixed editors. Spec Kit. Plain markdown specs in Git, reviewed in pull requests, executed by whichever agent each developer prefers.

Enterprise team, AWS-native, compliance pressure. Kiro. Guided artifacts, requirement traceability, and hooks that keep docs and tests closer to implementation.

Regulated environment. Kiro or Spec Kit plus your own validation checklist -- not Claude Code skills alone unless you encode compliance gates explicitly. Tooling does not replace audit trails. It only makes them easier to produce.

Existing codebase, brownfield change. OpenSpec or a lightweight Claude Code workflow. Full Spec Kit ceremony on every bugfix will feel like waterfall. Reserve heavier structure for cross-cutting features.

Greenfield product, many agents. Spec Kit. Portability matters more than IDE polish when Copilot, Claude Code, and Cursor may all touch the same repo.

Teams experimenting with multi-agent orchestration should also look at Oh My OpenCode Agents for patterns on splitting roles across agents -- complementary to SDD artifacts, not a replacement for them.

Practical Decision Table

If you want...	Start here	Why
Least lock-in	Spec Kit or plain markdown + Claude skills	Specs in Git, swap agents freely
Best guided IDE experience	Kiro	Requirements, design, tasks built into the editor
Claude Code only, minimal setup	Custom SDD skill in `.claude/skills/`	Fast, hackable, repo-local
Team review in pull requests	Spec Kit or OpenSpec	Markdown artifacts diff cleanly in PRs
Security / compliance traceability	Kiro + explicit validation checklist	Requirement-to-task mapping plus hooks
Lowest token overhead	OpenSpec or lightweight Claude workflow	Fewer generated artifacts per change
Maximum process for large builds	BMAD-METHOD	Role-based multi-agent ceremony
Spec literally drives generated code	Tessl (evaluate beta risk)	Strongest spec-as-source model

flowchart TD
  Q1{Need a new IDE?}
  Q1 -->|Yes, AWS OK| K[Kiro]
  Q1 -->|No| Q2{Team uses many agents?}
  Q2 -->|Yes| SK[Spec Kit]
  Q2 -->|No| Q3{Already on Claude Code?}
  Q3 -->|Yes| CC[Claude Code SDD skill]
  Q3 -->|No| SK
  Q4{Brownfield small change?}
  Q4 -->|Yes| OS[OpenSpec or minimal spec]
  Q4 -->|No| SK

What Actually Determines Success

Tool choice matters less than artifact quality. A Kiro requirements file with vague acceptance criteria will produce the same drift as a sloppy Claude Code prompt. A Spec Kit plan that lists fifty redundant tasks will feel like waterfall regardless of which agent implements it.

The practices that travel across every setup are boring and effective. Keep specs small enough to review in one sitting. Write non-goals explicitly. Break tasks into diffs a human can read. Validate against acceptance criteria before merge. Update the spec when implementation discovers a better path.

If you are still choosing between SDD and unstructured prompting for a given feature, read Spec-Driven Development vs Vibe Coding. The tool comparison in this article only matters once you have decided the feature deserves a spec at all.

Bad specs make every agent worse. Good specs travel across tools.

Conclusion

GitHub Spec Kit, Kiro, and Claude Code workflows are three answers to the same question -- how do you keep AI agents aligned across sessions -- with different bets on portability versus integration. Spec Kit optimizes for agent-agnostic markdown in your repository. Kiro optimizes for a guided spec-native IDE with AWS-backed agents. Claude Code skills optimize for hackable, lightweight workflows that succeed only when you maintain them.

Pick the shallowest setup that still removes ambiguity for the feature at hand. Add structure when coordination pain appears, not when a blog post tells you to. The developers who get value from SDD in 2026 are not the ones with the most elaborate toolchain. They are the ones who write specs worth implementing -- then let whichever tool they chose execute against them.

Useful Links

GitHub Spec Kit documentation -- official Spec Kit workflow reference
Martin Fowler on SDD tools -- analysis of Kiro, Spec Kit, and Tessl

A2A and MCP Agent Security: Identity, Delegation, and Audit Trails

Rost — Sat, 11 Jul 2026 06:46:43 +0000

Prompt injection gets most of the security attention in LLM systems, and it deserves attention, but it is not the whole problem once agents start calling tools and delegating work to other agents.

MCP gives an agent structured access to files, APIs, databases, and ticketing systems. A2A lets one agent send tasks, messages, and artifacts to another agent that may belong to a different team, vendor, or runtime. Those protocols are useful precisely because they cross trust boundaries, which means identity, authorization, delegation limits, and audit trails become first-class architecture rather than optional hardening.

This article is the canonical guide for agent protocol security in the LLM Architecture cluster. It covers threat models, identity, gateways, registries, delegation, and production checklists. For input validation, output filtering, and prompt safety patterns, see LLM Guardrails in Practice instead.

Guardrails vs Protocol Security vs Runtime Policy

These three layers solve different problems and fail in different ways when conflated.

LLM guardrails operate on model input and output: blocking injection patterns, filtering harmful content, validating JSON shape, and enforcing tone or compliance rules on generated text. They protect the conversation layer.

Protocol security operates on agent boundaries: who may call which MCP tool, which agent may delegate to which peer, what OAuth scopes attach to a task, and whether a downstream agent may act on a user's behalf. It protects the action layer.

Runtime policy sits between them: a policy engine that evaluates requests against rules regardless of whether the trigger was natural language or a structured protocol call. It can require human approval before a tool executes, block egress to unknown domains, or deny delegation when scope exceeds the originating user.

My opinion is blunt: guardrails without protocol security produce polite chatbots that still exfiltrate data through a tool call. Protocol security without guardrails produces well-authenticated agents that still follow malicious instructions embedded in an artifact. You need both, plus runtime policy for high-risk actions.

Threat Model for A2A and MCP Agent Systems

Start with assets and adversaries, not with a shopping list of controls.

Assets worth protecting: user data in prompts and artifacts, credentials for MCP servers, production systems reachable through tools, agent reputation, billing accounts tied to token usage, and audit integrity.

Realistic adversaries: external users abusing public agent endpoints, compromised MCP servers returning poisoned tool results, malicious agents misrepresenting skills in Agent Cards, insiders over-delegating authority, and supply-chain tampering with tool metadata that manipulates model behavior.

Malicious or compromised tools (MCP)

An MCP server is code plus data exposed to the model. A hostile server can return misleading tool descriptions, exfiltrate arguments passed by the model, or perform actions beyond what the user intended when the host executes tool calls without scoped credentials.

Malicious or impersonated agents (A2A)

An agent that accepts tasks may be evil, compromised, or simply over-permissioned. Agent Cards describe capabilities; they do not prove identity unless you verify signatures, TLS, and issuer trust.

Confused deputy

Agent B holds permission to access a finance API. Agent A, with lower privilege, asks B to "summarize this invoice" while smuggling a transfer instruction in an artifact. B executes using its own credentials unless delegation scope is enforced end to end.

Over-broad permissions and hidden delegation chains

User approves one step. The orchestrator silently chains three A2A hops and five MCP calls. The user never sees the full graph, but the organization is still accountable for the outcome.

Prompt injection through artifacts and cross-agent messages

Injection is not only a user-message problem. A PDF artifact, a web page fetched by a tool, or a message from Agent C can carry instructions aimed at Agent D's model. Treat all protocol-carried content as untrusted input at the model boundary.

Poisoned or misleading Agent Cards

Descriptions and skill names are prompt surface area. A card that advertises safe_read_only_analysis while accepting write-capable backends is a social-engineering layer, not a technical guarantee.

Identity Model for Multi-Agent Systems

Protocol security begins with clear identity types and what each one is allowed to prove.

Identity type	What it represents	Typical proof
Human user	End user or operator who initiated work	OIDC session, SSO token
Agent service	Deployed agent runtime (orchestrator, specialist)	OAuth client credentials, mTLS cert
MCP server	Tool provider process	API key, mTLS, scoped service account
Task / session	Unit of work spanning hops	task ID, trace ID, delegated scope token

A2A's Agent Card advertises supported authentication schemes (OAuth 2.0, API keys, mTLS, and similar patterns aligned with OpenAPI practice) and skills with optional security requirements. The card is discovery metadata, not a trust anchor. Clients obtain credentials out of band and send them in standard HTTP headers on every request; servers must validate on every call and return 401 or 403 when auth or scope fails.

Internal vs external views of the same agent

Production agents often publish a public Agent Card with a limited skill list and a richer authenticated card for internal callers. The A2A specification allows extended cards for authenticated clients. Use that split deliberately: partners should not see internal skills, and internal orchestrators should not rely on public discovery alone for authorization.

Authentication and Authorization for MCP and A2A

Authentication answers who is calling. Authorization answers what they may do.

MCP tool access

For each MCP connection, define:

which agent host may connect
which tools are enabled for that host
which OS user or service account executes side effects
whether the human user must approve each mutating call

Prefer tool allowlists over "connect everything" MCP configs. A coding agent does not need payroll MCP servers on the same profile as a public support bot.

A2A agent access

For each agent peer relationship, define:

which caller agent IDs may invoke which skills
maximum delegation depth
which artifact types may cross the boundary
whether user context must propagate as signed claims

Map OAuth scopes (or equivalent) to skills, not to blanket agent admin. Least privilege at the token layer beats hope at the prompt layer.

Gateway-enforced vs per-agent policy

Per-agent policy works when one team owns the whole graph and releases are coordinated. Gateway-enforced policy works when multiple teams, tenants, or vendors share an agent network and you need one place to enforce allowlists, rate limits, and audit.

flowchart LR
    U[User / client] --> G[A2A gateway]
    G --> O[Orchestrator agent]
    O -->|A2A scoped token| S1[Specialist agent]
    O -->|A2A scoped token| S2[Specialist agent]
    S1 --> MG[MCP gateway]
    S2 --> MG
    MG --> T1[MCP tool servers]
    MG --> T2[MCP tool servers]
    G --> A[Audit log]
    MG --> A
    S1 --> A
    S2 --> A

A2A Gateway as the Control Plane

An A2A gateway is not strictly required by the protocol, but it becomes necessary when agent traffic needs centralized governance.

A gateway typically handles:

authentication termination and token exchange
routing to the correct agent service by skill or tenant
policy checks before tasks are accepted or forwarded
protocol version negotiation
rate limiting and abuse detection
structured audit emission on every task transition

When a gateway is overkill vs necessary

A gateway is often overkill for a single orchestrator and two specialist agents in one Kubernetes namespace maintained by one team. It becomes necessary when partners invoke your agents, when multiple business units share infrastructure, when compliance requires uniform logging, or when you cannot trust every agent implementation to enforce policy correctly.

Pair an A2A gateway with an MCP gateway (or MCP proxy) so tool access receives the same treatment: identity, allowlists, egress controls, and audit at the tool boundary rather than only at the chat UI.

Partner-facing vs internal Agent Cards

Publish different discovery metadata for external and internal callers. External cards expose narrow skills and stricter auth. Internal cards may list maintenance or admin skills but must never be reachable without stronger authentication than the public card implies.

Agent Registry and Discovery Security

Discovery is part of the attack surface. Anyone who controls what agents appear "available" controls where orchestrators send work.

Registry vs well-known Agent Card URLs

Small deployments use well-known URLs per agent (/.well-known/agent-card.json). Enterprise deployments add a registry that indexes agent IDs, versions, endpoints, owners, and policy tags. The registry is a policy object: entries should record which tenants may discover which agents, not only where they live.

Versioning, deprecation, and ownership

Registry records need owners, change history, and deprecation dates. An orchestrator that caches Agent Cards must refresh on TTL and verify signatures where supported. Stale cards are how retired skills keep receiving traffic long after a vulnerability is patched.

Enterprise internal networks vs external partners

Internal agent meshes can rely on mTLS and private DNS. Partner agents need explicit federation rules, contractually scoped skills, and stronger artifact inspection because you do not control their runtime.

Delegation Across Agent Boundaries

Delegation is where A2A security is won or lost. When Agent A sends a task to Agent B, three questions must have crisp answers:

Whose authority is being exercised? The user's, A's service account, or a blended delegated token?
What is B allowed to do with that authority? Read-only analysis, or mutating tools on A's behalf?
Who is accountable if B exceeds scope? A, B, the gateway policy, or the human who approved an unclear prompt?

Propagating user intent vs over-delegation

Pass signed delegation claims that include user ID, original task ID, allowed skills, expiry, and maximum hop count. Downstream agents must reject tasks that expand scope silently. If B needs higher privilege than A held, transition to input_required and obtain explicit human approval rather than upgrading tokens invisibly.

Human-in-the-loop approval flows for risky delegation are covered in A2A Streaming and Async Tasks for Long-Running Agent Workflows where input_required is a first-class task state rather than an error.

sequenceDiagram
    participant User
    participant Orch as Orchestrator agent
    participant GW as A2A gateway
    participant Spec as Specialist agent
    participant MCP as MCP tool server
    User->>Orch: Request with user token
    Orch->>GW: Delegate task (scoped delegation token)
    GW->>GW: Policy check scope + hop count
    GW->>Spec: Forward task (reduced scope token)
    Spec->>MCP: Tool call (tool-scoped credential)
    MCP->>MCP: Enforce allowlist + user context
    Spec-->>GW: Artifact + audit events
    GW-->>Orch: Task update
    Orch-->>User: Final response

Separate reasoning from execution permissions

An agent may need broad read access to plan while write tools sit behind approval. Split credentials or use distinct MCP profiles for planning vs execution so a model mistake cannot immediately mutate production.

Audit Trails and Answer Provenance

If you cannot reconstruct a delegation chain, you cannot explain an incident, pass an audit, or dispute a billing anomaly.

Log at three layers:

Gateway: authentication result, policy decision, routed agent ID, task ID, parent task ID, rate-limit events.

Agent: task state transitions, messages sent/received, model/tool invocations (arguments redacted as needed), artifacts created, delegation outward.

MCP server: tool name, caller agent ID, user context, success/failure, latency, rows affected or resource IDs (policy permitting).

Correlate with trace ID across all layers. Observability for LLM Systems covers instrumentation backends; this article defines what must be captured so those backends have meaningful signal.

Final answer provenance should answer: which user, which orchestrator task, which specialist agents, which tools, which artifacts influenced the text the user saw, and which policy gates fired along the way.

Runtime Policy, Egress, and Secrets

Runtime policy engines (OPA, Cedar, custom rule services) evaluate structured events: "tool X with args Y for user Z." They complement guardrails because they do not depend on the model behaving well.

Human approval belongs in runtime policy for irreversible or high-cost actions: payments, external email, production config changes, privilege grants.

Egress controls limit which domains MCP servers and agents may call. An agent that can both read secrets and POST to arbitrary URLs is a data-loss waiting to happen.

Secrets never belong in Agent Cards or prompts. MCP hosts should inject short-lived credentials at execution time from a secrets manager. For transport encryption, key management, and baseline infra security patterns, see Architectural Patterns for Securing Data.

Push notification webhooks in async A2A flows need the same rigor: verify sender identity, reject stale events, and never treat a webhook payload as authorization on its own.

Reference Security Architecture

The following diagram summarizes a production-oriented layout for A2A outside, MCP inside deployments at scale.

flowchart TB
    subgraph Client layer
        U[User / API client]
    end
    subgraph Control plane
        GW[A2A gateway]
        REG[Agent registry]
        POL[Policy engine]
        AUD[Audit log]
        SEC[Secrets manager]
    end
    subgraph Agent layer
        OR[Orchestrator]
        SA[Specialist agents]
    end
    subgraph Tool layer
        MG[MCP gateway]
        MCP[MCP servers]
    end
    subgraph Observability
        OBS[Tracing + metrics]
    end
    U --> GW
    GW --> REG
    GW --> POL
    GW --> OR
    OR --> GW
    GW --> SA
    SA --> MG
    MG --> MCP
    POL --> GW
    POL --> MG
    SEC --> SA
    SEC --> MCP
    GW --> AUD
    MG --> AUD
    SA --> AUD
    AUD --> OBS

The orchestrator sees specialist agents through A2A. Specialists see tools through MCP. Users never receive raw MCP credentials, and partners never receive internal skill surfaces without policy review.

For protocol concepts (Agent Cards, tasks, artifacts), see What Is the A2A Protocol?. For adoption and enterprise framing, see Google A2A Protocol in 2026. For topology when many agents coordinate, see Multi-Agent Orchestration Patterns.

Production Checklist for A2A and MCP Security

Before exposing agent protocols beyond a trusted sandbox, verify:

Identity and auth

[ ] No anonymous agents in production paths
[ ] Every MCP and A2A call authenticated on every request
[ ] OAuth scopes or equivalent mapped to skills/tools, not blanket admin
[ ] Public vs authenticated Agent Card views defined intentionally

Delegation and policy

[ ] Delegation tokens carry user ID, task ID, scope, expiry, hop limit
[ ] Downstream agents reject scope expansion without explicit approval
[ ] High-risk tools require runtime policy or human approval
[ ] Reasoning and execution use separate credentials where possible

Discovery and registry

[ ] Agent registry entries have owners and version history
[ ] Agent Cards refreshed on TTL; signatures verified where supported
[ ] Partner agents federated with explicit skill allowlists

Audit and observability

[ ] Gateway, agent, and MCP layers emit correlated audit events
[ ] Delegation chains logged with parent and child task IDs
[ ] Artifact provenance recorded for final answers
[ ] Trace IDs connect to observability backends

Abuse and resilience

[ ] Rate limits per user, agent, and tenant
[ ] Timeout policies on delegated tasks
[ ] Egress allowlists on tool servers
[ ] Secrets in a manager, not in cards, prompts, or repos

Conclusion

A2A and MCP interoperability is powerful because agents and tools can compose across team and vendor boundaries, but that power is unsafe without identity, authorization, delegation limits, and audit design. Guardrails protect the model conversation; protocol security protects the actions agents take on behalf of users.

Treat Agent Cards as advertisements, delegation as a signed contract, MCP tools as privileged code execution, and audit logs as the evidence chain you will need when something interesting happens at 2 a.m.

Build the gateway when governance needs a single throat to choke. Split credentials before you split agents. Log every hop so the answer "the model decided" is never the final incident report.

Frequently Asked Questions

What is the difference between LLM guardrails and A2A MCP agent security?
Guardrails constrain model input and output. Protocol security constrains who may invoke tools, delegate tasks, and act on whose behalf across MCP and A2A with identity, authorization, and audit trails.

How should agent identity work in an A2A deployment?
Separate human, agent service, and task identities. Validate credentials on every request, use scoped tokens, and treat Agent Cards as discovery metadata rather than proof of trust.

What is the confused deputy problem in multi-agent systems?
It occurs when a privileged agent or tool performs a sensitive action because a less privileged caller smuggled instructions through delegation or artifacts. Enforce scope at every hop.

Do you need an A2A gateway in production?
Single-team internal deployments may enforce policy per agent. Multi-tenant, multi-vendor, or partner-facing networks usually need a gateway for centralized auth, routing, rate limits, and audit.

What should an A2A MCP audit log contain?
User ID, agent ID, task ID, parent task ID, tool calls, policy decisions, artifacts, and timestamps correlated with trace IDs across gateway, agent, and MCP layers.

Sources

A2A Protocol -- Enterprise-ready security topics: https://github.com/a2aproject/A2A/blob/main/docs/topics/enterprise-ready.md
A2A Protocol -- Specification overview: https://a2a-protocol.org/latest/specification/
A2A Protocol -- Streaming and push notification security: https://a2a-protocol.org/latest/topics/streaming-and-async/

A2A Streaming and Async Tasks for Long-Running Agent Workflows

Rost — Fri, 10 Jul 2026 09:36:29 +0000

Most AI agent demos still behave like chat completions with extra steps: you send a prompt, wait a few seconds, and get an answer back in one response.

Real agent work often does not fit that pattern. Research, code review, procurement analysis, incident investigation, and multi-step planning can run for minutes or hours, and they may need clarification halfway through, stream partial results, delegate to another agent, and produce files rather than a single text reply. That is where the A2A protocol's async model matters within the broader AI Systems cluster, because A2A treats long-running work as a Task with a lifecycle instead of a one-shot HTTP response. Clients can stay connected via Server-Sent Events (SSE), poll task state, or register push webhooks when they cannot hold a connection open.

This article covers operational design for those workflows, including when to stream versus poll versus push, how input_required fits human-in-the-loop flows, failure handling, and what to instrument in production. For Agent Cards, messages, parts, and the full task model, see What Is the A2A Protocol? Agent Cards and Tasks Explained.

Why Long-Running A2A Agent Tasks Need Async Design

A synchronous request/response mental model breaks down quickly once agent work spans tools, delegation, approvals, and large artifacts. An agent task may call multiple MCP servers internally, delegate sub-work to another agent over A2A, wait for human approval, generate large artifacts in chunks, fail partway through and need partial recovery, and accumulate token cost across several hops. HTTP APIs can approximate this with timeouts, background jobs, and ad hoc status endpoints, but A2A bakes task identity and state into the protocol so clients and gateways can reason about work consistently. For how those layers fit inside a production assistant before you add async A2A boundaries, see AI Assistant Architecture: LLM, Memory, Tools, Routing, Observability.

My bias is practical: do not create a Task for everything, because a one-line summary does not need a lifecycle. Use a Task when work is stateful, auditable, long-running, artifact-producing, or may need input mid-flight. The rule of thumb from the explainer still holds: simple interactions can return a Message, while complex work should return a Task.

A2A Task Lifecycle and State Transitions

An A2A Task moves through states that clients can query at any time. Exact naming varies slightly by implementation, but the model is stable across servers that follow the protocol.

stateDiagram-v2
    [*] --> submitted
    submitted --> working
    working --> input_required
    input_required --> working
    working --> completed
    working --> failed
    working --> canceled
    working --> rejected
    submitted --> rejected
    input_required --> failed
    input_required --> canceled
    completed --> [*]
    failed --> [*]
    canceled --> [*]
    rejected --> [*]

The submitted state means the client sent work and the agent accepted or queued it. In working, the agent is actively processing, which may include tool calls, delegation, or streaming partial output. The input_required state indicates the agent paused because it needs more input, clarification, or human approval, and it is not a failure state. completed is terminal success with artifacts available; failed is a terminal error whose details and partial artifacts depend on implementation; canceled means a client, gateway, or authorized caller stopped the task; and rejected means the agent refused the task because of policy, capability mismatch, or auth.

When input_required pauses versus fails a workflow

Treat input_required as a deliberate pause, not an exception. The agent is telling you it cannot proceed without something from you, whether that is a missing parameter, a policy confirmation, or a manager sign-off on a high-risk action. A workflow fails when the task reaches failed or rejected, or when a caller exceeds a timeout waiting for input that never arrives, so you should design explicit timeouts for human steps rather than letting approvals sit indefinitely.

An approval that waits three days without escalation is a stuck workflow, not a patient one, and stuck workflows clog task stores while making observability dashboards harder to read.

Who can cancel an A2A task

Cancellation authority should be defined at design time rather than debated during an incident. The client usually can cancel tasks it created; a gateway may cancel on behalf of tenants, policy violations, or budget limits; and an upstream agent may cancel delegated work when orchestrating over A2A if the protocol and policy allow it. Log who canceled and why, because in multi-agent chains orphan work is a common source of surprise token bills.

Human-in-the-Loop with input_required Task States

input_required is one of A2A's most underused design features, and many teams treat it as an error code when it is actually a first-class workflow state. In production you will hit cases where the agent should stop, such as spending budget on an ambiguous request, executing an irreversible action, accessing sensitive data without scope confirmation, or delegating to a specialist that needs explicit user intent. Model these as deliberate transitions to input_required, with a clear message explaining what is needed.

Approval flows for risky A2A delegation

When Agent A delegates to Agent B over A2A and Agent B enters input_required for human approval, three systems need to agree on what happens next. The downstream agent pauses and exposes what it needs, the orchestrator or gateway surfaces that pause to the user, and the user's response resumes the task via a new message. The A2A vs MCP comparison explains why delegation across agent boundaries is a different problem from tool access, and why approval semantics belong at the task layer rather than inside a single MCP call. Do not silently auto-approve because the UX is inconvenient, since expensive mistakes usually come from convenience shortcuts rather than from missing models.

UX patterns for paused A2A tasks

Blocking wait means the UI shows a spinner or approval card until the task leaves input_required, which works well for short human steps. Non-blocking wait means the client records the task ID, lets the user continue elsewhere, and uses polling or push to notify when input is needed again, which is required for mobile, email-linked approvals, or multi-tab assistants. Timeout when humans are slow means defining an SLA per step and, after N hours, transitioning to failed or escalating to another queue, because unbounded waits clog task stores and confuse observability dashboards.

How an A2A gateway handles input_required

If you run an A2A gateway, decide whether it forwards input_required events transparently, aggregates pauses from multiple downstream agents into one user prompt, or enforces that certain skills always require approval before leaving input_required. Auth and policy for approved actions belong in a dedicated security article; for now, assume every resumed task should carry the same user identity and scope as the original request.

Choosing Sync, SSE Streaming, Polling, or Push Notifications

A2A supports multiple interaction modes, and the right choice depends on client capabilities and latency needs rather than on which mode sounds most modern.

Mode	Best for	Client requirements	Tradeoffs
Sync (SendMessage, short Task)	Quick work, immediate Messages	Simple HTTP client	Timeouts on slow agents
SSE streaming	Live progress, incremental artifacts	Long-lived connection	Proxies, mobile background limits
Polling (GetTask)	Batch clients, simple integrations	Timer + task ID	Higher latency, more requests
Push webhooks	Mobile, serverless, multi-hour jobs	HTTPS receiver + verification	Async complexity, security hardening

Read Agent Card capability flags first

Before choosing a mode, read the agent's Agent Card, because streaming requires capabilities.streaming: true and push notification support is advertised separately. Clients that assume every agent streams will break against minimal implementations, so negotiation is not ceremonial: it prevents runtime failures when a specialist agent only supports poll-based status checks.

When to use assistant-side polling around A2A

Your assistant runtime may wrap A2A task polling in a scheduler loop rather than exposing raw protocol details to the user. That pattern overlaps with general polling agents, which are background processes that wake up, check state, and act. For durable scheduling, idempotency, and queue patterns outside A2A specifically, see Polling Agents in AI Assistants: 11 Implementation Patterns. Use assistant polling when you orchestrate many A2A tasks from a single control plane, and use native A2A streaming or push when the client connects directly to the agent boundary.

A2A Server-Sent Events (SSE) Streaming

SSE is A2A's primary real-time channel. The client calls SendStreamingMessage, opens an HTTP connection, and receives a text/event-stream response until the task reaches a terminal or interrupted state. Each event's payload is JSON-RPC-shaped, and typical result types include a Task snapshot, a TaskStatusUpdateEvent for lifecycle transitions and intermediate agent messages, and a TaskArtifactUpdateEvent for chunked artifact delivery with append and lastChunk hints for reassembly.

sequenceDiagram
    participant Client
    participant A2A Server
    Client->>A2A Server: SendStreamingMessage
    A2A Server-->>Client: HTTP 200 text/event-stream
    loop Until terminal or input_required
        A2A Server-->>Client: TaskStatusUpdateEvent
        A2A Server-->>Client: TaskArtifactUpdateEvent (optional)
    end
    A2A Server-->>Client: Close stream
    Note over Client,A2A Server: On disconnect before terminal state,<br/>client may call SubscribeToTask

Streaming progress updates and partial artifacts

Streaming shines when users should see work happening, whether that means step counters ("3 of 7 sources reviewed"), partial text generation, incremental file chunks for large reports, or state transitions from working to input_required without polling. Design UI around event types rather than around a single final blob, because if you only display output when completed arrives you might as well poll.

SSE connection drops and resubscription

Networks drop, laptops sleep, and load balancers idle-timeout SSE connections, so long streams need recovery logic rather than optimistic assumptions. A2A provides SubscribeToTask so clients can reconnect to an in-progress task stream, and your client SDK should persist taskId locally, detect stream closure before terminal state, resubscribe with backoff, and de-duplicate events if the server replays overlapping state. Without resubscription logic, long tasks feel fragile in production even when the agent backend is healthy.

A2A Push Notifications and Webhooks

Push fits scenarios where SSE is a poor match, such as mobile apps in the background, serverless handlers, or tasks that run for hours or days. The client supplies a PushNotificationConfig with a url (HTTPS webhook on the client side), an optional token for validating incoming POSTs, and optional authentication details for how the A2A server authenticates to the webhook. Configuration can ride along with the initial SendMessage or SendStreamingMessage call, or be added later via CreateTaskPushNotificationConfig for an existing task.

sequenceDiagram
    participant Client
    participant A2A Server
    participant Webhook
    Client->>A2A Server: SendMessage + PushNotificationConfig
    A2A Server-->>Client: taskId
    Note over A2A Server: Task runs asynchronously
    A2A Server->>Webhook: POST state change notification
    Webhook->>A2A Server: GetTask(taskId)
    A2A Server-->>Webhook: Updated Task + artifacts
    Webhook->>Client: Resume workflow / notify user

When a significant update occurs, the A2A server POSTs to the webhook and the client typically calls GetTask with the notified taskId to fetch the full updated Task and artifacts. Push is a signal, not a full payload transport.

When push beats an open SSE connection

Prefer push when the client cannot maintain SSE (mobile, edge functions), when updates are infrequent and milestone-based rather than token-by-token, or when you want the server to wake a disconnected workflow engine. Prefer SSE when users watch progress live, when artifacts stream in many small chunks, or when latency below a few seconds matters.

Correlating push notifications to A2A tasks

Every push handler should log and propagate the taskId, a trace or correlation ID from the original request, the event type or state transition, and a timestamp from the notification so stale events can be rejected. Replay attacks and duplicate deliveries happen in production, so idempotent handlers are not optional.

Push endpoint security overview

Push introduces SSRF risk on the server when malicious clients register internal URLs, and impersonation risk on the client when fake POSTs arrive at the webhook. Mitigations include URL allowlists, ownership verification, signed JWTs with JWKS, timestamp checks, and validating the config token. The full threat model, identity layers, and gateway controls live in A2A and MCP Agent Security: Identity, Delegation, and Audit Trails; until you have read it, treat webhook verification with the same seriousness as payment callbacks.

Async A2A Workflow Patterns

Fire-and-follow task submission

The client submits a task, receives a task ID immediately, and disconnects, then later polls GetTask or waits for push. This is the default pattern for serverless and batch pipelines, but you should persist the task ID in durable storage before acknowledging the user, because serverless invocations that forget the ID lose the work.

Resuming a task after input_required

After input_required, the user sends a new message against the same task and the agent transitions back to working. Design messages so resumption context is explicit, because "Approved: proceed with vendor X" beats a bare "yes" when you need to audit what was approved six hours later.

Chained A2A delegation with intermediate artifacts

Consider a research workflow where an orchestrator owns Task T1 and delegates retrieval, summarization, and verification to specialist agents, each with its own task ID and artifacts along the way.

flowchart TD
    U[User] --> O[Orchestrator Task T1]
    O -->|A2A| R[Retrieval agent T2]
    R --> A2[artifact: raw sources]
    O -->|A2A| S[Summarization agent T3]
    S --> A3[artifact: draft summary]
    O -->|A2A| V[Verification agent T4]
    V --> A4[artifact: fact-check report]
    O --> F[final artifact: recommendation memo]

Each hop has its own task ID and state machine, so the orchestrator should stream or poll downstream tasks independently, persist intermediate artifacts before starting the next hop, and fail gracefully if T3 completes but T4 rejects the draft. Multi-Agent Orchestration Patterns covers topology choice when those specialists run as separate services rather than in one runtime. Partial progress is valuable, and a failed verification should not delete a usable draft without a clear reason.

Durable task storage for delayed completion

Task state and artifacts should survive process restarts. If your agent runs in Kubernetes, assume pods die mid-task and back task records and artifact blobs to a store the agent container does not own exclusively.

Failure Handling for Long-Running A2A Workflows

Long-running workflows fail in predictable ways through timeouts, retries, partial artifacts, and unsafe cancellation, and each needs an explicit policy rather than ad hoc handling in client code.

Per-hop and end-to-end timeout budgets

Set timeouts at two levels: a per-hop maximum for one agent task before escalation or cancel, and an end-to-end maximum for the user-visible workflow. A retrieval agent that hangs should not block the entire orchestrator until the user's browser times out.

Retries and idempotency for A2A tasks

Retries without idempotency duplicate side effects such as double charges, duplicate tickets, and repeated emails. Use stable client message IDs or idempotency keys where the protocol allows, and for business mutations align with Idempotency in Distributed Systems That Actually Works. Retry only transient failures like network blips or 503s, and do not retry rejected or policy failures blindly because you will amplify cost and annoy downstream agents.

Partial artifact recovery policies

When a task fails after producing partial artifacts, define whether you expose partial output to the user with a clear "incomplete" label, allow resume from the last good checkpoint, or discard partial output when it could mislead in medical, legal, or financial contexts.

Safe cancellation across delegation chains

Cancel downstream tasks when an upstream user aborts, use a delegation graph so cancel propagates, and log canceled tasks that already incurred cost because finance teams notice.

Observability for Async A2A Workflows

You cannot debug multi-agent async work unless you can trace it across boundaries, which means correlating identifiers on every hop rather than relying on unstructured logs. Minimum correlation fields include a trace ID per user-initiated workflow, a task ID per agent task including delegated children, an agent ID for the Agent Card or service that handled the hop, and a parent task ID that links delegation chains.

Log every state transition with timestamps, and log artifact creation events with size and hash rather than necessarily full content when PII policies apply. Attribute cost and latency per hop, because multi-agent workflows hide token spend until the bill arrives and per-task cost labels make "which specialist is expensive?" answerable. For metrics, tracing backends, and LLM-specific instrumentation patterns, see Observability for LLM Systems and the broader Observability pillar for how those signals fit into a production telemetry stack.
When a user asks "why did the agent do that?", your answer should be a trace spanning orchestrator, A2A hops, MCP tool calls, and any input_required pauses rather than a shrug and a log grep.

Production Checklist for A2A Streaming and Async Tasks

Before shipping long-running A2A paths to production, verify the following areas.

Agent Card and capabilities

[ ] capabilities.streaming reflects actual SSE support
[ ] Push notification support documented if implemented
[ ] Skills that require human approval document expected input_required behavior

Client modes

[ ] SSE client handles resubscription via SubscribeToTask
[ ] Poll interval backs off under load
[ ] Push webhook verifies authenticity and rejects stale events

Durability

[ ] Task state survives agent process restarts
[ ] Artifacts stored outside ephemeral container filesystem
[ ] Intermediate artifacts available for partial recovery

Failure and policy

[ ] Per-hop and end-to-end timeout budgets defined
[ ] Retries idempotent for mutating operations
[ ] Cancel propagates across delegation edges

Observability

[ ] trace ID + task ID + agent ID on every hop
[ ] State transitions logged
[ ] Cost attribution per task or per agent

Load testing

[ ] SSE through your reverse proxy (buffering breaks streams)
[ ] Concurrent long tasks without memory leaks on open connections
[ ] Push flood handling without webhook overload

Conclusion

A2A's value shows up most clearly when work does not fit a single synchronous API call, because streaming, async tasks, push notifications, and explicit task states are how the protocol handles real agent workloads such as research, delegation, approvals, and large artifacts without pretending everything completes in one HTTP round trip. Start with the simplest mode that works, add SSE when users need live progress, add push when connections cannot stay open, treat input_required as a first-class design tool rather than a failure, and instrument every hop so multi-agent async workflows do not outrun your ability to explain them.

Frequently Asked Questions

When should you use A2A streaming instead of polling?
Use streaming when the client can hold an open HTTP connection and you need low-latency progress updates or incremental artifacts. Use polling when connections are unreliable, clients are batch-oriented, or you only need periodic status checks on long-running tasks.

What does input_required mean in an A2A task?
It is a pause state where the agent needs more information or human approval. Design UX and timeouts around it explicitly rather than treating it as an error.

How do A2A push notifications work?
Register a PushNotificationConfig with an HTTPS webhook. The server POSTs on significant updates; the client calls GetTask to retrieve full state and artifacts.

How should you retry failed A2A tasks?
Retry transient failures with idempotency keys, respect timeout budgets, and do not blindly retry terminal states like rejected or policy failures.

What should you log for long-running A2A workflows?
Correlate trace ID, task ID, and agent ID across hops. Log state transitions, artifacts, delegation, approvals, and per-step cost so you can reconstruct the full workflow.

Sources

A2A Protocol -- Streaming and Asynchronous Operations: https://a2a-protocol.org/latest/topics/streaming-and-async/
A2A Protocol -- Specification overview: https://a2a-protocol.org/latest/specification/

Run Docker Compose as a Linux Service with systemd

Rost — Thu, 09 Jul 2026 11:37:13 +0000

Docker Compose on a Linux server should start on boot, stop cleanly on shutdown, and survive reboots without manual intervention.

Docker Compose is not Kubernetes, and that is fine for the workloads this guide targets. For many real systems, a Compose project on a single Linux host is the right amount of infrastructure — simple, readable, easy to back up, and good enough for internal tools, side projects, self-hosted services, staging environments, small production apps, and developer infrastructure.

The missing piece is usually service management. Running this manually is not enough:

docker compose up -d

A single command starts the stack, but it does not document how the stack should start on boot, stop during shutdown, reload after changes, write logs, recover from failures, or get updated safely. That is where systemd helps.

This guide walks through running a Docker Compose project as a Linux service with systemd — unit files, boot ordering, updates, logs, and backups. The split of responsibility is deliberate: Docker runs containers, Compose defines the stack, and systemd starts and stops the project on the host. It is part of Developer Tools - a Guide to Development Workflows.

When Docker Compose as a Service Makes Sense

Running Compose under systemd makes sense when you have:

A single Linux server
A small self-hosted application
A reverse proxy stack
A monitoring stack
A local development platform
An internal tool
A staging environment
A simple production service with known limits

Examples:

Nginx Proxy Manager
Traefik
Gitea
Grafana and Prometheus
PostgreSQL plus a small web app
Uptime Kuma
Home Assistant helper services
Private registry
Internal API plus worker plus Redis

Compose is a good fit when the operational model is still understandable by one person reading one directory.

When Docker Compose Is Not Enough

Use something else when you need:

Multi-node scheduling
Automatic rescheduling across hosts
Cluster-level service discovery
Horizontal autoscaling
Rolling deployments across many machines
Fine-grained workload identity
Complex network policy
Large multi-team platform operations

At that point, Kubernetes, Nomad, Swarm, or a managed platform may be a better fit.

My practical rule is to avoid using Kubernetes just to skip learning systemd, and to avoid using Compose when the workload clearly needs orchestration across multiple hosts.

The Basic Architecture

A clean setup separates project files, the systemd unit, and persistent data on the host. The Compose project lives under /opt/myapp/ with compose.yaml, .env, data/, backups/, and optional scripts such as scripts/update.sh. The systemd unit file sits at /etc/systemd/system/myapp.service.

flowchart TB
  subgraph host["Linux host"]
    systemd["systemd unit\n/etc/systemd/system/myapp.service"]
    compose["Docker Compose\n/opt/myapp/compose.yaml"]
    docker["Docker Engine"]
    fs["Persistent data\n/opt/myapp/data/"]
  end
  systemd -->|"ExecStart: docker compose up -d"| compose
  compose --> docker
  docker --> fs

Each layer has a clear job: Docker runs containers, Compose defines the application stack, systemd starts and stops the Compose project on boot and shutdown, the host filesystem stores persistent data, backups stay explicit, and updates go through scripted, reviewable steps. This layout is deliberately boring, because boring infrastructure is easier to repair when something breaks at 2 a.m.

Prepare the Compose Project Directory

Create a directory under /opt:

sudo mkdir -p /opt/myapp
sudo chown -R "$USER":"$USER" /opt/myapp
cd /opt/myapp

Create a Compose file:

nano compose.yaml

Example:

services:
  web:
    image: nginx:stable
    restart: unless-stopped
    ports:
      - "8080:80"
    volumes:
      - ./html:/usr/share/nginx/html:ro
    healthcheck:
      test: ["CMD-SHELL", "nginx -t || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

volumes: {}

Create the content directory:

mkdir -p html
echo "Hello from Docker Compose" > html/index.html

Test manually first:

docker compose up -d
docker compose ps
docker compose logs --tail=50

Then stop it before handing lifecycle to systemd:

docker compose down

Do not create a systemd service until the Compose project works manually. While you test, keep the Docker Compose Cheatsheet nearby for ps, logs, pull, and project structure.

Use the Modern `docker compose` Command

Docker Engine and the Compose plugin must be installed before you write a unit file. On Ubuntu, Install Docker on Ubuntu walks through APT, Snap, rootless mode, and post-install security so you end up with a working docker compose command.

Use this:

docker compose version

Not this:

docker-compose version

The old docker-compose binary still exists on many machines, but modern Docker uses Compose as a Docker CLI plugin.

In service files and scripts, prefer:

/usr/bin/docker compose

You can find the Docker path with:

command -v docker

Usually it is:

/usr/bin/docker

Create a systemd Service for Docker Compose

If unit files are new to you, Run any Executable as a Service in Linux explains Type, ExecStart, systemctl, and the general systemd workflow. This section applies those patterns specifically to a Compose stack.

Create the service file:

sudo nano /etc/systemd/system/myapp.service

Use this unit:

[Unit]
Description=MyApp Docker Compose stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Reload systemd:

sudo systemctl daemon-reload

Start the service:

sudo systemctl start myapp.service

Enable it on boot:

sudo systemctl enable myapp.service

Check status:

systemctl status myapp.service

Check containers:

cd /opt/myapp
docker compose ps

Why Type=oneshot and RemainAfterExit=yes?

This is the part many guides get subtly wrong.

docker compose up -d starts containers in detached mode and exits, so there is no long-running foreground Compose process for systemd to supervise. The systemd unit should not pretend that docker compose up -d is a long-running daemon.

Use:

Type=oneshot
RemainAfterExit=yes

This tells systemd:

Run the start command.
Consider the unit active after the command exits successfully.
Run ExecStop when the service is stopped.

That matches the actual behavior of detached Compose, which is why Type=oneshot with RemainAfterExit=yes is the right default for most stacks.

Why Not Type=simple?

With Type=simple, systemd expects the ExecStart process to keep running, but docker compose up -d exits after starting containers. That can make systemd think the service ended, then call stop logic or mark the unit inactive depending on configuration.

If you want Type=simple, you would usually run Compose in the foreground:

ExecStart=/usr/bin/docker compose up

That can work, but I usually do not prefer it for Compose stacks on servers. Detached containers plus explicit ExecStop are easier to operate.

A More Production-Friendly Unit

For a real server, I prefer a slightly stricter unit:

[Unit]
Description=MyApp Docker Compose stack
Documentation=https://example.com/docs/myapp
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
EnvironmentFile=-/opt/myapp/.env.systemd
ExecStartPre=/usr/bin/docker compose config --quiet
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecReload=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Important details:

WorkingDirectory points to the Compose project.
ExecStartPre validates the Compose config.
ExecReload recreates changed services.
ExecStop stops and removes the Compose project containers and default network.
EnvironmentFile=-... means the file is optional.

Create the optional systemd environment file:

nano /opt/myapp/.env.systemd

Example:

COMPOSE_PROJECT_NAME=myapp

Then reload systemd:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

Compose .env vs systemd EnvironmentFile

Compose and systemd each have their own environment mechanism, and mixing them up causes confusing "variable not set" failures at boot.

Compose automatically reads a .env file in the project directory for variable substitution in the Compose file.

Example .env:

APP_TAG=1.2.3
WEB_PORT=8080

Example compose.yaml:

services:
  web:
    image: nginx:${APP_TAG}
    ports:
      - "${WEB_PORT}:80"

A systemd EnvironmentFile sets environment variables for the docker compose command itself.

Example:

EnvironmentFile=-/opt/myapp/.env.systemd

For many projects, you only need Compose .env.

Use a systemd environment file when you want to define things such as:

COMPOSE_PROJECT_NAME=myapp
COMPOSE_FILE=compose.yaml
DOCKER_HOST=unix:///var/run/docker.sock

Do not use either file as a casual secrets vault. If secrets matter, use Docker secrets, an external secret manager, encrypted files, or at least strict permissions.

Set restrictive permissions:

chmod 600 /opt/myapp/.env
chmod 600 /opt/myapp/.env.systemd

Restart Policies: Docker vs systemd

There are two restart layers — container restart policy in Compose and systemd service restart policy — and they should not be mixed blindly.

For long-running containers, set restart policies in Compose:

services:
  web:
    image: nginx:stable
    restart: unless-stopped

Common restart values:

Policy	Meaning
no	Do not restart automatically
always	Restart after exit and daemon restart
on-failure	Restart only after failure
unless-stopped	Restart unless manually stopped

For most persistent services, I prefer:

restart: unless-stopped

It is predictable and respects intentional manual stops.

The systemd unit itself should usually not restart repeatedly, because docker compose up -d is not the running workload. The containers are.

So avoid this unless you have a specific reason:

Restart=always

In most Compose-as-service units, let Docker handle container restarts.

Health Checks

Restart policies restart containers when processes exit. They do not magically fix every unhealthy application.

Add health checks where they are useful:

services:
  app:
    image: example/app:latest
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s

Check health:

docker compose ps

Inspect a container:

docker inspect container-name

Health checks are especially useful for:

Web apps
Reverse proxies
Databases
Queues
Internal APIs
Workers with a health endpoint

They are less useful when they only check that a process exists, because a process that is alive but wedged still looks healthy. A bad health check is just another lie in YAML.

Startup Order and depends_on

Compose can define dependencies:

services:
  app:
    image: example/app:latest
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

This can help startup ordering, but do not over-trust it. Applications should still handle retries — databases restart, networks flap, DNS takes time, and a resilient app retries connections instead of assuming perfect startup order.

Logs: journalctl and docker compose logs

Two log views cover most debugging: systemd captures the lifecycle of the unit itself, while Compose captures application output from running containers.

systemd service logs:

journalctl -u myapp.service -n 100 --no-pager

Follow systemd logs:

journalctl -u myapp.service -f

Compose service logs:

cd /opt/myapp
docker compose logs --tail=100
docker compose logs -f
docker compose logs -f web

For most app debugging, docker compose logs is more useful; for lifecycle debugging — start failures, unit crashes, permission errors — journalctl is more useful. If systemctl start myapp fails, check journalctl first. If the stack starts but the app is broken, check docker compose logs.

Log Rotation

Docker logs can grow forever if you do not configure them.

For small servers, configure Docker log rotation in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

Restart Docker:

sudo systemctl restart docker

Then restart the Compose stack:

sudo systemctl restart myapp.service

This applies to newly created containers. Recreate containers if needed:

cd /opt/myapp
docker compose up -d --force-recreate

Log rotation is not glamorous, but it is one of the easiest ways to prevent a disk-full outage on a small server.

Updating a Compose Service

A simple manual update flow:

cd /opt/myapp
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f

If managed by systemd, you can use:

sudo systemctl reload myapp.service

If your unit has:

ExecReload=/usr/bin/docker compose up -d --remove-orphans

But note: ExecReload does not pull images unless you include that step.

For explicit updates, create a script.

mkdir -p /opt/myapp/scripts
nano /opt/myapp/scripts/update.sh

Script:

#!/usr/bin/env bash
set -euo pipefail

cd /opt/myapp

docker compose config --quiet
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f
docker compose ps

Make it executable:

chmod +x /opt/myapp/scripts/update.sh

Run it:

/opt/myapp/scripts/update.sh

Then the service unit can remain focused on lifecycle, while the update script handles deployment.

Safer Update Script with Backup Hook

For stateful services, update only after backup.

#!/usr/bin/env bash
set -euo pipefail

APP_DIR="/opt/myapp"
BACKUP_DIR="/opt/myapp/backups"

cd "$APP_DIR"

mkdir -p "$BACKUP_DIR"

echo "Validating compose file"
docker compose config --quiet

echo "Running backup hook"
if [ -x "$APP_DIR/scripts/backup.sh" ]; then
  "$APP_DIR/scripts/backup.sh"
else
  echo "No backup hook found"
fi

echo "Pulling images"
docker compose pull

echo "Recreating services"
docker compose up -d --remove-orphans

echo "Pruning unused images"
docker image prune -f

echo "Current status"
docker compose ps

This is still simple, but now it encodes an operational habit: backup before change.

Stopping the Service

Stop the stack:

sudo systemctl stop myapp.service

That runs:

docker compose down

By default, docker compose down removes:

Containers for services in the Compose file
Networks defined by the Compose file
The default network

It does not remove named volumes unless you ask it to.

Do not casually use:

docker compose down -v

That removes named volumes declared in the Compose file and anonymous volumes attached to containers. For databases and stateful apps, that can mean deleting real data.

Use down -v only when you mean "destroy this environment".

Restarting the Service

Restart the systemd unit:

sudo systemctl restart myapp.service

This runs the stop command and then the start command.

For only restarting containers without recreating them:

cd /opt/myapp
docker compose restart

Important distinction:

docker compose restart restarts existing containers.
docker compose up -d applies config or image changes by recreating containers when needed.

If you changed compose.yaml, use:

docker compose up -d

Not just:

docker compose restart

Handling Orphan Containers

If you rename or remove a service in compose.yaml, old containers may remain as orphans.

Use:

docker compose up -d --remove-orphans

That is why the systemd service examples in this guide use:

ExecStart=/usr/bin/docker compose up -d --remove-orphans

It keeps the stack closer to the current Compose file.

Backups

Backups depend on the workload, but the principles are stable.

For bind mounts:

/opt/myapp/data/

Back up that directory.

For named volumes:

docker volume ls

Inspect a volume:

docker volume inspect volume-name

For databases, filesystem copies are not always enough. Use application-aware backups:

PostgreSQL example:

docker compose exec -T db pg_dump -U postgres appdb > backups/appdb.sql

MariaDB example:

docker compose exec -T db mariadb-dump -u root -p appdb > backups/appdb.sql

Redis example:

docker compose exec redis redis-cli BGSAVE

A Compose stack without a backup plan is not a service — it is a temporary experiment that happens to have uptime.

Security Baseline

For a small Compose service on Linux, start with this baseline:

Keep the Compose project under /opt/appname.
Use explicit image tags, not only latest, when stability matters.
Use bind mounts or named volumes deliberately.
Do not expose ports you do not need.
Put public services behind a reverse proxy.
Use HTTPS at the edge.
Keep secrets out of Git.
Restrict .env permissions.
Avoid privileged containers unless truly required.
Avoid mounting the Docker socket into containers.
Keep Docker and images updated.
Test firewall behavior from another machine.

A dangerous pattern:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

This gives the container control over Docker. In practice, that can become host-level control. Use it only when you understand the risk.

Resource Limits

On small servers, one bad container can consume the host.

Compose supports resource-related settings, but behavior can depend on Docker Engine and Compose version. For simple protection, start with application-level limits and Docker logging limits.

For some workloads, you can add memory limits:

services:
  app:
    image: example/app:stable
    restart: unless-stopped
    mem_limit: 512m

Also configure app-level worker counts, queue limits, and cache sizes. Container limits are useful, but they are not a substitute for understanding the application.

Example: A Realistic Compose Service

Directory:

/opt/whoami/
  compose.yaml
  .env

Compose file:

services:
  whoami:
    image: traefik/whoami:v1.10
    restart: unless-stopped
    ports:
      - "${WHOAMI_PORT}:80"
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3

.env file:

WHOAMI_PORT=8080
COMPOSE_PROJECT_NAME=whoami

systemd unit:

[Unit]
Description=Whoami Docker Compose stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/whoami
ExecStartPre=/usr/bin/docker compose config --quiet
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecReload=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Install it:

sudo systemctl daemon-reload
sudo systemctl enable --now whoami.service

Test:

curl http://localhost:8080

Check status:

systemctl status whoami.service
cd /opt/whoami
docker compose ps

Troubleshooting

Service Starts but Containers Are Not Running

Check systemd:

journalctl -u myapp.service -n 100 --no-pager

Validate Compose:

cd /opt/myapp
docker compose config

Check Docker:

systemctl status docker
docker info

WorkingDirectory Is Wrong

If systemd cannot find your Compose file, confirm:

WorkingDirectory=/opt/myapp

Then check:

ls -la /opt/myapp
ls -la /opt/myapp/compose.yaml

The service runs from WorkingDirectory, not from your current shell directory.

Docker Permission Denied

If the unit runs as root, it can normally access Docker.

If you set User=someuser, that user must be able to access Docker. Usually that means membership in the docker group, or a rootless Docker setup.

Check:

groups someuser

Add the user if appropriate:

sudo usermod -aG docker someuser

Be careful. The Docker group is effectively privileged.

Compose Command Not Found

Find Docker:

command -v docker

Use the full path in the unit:

ExecStart=/usr/bin/docker compose up -d --remove-orphans

If Compose plugin is missing:

docker compose version

Install it using your Docker package source.

Environment Variables Are Missing

Check the Compose config as systemd would see it:

cd /opt/myapp
docker compose config

If systemd needs extra environment variables, use:

EnvironmentFile=-/opt/myapp/.env.systemd

If Compose needs variables for substitution, use:

/opt/myapp/.env

These are related, but not identical.

Containers Do Not Start After Reboot

Check whether the systemd service is enabled:

systemctl is-enabled myapp.service

Enable it:

sudo systemctl enable myapp.service

Check Docker:

systemctl is-enabled docker
systemctl status docker

Check boot logs:

journalctl -u myapp.service -b --no-pager

App Starts Before Database Is Ready

Add a database health check and depends_on with service_healthy.

Also fix the application. It should retry database connections. Infrastructure startup ordering is helpful, but application retry logic is better.

Disk Filled with Docker Logs

Check Docker disk usage:

docker system df

Check large container logs:

sudo du -h /var/lib/docker/containers | sort -h | tail

Configure Docker log rotation in /etc/docker/daemon.json.

Then recreate containers.

Common Mistakes

Mistake 1: Running docker compose up in rc.local

Running docker compose up from rc.local or a login script works until it does not — use a proper systemd unit instead.

Mistake 2: Using Restart=always in systemd and restart: always in Compose

Usually you only need container restart policies in Compose. Avoid two supervisors fighting each other.

Mistake 3: Forgetting --remove-orphans

Service renames and removals can leave old containers behind. Use:

docker compose up -d --remove-orphans

Mistake 4: Using docker compose restart After Config Changes

restart restarts containers. It does not apply all configuration changes.

Use:

docker compose up -d

Mistake 5: Running down -v Without Thinking

This can delete volumes. For stateful services, that can mean deleting data.

Mistake 6: No Backup Before Pull

New images can break. Databases can migrate. Tags can move. Back up first.

Mistake 7: Publishing Every Port

Only publish what the host needs to expose. Internal service-to-service traffic can stay on the Compose network.

Final Recommended Pattern

For most single-host Linux services, use this pattern:

Compose file:

services:
  app:
    image: example/app:stable
    restart: unless-stopped
    ports:
      - "8080:8080"
    env_file:
      - .env

systemd unit:

[Unit]
Description=MyApp Docker Compose stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
ExecStartPre=/usr/bin/docker compose config --quiet
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecReload=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service

Operate it:

sudo systemctl status myapp.service
sudo systemctl restart myapp.service
journalctl -u myapp.service -f
cd /opt/myapp && docker compose logs -f

This pattern is not fancy, and that is the point. Docker Compose is excellent for small, understandable systems, systemd is excellent at starting and stopping host services, and together they give you a reliable single-server deployment model without pretending every project needs a cluster. For container-level commands outside Compose — images, volumes, networks, and cleanup — see the Docker Cheatsheet.