DEV Community: Rost

A2A and MCP Agent Security: Identity, Delegation, and Audit Trails

Rost — Sat, 11 Jul 2026 06:46:43 +0000

Prompt injection gets most of the security attention in LLM systems, and it deserves attention, but it is not the whole problem once agents start calling tools and delegating work to other agents.

MCP gives an agent structured access to files, APIs, databases, and ticketing systems. A2A lets one agent send tasks, messages, and artifacts to another agent that may belong to a different team, vendor, or runtime. Those protocols are useful precisely because they cross trust boundaries, which means identity, authorization, delegation limits, and audit trails become first-class architecture rather than optional hardening.

This article is the canonical guide for agent protocol security in the LLM Architecture cluster. It covers threat models, identity, gateways, registries, delegation, and production checklists. For input validation, output filtering, and prompt safety patterns, see LLM Guardrails in Practice instead.

Guardrails vs Protocol Security vs Runtime Policy

These three layers solve different problems and fail in different ways when conflated.

LLM guardrails operate on model input and output: blocking injection patterns, filtering harmful content, validating JSON shape, and enforcing tone or compliance rules on generated text. They protect the conversation layer.

Protocol security operates on agent boundaries: who may call which MCP tool, which agent may delegate to which peer, what OAuth scopes attach to a task, and whether a downstream agent may act on a user's behalf. It protects the action layer.

Runtime policy sits between them: a policy engine that evaluates requests against rules regardless of whether the trigger was natural language or a structured protocol call. It can require human approval before a tool executes, block egress to unknown domains, or deny delegation when scope exceeds the originating user.

My opinion is blunt: guardrails without protocol security produce polite chatbots that still exfiltrate data through a tool call. Protocol security without guardrails produces well-authenticated agents that still follow malicious instructions embedded in an artifact. You need both, plus runtime policy for high-risk actions.

Threat Model for A2A and MCP Agent Systems

Start with assets and adversaries, not with a shopping list of controls.

Assets worth protecting: user data in prompts and artifacts, credentials for MCP servers, production systems reachable through tools, agent reputation, billing accounts tied to token usage, and audit integrity.

Realistic adversaries: external users abusing public agent endpoints, compromised MCP servers returning poisoned tool results, malicious agents misrepresenting skills in Agent Cards, insiders over-delegating authority, and supply-chain tampering with tool metadata that manipulates model behavior.

Malicious or compromised tools (MCP)

An MCP server is code plus data exposed to the model. A hostile server can return misleading tool descriptions, exfiltrate arguments passed by the model, or perform actions beyond what the user intended when the host executes tool calls without scoped credentials.

Malicious or impersonated agents (A2A)

An agent that accepts tasks may be evil, compromised, or simply over-permissioned. Agent Cards describe capabilities; they do not prove identity unless you verify signatures, TLS, and issuer trust.

Confused deputy

Agent B holds permission to access a finance API. Agent A, with lower privilege, asks B to "summarize this invoice" while smuggling a transfer instruction in an artifact. B executes using its own credentials unless delegation scope is enforced end to end.

Over-broad permissions and hidden delegation chains

User approves one step. The orchestrator silently chains three A2A hops and five MCP calls. The user never sees the full graph, but the organization is still accountable for the outcome.

Prompt injection through artifacts and cross-agent messages

Injection is not only a user-message problem. A PDF artifact, a web page fetched by a tool, or a message from Agent C can carry instructions aimed at Agent D's model. Treat all protocol-carried content as untrusted input at the model boundary.

Poisoned or misleading Agent Cards

Descriptions and skill names are prompt surface area. A card that advertises safe_read_only_analysis while accepting write-capable backends is a social-engineering layer, not a technical guarantee.

Identity Model for Multi-Agent Systems

Protocol security begins with clear identity types and what each one is allowed to prove.

Identity type	What it represents	Typical proof
Human user	End user or operator who initiated work	OIDC session, SSO token
Agent service	Deployed agent runtime (orchestrator, specialist)	OAuth client credentials, mTLS cert
MCP server	Tool provider process	API key, mTLS, scoped service account
Task / session	Unit of work spanning hops	task ID, trace ID, delegated scope token

A2A's Agent Card advertises supported authentication schemes (OAuth 2.0, API keys, mTLS, and similar patterns aligned with OpenAPI practice) and skills with optional security requirements. The card is discovery metadata, not a trust anchor. Clients obtain credentials out of band and send them in standard HTTP headers on every request; servers must validate on every call and return 401 or 403 when auth or scope fails.

Internal vs external views of the same agent

Production agents often publish a public Agent Card with a limited skill list and a richer authenticated card for internal callers. The A2A specification allows extended cards for authenticated clients. Use that split deliberately: partners should not see internal skills, and internal orchestrators should not rely on public discovery alone for authorization.

Authentication and Authorization for MCP and A2A

Authentication answers who is calling. Authorization answers what they may do.

MCP tool access

For each MCP connection, define:

which agent host may connect
which tools are enabled for that host
which OS user or service account executes side effects
whether the human user must approve each mutating call

Prefer tool allowlists over "connect everything" MCP configs. A coding agent does not need payroll MCP servers on the same profile as a public support bot.

A2A agent access

For each agent peer relationship, define:

which caller agent IDs may invoke which skills
maximum delegation depth
which artifact types may cross the boundary
whether user context must propagate as signed claims

Map OAuth scopes (or equivalent) to skills, not to blanket agent admin. Least privilege at the token layer beats hope at the prompt layer.

Gateway-enforced vs per-agent policy

Per-agent policy works when one team owns the whole graph and releases are coordinated. Gateway-enforced policy works when multiple teams, tenants, or vendors share an agent network and you need one place to enforce allowlists, rate limits, and audit.

flowchart LR
    U[User / client] --> G[A2A gateway]
    G --> O[Orchestrator agent]
    O -->|A2A scoped token| S1[Specialist agent]
    O -->|A2A scoped token| S2[Specialist agent]
    S1 --> MG[MCP gateway]
    S2 --> MG
    MG --> T1[MCP tool servers]
    MG --> T2[MCP tool servers]
    G --> A[Audit log]
    MG --> A
    S1 --> A
    S2 --> A

A2A Gateway as the Control Plane

An A2A gateway is not strictly required by the protocol, but it becomes necessary when agent traffic needs centralized governance.

A gateway typically handles:

authentication termination and token exchange
routing to the correct agent service by skill or tenant
policy checks before tasks are accepted or forwarded
protocol version negotiation
rate limiting and abuse detection
structured audit emission on every task transition

When a gateway is overkill vs necessary

A gateway is often overkill for a single orchestrator and two specialist agents in one Kubernetes namespace maintained by one team. It becomes necessary when partners invoke your agents, when multiple business units share infrastructure, when compliance requires uniform logging, or when you cannot trust every agent implementation to enforce policy correctly.

Pair an A2A gateway with an MCP gateway (or MCP proxy) so tool access receives the same treatment: identity, allowlists, egress controls, and audit at the tool boundary rather than only at the chat UI.

Partner-facing vs internal Agent Cards

Publish different discovery metadata for external and internal callers. External cards expose narrow skills and stricter auth. Internal cards may list maintenance or admin skills but must never be reachable without stronger authentication than the public card implies.

Agent Registry and Discovery Security

Discovery is part of the attack surface. Anyone who controls what agents appear "available" controls where orchestrators send work.

Registry vs well-known Agent Card URLs

Small deployments use well-known URLs per agent (/.well-known/agent-card.json). Enterprise deployments add a registry that indexes agent IDs, versions, endpoints, owners, and policy tags. The registry is a policy object: entries should record which tenants may discover which agents, not only where they live.

Versioning, deprecation, and ownership

Registry records need owners, change history, and deprecation dates. An orchestrator that caches Agent Cards must refresh on TTL and verify signatures where supported. Stale cards are how retired skills keep receiving traffic long after a vulnerability is patched.

Enterprise internal networks vs external partners

Internal agent meshes can rely on mTLS and private DNS. Partner agents need explicit federation rules, contractually scoped skills, and stronger artifact inspection because you do not control their runtime.

Delegation Across Agent Boundaries

Delegation is where A2A security is won or lost. When Agent A sends a task to Agent B, three questions must have crisp answers:

Whose authority is being exercised? The user's, A's service account, or a blended delegated token?
What is B allowed to do with that authority? Read-only analysis, or mutating tools on A's behalf?
Who is accountable if B exceeds scope? A, B, the gateway policy, or the human who approved an unclear prompt?

Propagating user intent vs over-delegation

Pass signed delegation claims that include user ID, original task ID, allowed skills, expiry, and maximum hop count. Downstream agents must reject tasks that expand scope silently. If B needs higher privilege than A held, transition to input_required and obtain explicit human approval rather than upgrading tokens invisibly.

Human-in-the-loop approval flows for risky delegation are covered in A2A Streaming and Async Tasks for Long-Running Agent Workflows where input_required is a first-class task state rather than an error.

sequenceDiagram
    participant User
    participant Orch as Orchestrator agent
    participant GW as A2A gateway
    participant Spec as Specialist agent
    participant MCP as MCP tool server
    User->>Orch: Request with user token
    Orch->>GW: Delegate task (scoped delegation token)
    GW->>GW: Policy check scope + hop count
    GW->>Spec: Forward task (reduced scope token)
    Spec->>MCP: Tool call (tool-scoped credential)
    MCP->>MCP: Enforce allowlist + user context
    Spec-->>GW: Artifact + audit events
    GW-->>Orch: Task update
    Orch-->>User: Final response

Separate reasoning from execution permissions

An agent may need broad read access to plan while write tools sit behind approval. Split credentials or use distinct MCP profiles for planning vs execution so a model mistake cannot immediately mutate production.

Audit Trails and Answer Provenance

If you cannot reconstruct a delegation chain, you cannot explain an incident, pass an audit, or dispute a billing anomaly.

Log at three layers:

Gateway: authentication result, policy decision, routed agent ID, task ID, parent task ID, rate-limit events.

Agent: task state transitions, messages sent/received, model/tool invocations (arguments redacted as needed), artifacts created, delegation outward.

MCP server: tool name, caller agent ID, user context, success/failure, latency, rows affected or resource IDs (policy permitting).

Correlate with trace ID across all layers. Observability for LLM Systems covers instrumentation backends; this article defines what must be captured so those backends have meaningful signal.

Final answer provenance should answer: which user, which orchestrator task, which specialist agents, which tools, which artifacts influenced the text the user saw, and which policy gates fired along the way.

Runtime Policy, Egress, and Secrets

Runtime policy engines (OPA, Cedar, custom rule services) evaluate structured events: "tool X with args Y for user Z." They complement guardrails because they do not depend on the model behaving well.

Human approval belongs in runtime policy for irreversible or high-cost actions: payments, external email, production config changes, privilege grants.

Egress controls limit which domains MCP servers and agents may call. An agent that can both read secrets and POST to arbitrary URLs is a data-loss waiting to happen.

Secrets never belong in Agent Cards or prompts. MCP hosts should inject short-lived credentials at execution time from a secrets manager. For transport encryption, key management, and baseline infra security patterns, see Architectural Patterns for Securing Data.

Push notification webhooks in async A2A flows need the same rigor: verify sender identity, reject stale events, and never treat a webhook payload as authorization on its own.

Reference Security Architecture

The following diagram summarizes a production-oriented layout for A2A outside, MCP inside deployments at scale.

flowchart TB
    subgraph Client layer
        U[User / API client]
    end
    subgraph Control plane
        GW[A2A gateway]
        REG[Agent registry]
        POL[Policy engine]
        AUD[Audit log]
        SEC[Secrets manager]
    end
    subgraph Agent layer
        OR[Orchestrator]
        SA[Specialist agents]
    end
    subgraph Tool layer
        MG[MCP gateway]
        MCP[MCP servers]
    end
    subgraph Observability
        OBS[Tracing + metrics]
    end
    U --> GW
    GW --> REG
    GW --> POL
    GW --> OR
    OR --> GW
    GW --> SA
    SA --> MG
    MG --> MCP
    POL --> GW
    POL --> MG
    SEC --> SA
    SEC --> MCP
    GW --> AUD
    MG --> AUD
    SA --> AUD
    AUD --> OBS

The orchestrator sees specialist agents through A2A. Specialists see tools through MCP. Users never receive raw MCP credentials, and partners never receive internal skill surfaces without policy review.

For protocol concepts (Agent Cards, tasks, artifacts), see What Is the A2A Protocol?. For adoption and enterprise framing, see Google A2A Protocol in 2026. For topology when many agents coordinate, see Multi-Agent Orchestration Patterns.

Production Checklist for A2A and MCP Security

Before exposing agent protocols beyond a trusted sandbox, verify:

Identity and auth

[ ] No anonymous agents in production paths
[ ] Every MCP and A2A call authenticated on every request
[ ] OAuth scopes or equivalent mapped to skills/tools, not blanket admin
[ ] Public vs authenticated Agent Card views defined intentionally

Delegation and policy

[ ] Delegation tokens carry user ID, task ID, scope, expiry, hop limit
[ ] Downstream agents reject scope expansion without explicit approval
[ ] High-risk tools require runtime policy or human approval
[ ] Reasoning and execution use separate credentials where possible

Discovery and registry

[ ] Agent registry entries have owners and version history
[ ] Agent Cards refreshed on TTL; signatures verified where supported
[ ] Partner agents federated with explicit skill allowlists

Audit and observability

[ ] Gateway, agent, and MCP layers emit correlated audit events
[ ] Delegation chains logged with parent and child task IDs
[ ] Artifact provenance recorded for final answers
[ ] Trace IDs connect to observability backends

Abuse and resilience

[ ] Rate limits per user, agent, and tenant
[ ] Timeout policies on delegated tasks
[ ] Egress allowlists on tool servers
[ ] Secrets in a manager, not in cards, prompts, or repos

Conclusion

A2A and MCP interoperability is powerful because agents and tools can compose across team and vendor boundaries, but that power is unsafe without identity, authorization, delegation limits, and audit design. Guardrails protect the model conversation; protocol security protects the actions agents take on behalf of users.

Treat Agent Cards as advertisements, delegation as a signed contract, MCP tools as privileged code execution, and audit logs as the evidence chain you will need when something interesting happens at 2 a.m.

Build the gateway when governance needs a single throat to choke. Split credentials before you split agents. Log every hop so the answer "the model decided" is never the final incident report.

Frequently Asked Questions

What is the difference between LLM guardrails and A2A MCP agent security?
Guardrails constrain model input and output. Protocol security constrains who may invoke tools, delegate tasks, and act on whose behalf across MCP and A2A with identity, authorization, and audit trails.

How should agent identity work in an A2A deployment?
Separate human, agent service, and task identities. Validate credentials on every request, use scoped tokens, and treat Agent Cards as discovery metadata rather than proof of trust.

What is the confused deputy problem in multi-agent systems?
It occurs when a privileged agent or tool performs a sensitive action because a less privileged caller smuggled instructions through delegation or artifacts. Enforce scope at every hop.

Do you need an A2A gateway in production?
Single-team internal deployments may enforce policy per agent. Multi-tenant, multi-vendor, or partner-facing networks usually need a gateway for centralized auth, routing, rate limits, and audit.

What should an A2A MCP audit log contain?
User ID, agent ID, task ID, parent task ID, tool calls, policy decisions, artifacts, and timestamps correlated with trace IDs across gateway, agent, and MCP layers.

Sources

A2A Protocol -- Enterprise-ready security topics: https://github.com/a2aproject/A2A/blob/main/docs/topics/enterprise-ready.md
A2A Protocol -- Specification overview: https://a2a-protocol.org/latest/specification/
A2A Protocol -- Streaming and push notification security: https://a2a-protocol.org/latest/topics/streaming-and-async/

A2A Streaming and Async Tasks for Long-Running Agent Workflows

Rost — Fri, 10 Jul 2026 09:36:29 +0000

Most AI agent demos still behave like chat completions with extra steps: you send a prompt, wait a few seconds, and get an answer back in one response.

Real agent work often does not fit that pattern. Research, code review, procurement analysis, incident investigation, and multi-step planning can run for minutes or hours, and they may need clarification halfway through, stream partial results, delegate to another agent, and produce files rather than a single text reply. That is where the A2A protocol's async model matters within the broader AI Systems cluster, because A2A treats long-running work as a Task with a lifecycle instead of a one-shot HTTP response. Clients can stay connected via Server-Sent Events (SSE), poll task state, or register push webhooks when they cannot hold a connection open.

This article covers operational design for those workflows, including when to stream versus poll versus push, how input_required fits human-in-the-loop flows, failure handling, and what to instrument in production. For Agent Cards, messages, parts, and the full task model, see What Is the A2A Protocol? Agent Cards and Tasks Explained.

Why Long-Running A2A Agent Tasks Need Async Design

A synchronous request/response mental model breaks down quickly once agent work spans tools, delegation, approvals, and large artifacts. An agent task may call multiple MCP servers internally, delegate sub-work to another agent over A2A, wait for human approval, generate large artifacts in chunks, fail partway through and need partial recovery, and accumulate token cost across several hops. HTTP APIs can approximate this with timeouts, background jobs, and ad hoc status endpoints, but A2A bakes task identity and state into the protocol so clients and gateways can reason about work consistently. For how those layers fit inside a production assistant before you add async A2A boundaries, see AI Assistant Architecture: LLM, Memory, Tools, Routing, Observability.

My bias is practical: do not create a Task for everything, because a one-line summary does not need a lifecycle. Use a Task when work is stateful, auditable, long-running, artifact-producing, or may need input mid-flight. The rule of thumb from the explainer still holds: simple interactions can return a Message, while complex work should return a Task.

A2A Task Lifecycle and State Transitions

An A2A Task moves through states that clients can query at any time. Exact naming varies slightly by implementation, but the model is stable across servers that follow the protocol.

stateDiagram-v2
    [*] --> submitted
    submitted --> working
    working --> input_required
    input_required --> working
    working --> completed
    working --> failed
    working --> canceled
    working --> rejected
    submitted --> rejected
    input_required --> failed
    input_required --> canceled
    completed --> [*]
    failed --> [*]
    canceled --> [*]
    rejected --> [*]

The submitted state means the client sent work and the agent accepted or queued it. In working, the agent is actively processing, which may include tool calls, delegation, or streaming partial output. The input_required state indicates the agent paused because it needs more input, clarification, or human approval, and it is not a failure state. completed is terminal success with artifacts available; failed is a terminal error whose details and partial artifacts depend on implementation; canceled means a client, gateway, or authorized caller stopped the task; and rejected means the agent refused the task because of policy, capability mismatch, or auth.

When input_required pauses versus fails a workflow

Treat input_required as a deliberate pause, not an exception. The agent is telling you it cannot proceed without something from you, whether that is a missing parameter, a policy confirmation, or a manager sign-off on a high-risk action. A workflow fails when the task reaches failed or rejected, or when a caller exceeds a timeout waiting for input that never arrives, so you should design explicit timeouts for human steps rather than letting approvals sit indefinitely.

An approval that waits three days without escalation is a stuck workflow, not a patient one, and stuck workflows clog task stores while making observability dashboards harder to read.

Who can cancel an A2A task

Cancellation authority should be defined at design time rather than debated during an incident. The client usually can cancel tasks it created; a gateway may cancel on behalf of tenants, policy violations, or budget limits; and an upstream agent may cancel delegated work when orchestrating over A2A if the protocol and policy allow it. Log who canceled and why, because in multi-agent chains orphan work is a common source of surprise token bills.

Human-in-the-Loop with input_required Task States

input_required is one of A2A's most underused design features, and many teams treat it as an error code when it is actually a first-class workflow state. In production you will hit cases where the agent should stop, such as spending budget on an ambiguous request, executing an irreversible action, accessing sensitive data without scope confirmation, or delegating to a specialist that needs explicit user intent. Model these as deliberate transitions to input_required, with a clear message explaining what is needed.

Approval flows for risky A2A delegation

When Agent A delegates to Agent B over A2A and Agent B enters input_required for human approval, three systems need to agree on what happens next. The downstream agent pauses and exposes what it needs, the orchestrator or gateway surfaces that pause to the user, and the user's response resumes the task via a new message. The A2A vs MCP comparison explains why delegation across agent boundaries is a different problem from tool access, and why approval semantics belong at the task layer rather than inside a single MCP call. Do not silently auto-approve because the UX is inconvenient, since expensive mistakes usually come from convenience shortcuts rather than from missing models.

UX patterns for paused A2A tasks

Blocking wait means the UI shows a spinner or approval card until the task leaves input_required, which works well for short human steps. Non-blocking wait means the client records the task ID, lets the user continue elsewhere, and uses polling or push to notify when input is needed again, which is required for mobile, email-linked approvals, or multi-tab assistants. Timeout when humans are slow means defining an SLA per step and, after N hours, transitioning to failed or escalating to another queue, because unbounded waits clog task stores and confuse observability dashboards.

How an A2A gateway handles input_required

If you run an A2A gateway, decide whether it forwards input_required events transparently, aggregates pauses from multiple downstream agents into one user prompt, or enforces that certain skills always require approval before leaving input_required. Auth and policy for approved actions belong in a dedicated security article; for now, assume every resumed task should carry the same user identity and scope as the original request.

Choosing Sync, SSE Streaming, Polling, or Push Notifications

A2A supports multiple interaction modes, and the right choice depends on client capabilities and latency needs rather than on which mode sounds most modern.

Mode	Best for	Client requirements	Tradeoffs
Sync (SendMessage, short Task)	Quick work, immediate Messages	Simple HTTP client	Timeouts on slow agents
SSE streaming	Live progress, incremental artifacts	Long-lived connection	Proxies, mobile background limits
Polling (GetTask)	Batch clients, simple integrations	Timer + task ID	Higher latency, more requests
Push webhooks	Mobile, serverless, multi-hour jobs	HTTPS receiver + verification	Async complexity, security hardening

Read Agent Card capability flags first

Before choosing a mode, read the agent's Agent Card, because streaming requires capabilities.streaming: true and push notification support is advertised separately. Clients that assume every agent streams will break against minimal implementations, so negotiation is not ceremonial: it prevents runtime failures when a specialist agent only supports poll-based status checks.

When to use assistant-side polling around A2A

Your assistant runtime may wrap A2A task polling in a scheduler loop rather than exposing raw protocol details to the user. That pattern overlaps with general polling agents, which are background processes that wake up, check state, and act. For durable scheduling, idempotency, and queue patterns outside A2A specifically, see Polling Agents in AI Assistants: 11 Implementation Patterns. Use assistant polling when you orchestrate many A2A tasks from a single control plane, and use native A2A streaming or push when the client connects directly to the agent boundary.

A2A Server-Sent Events (SSE) Streaming

SSE is A2A's primary real-time channel. The client calls SendStreamingMessage, opens an HTTP connection, and receives a text/event-stream response until the task reaches a terminal or interrupted state. Each event's payload is JSON-RPC-shaped, and typical result types include a Task snapshot, a TaskStatusUpdateEvent for lifecycle transitions and intermediate agent messages, and a TaskArtifactUpdateEvent for chunked artifact delivery with append and lastChunk hints for reassembly.

sequenceDiagram
    participant Client
    participant A2A Server
    Client->>A2A Server: SendStreamingMessage
    A2A Server-->>Client: HTTP 200 text/event-stream
    loop Until terminal or input_required
        A2A Server-->>Client: TaskStatusUpdateEvent
        A2A Server-->>Client: TaskArtifactUpdateEvent (optional)
    end
    A2A Server-->>Client: Close stream
    Note over Client,A2A Server: On disconnect before terminal state,<br/>client may call SubscribeToTask

Streaming progress updates and partial artifacts

Streaming shines when users should see work happening, whether that means step counters ("3 of 7 sources reviewed"), partial text generation, incremental file chunks for large reports, or state transitions from working to input_required without polling. Design UI around event types rather than around a single final blob, because if you only display output when completed arrives you might as well poll.

SSE connection drops and resubscription

Networks drop, laptops sleep, and load balancers idle-timeout SSE connections, so long streams need recovery logic rather than optimistic assumptions. A2A provides SubscribeToTask so clients can reconnect to an in-progress task stream, and your client SDK should persist taskId locally, detect stream closure before terminal state, resubscribe with backoff, and de-duplicate events if the server replays overlapping state. Without resubscription logic, long tasks feel fragile in production even when the agent backend is healthy.

A2A Push Notifications and Webhooks

Push fits scenarios where SSE is a poor match, such as mobile apps in the background, serverless handlers, or tasks that run for hours or days. The client supplies a PushNotificationConfig with a url (HTTPS webhook on the client side), an optional token for validating incoming POSTs, and optional authentication details for how the A2A server authenticates to the webhook. Configuration can ride along with the initial SendMessage or SendStreamingMessage call, or be added later via CreateTaskPushNotificationConfig for an existing task.

sequenceDiagram
    participant Client
    participant A2A Server
    participant Webhook
    Client->>A2A Server: SendMessage + PushNotificationConfig
    A2A Server-->>Client: taskId
    Note over A2A Server: Task runs asynchronously
    A2A Server->>Webhook: POST state change notification
    Webhook->>A2A Server: GetTask(taskId)
    A2A Server-->>Webhook: Updated Task + artifacts
    Webhook->>Client: Resume workflow / notify user

When a significant update occurs, the A2A server POSTs to the webhook and the client typically calls GetTask with the notified taskId to fetch the full updated Task and artifacts. Push is a signal, not a full payload transport.

When push beats an open SSE connection

Prefer push when the client cannot maintain SSE (mobile, edge functions), when updates are infrequent and milestone-based rather than token-by-token, or when you want the server to wake a disconnected workflow engine. Prefer SSE when users watch progress live, when artifacts stream in many small chunks, or when latency below a few seconds matters.

Correlating push notifications to A2A tasks

Every push handler should log and propagate the taskId, a trace or correlation ID from the original request, the event type or state transition, and a timestamp from the notification so stale events can be rejected. Replay attacks and duplicate deliveries happen in production, so idempotent handlers are not optional.

Push endpoint security overview

Push introduces SSRF risk on the server when malicious clients register internal URLs, and impersonation risk on the client when fake POSTs arrive at the webhook. Mitigations include URL allowlists, ownership verification, signed JWTs with JWKS, timestamp checks, and validating the config token. The full threat model, identity layers, and gateway controls live in A2A and MCP Agent Security: Identity, Delegation, and Audit Trails; until you have read it, treat webhook verification with the same seriousness as payment callbacks.

Async A2A Workflow Patterns

Fire-and-follow task submission

The client submits a task, receives a task ID immediately, and disconnects, then later polls GetTask or waits for push. This is the default pattern for serverless and batch pipelines, but you should persist the task ID in durable storage before acknowledging the user, because serverless invocations that forget the ID lose the work.

Resuming a task after input_required

After input_required, the user sends a new message against the same task and the agent transitions back to working. Design messages so resumption context is explicit, because "Approved: proceed with vendor X" beats a bare "yes" when you need to audit what was approved six hours later.

Chained A2A delegation with intermediate artifacts

Consider a research workflow where an orchestrator owns Task T1 and delegates retrieval, summarization, and verification to specialist agents, each with its own task ID and artifacts along the way.

flowchart TD
    U[User] --> O[Orchestrator Task T1]
    O -->|A2A| R[Retrieval agent T2]
    R --> A2[artifact: raw sources]
    O -->|A2A| S[Summarization agent T3]
    S --> A3[artifact: draft summary]
    O -->|A2A| V[Verification agent T4]
    V --> A4[artifact: fact-check report]
    O --> F[final artifact: recommendation memo]

Each hop has its own task ID and state machine, so the orchestrator should stream or poll downstream tasks independently, persist intermediate artifacts before starting the next hop, and fail gracefully if T3 completes but T4 rejects the draft. Multi-Agent Orchestration Patterns covers topology choice when those specialists run as separate services rather than in one runtime. Partial progress is valuable, and a failed verification should not delete a usable draft without a clear reason.

Durable task storage for delayed completion

Task state and artifacts should survive process restarts. If your agent runs in Kubernetes, assume pods die mid-task and back task records and artifact blobs to a store the agent container does not own exclusively.

Failure Handling for Long-Running A2A Workflows

Long-running workflows fail in predictable ways through timeouts, retries, partial artifacts, and unsafe cancellation, and each needs an explicit policy rather than ad hoc handling in client code.

Per-hop and end-to-end timeout budgets

Set timeouts at two levels: a per-hop maximum for one agent task before escalation or cancel, and an end-to-end maximum for the user-visible workflow. A retrieval agent that hangs should not block the entire orchestrator until the user's browser times out.

Retries and idempotency for A2A tasks

Retries without idempotency duplicate side effects such as double charges, duplicate tickets, and repeated emails. Use stable client message IDs or idempotency keys where the protocol allows, and for business mutations align with Idempotency in Distributed Systems That Actually Works. Retry only transient failures like network blips or 503s, and do not retry rejected or policy failures blindly because you will amplify cost and annoy downstream agents.

Partial artifact recovery policies

When a task fails after producing partial artifacts, define whether you expose partial output to the user with a clear "incomplete" label, allow resume from the last good checkpoint, or discard partial output when it could mislead in medical, legal, or financial contexts.

Safe cancellation across delegation chains

Cancel downstream tasks when an upstream user aborts, use a delegation graph so cancel propagates, and log canceled tasks that already incurred cost because finance teams notice.

Observability for Async A2A Workflows

You cannot debug multi-agent async work unless you can trace it across boundaries, which means correlating identifiers on every hop rather than relying on unstructured logs. Minimum correlation fields include a trace ID per user-initiated workflow, a task ID per agent task including delegated children, an agent ID for the Agent Card or service that handled the hop, and a parent task ID that links delegation chains.

Log every state transition with timestamps, and log artifact creation events with size and hash rather than necessarily full content when PII policies apply. Attribute cost and latency per hop, because multi-agent workflows hide token spend until the bill arrives and per-task cost labels make "which specialist is expensive?" answerable. For metrics, tracing backends, and LLM-specific instrumentation patterns, see Observability for LLM Systems and the broader Observability pillar for how those signals fit into a production telemetry stack.
When a user asks "why did the agent do that?", your answer should be a trace spanning orchestrator, A2A hops, MCP tool calls, and any input_required pauses rather than a shrug and a log grep.

Production Checklist for A2A Streaming and Async Tasks

Before shipping long-running A2A paths to production, verify the following areas.

Agent Card and capabilities

[ ] capabilities.streaming reflects actual SSE support
[ ] Push notification support documented if implemented
[ ] Skills that require human approval document expected input_required behavior

Client modes

[ ] SSE client handles resubscription via SubscribeToTask
[ ] Poll interval backs off under load
[ ] Push webhook verifies authenticity and rejects stale events

Durability

[ ] Task state survives agent process restarts
[ ] Artifacts stored outside ephemeral container filesystem
[ ] Intermediate artifacts available for partial recovery

Failure and policy

[ ] Per-hop and end-to-end timeout budgets defined
[ ] Retries idempotent for mutating operations
[ ] Cancel propagates across delegation edges

Observability

[ ] trace ID + task ID + agent ID on every hop
[ ] State transitions logged
[ ] Cost attribution per task or per agent

Load testing

[ ] SSE through your reverse proxy (buffering breaks streams)
[ ] Concurrent long tasks without memory leaks on open connections
[ ] Push flood handling without webhook overload

Conclusion

A2A's value shows up most clearly when work does not fit a single synchronous API call, because streaming, async tasks, push notifications, and explicit task states are how the protocol handles real agent workloads such as research, delegation, approvals, and large artifacts without pretending everything completes in one HTTP round trip. Start with the simplest mode that works, add SSE when users need live progress, add push when connections cannot stay open, treat input_required as a first-class design tool rather than a failure, and instrument every hop so multi-agent async workflows do not outrun your ability to explain them.

Frequently Asked Questions

When should you use A2A streaming instead of polling?
Use streaming when the client can hold an open HTTP connection and you need low-latency progress updates or incremental artifacts. Use polling when connections are unreliable, clients are batch-oriented, or you only need periodic status checks on long-running tasks.

What does input_required mean in an A2A task?
It is a pause state where the agent needs more information or human approval. Design UX and timeouts around it explicitly rather than treating it as an error.

How do A2A push notifications work?
Register a PushNotificationConfig with an HTTPS webhook. The server POSTs on significant updates; the client calls GetTask to retrieve full state and artifacts.

How should you retry failed A2A tasks?
Retry transient failures with idempotency keys, respect timeout budgets, and do not blindly retry terminal states like rejected or policy failures.

What should you log for long-running A2A workflows?
Correlate trace ID, task ID, and agent ID across hops. Log state transitions, artifacts, delegation, approvals, and per-step cost so you can reconstruct the full workflow.

Sources

A2A Protocol -- Streaming and Asynchronous Operations: https://a2a-protocol.org/latest/topics/streaming-and-async/
A2A Protocol -- Specification overview: https://a2a-protocol.org/latest/specification/

Run Docker Compose as a Linux Service with systemd

Rost — Thu, 09 Jul 2026 11:37:13 +0000

Docker Compose on a Linux server should start on boot, stop cleanly on shutdown, and survive reboots without manual intervention.

Docker Compose is not Kubernetes, and that is fine for the workloads this guide targets. For many real systems, a Compose project on a single Linux host is the right amount of infrastructure — simple, readable, easy to back up, and good enough for internal tools, side projects, self-hosted services, staging environments, small production apps, and developer infrastructure.

The missing piece is usually service management. Running this manually is not enough:

docker compose up -d

A single command starts the stack, but it does not document how the stack should start on boot, stop during shutdown, reload after changes, write logs, recover from failures, or get updated safely. That is where systemd helps.

This guide walks through running a Docker Compose project as a Linux service with systemd — unit files, boot ordering, updates, logs, and backups. The split of responsibility is deliberate: Docker runs containers, Compose defines the stack, and systemd starts and stops the project on the host. It is part of Developer Tools - a Guide to Development Workflows.

When Docker Compose as a Service Makes Sense

Running Compose under systemd makes sense when you have:

A single Linux server
A small self-hosted application
A reverse proxy stack
A monitoring stack
A local development platform
An internal tool
A staging environment
A simple production service with known limits

Examples:

Nginx Proxy Manager
Traefik
Gitea
Grafana and Prometheus
PostgreSQL plus a small web app
Uptime Kuma
Home Assistant helper services
Private registry
Internal API plus worker plus Redis

Compose is a good fit when the operational model is still understandable by one person reading one directory.

When Docker Compose Is Not Enough

Use something else when you need:

Multi-node scheduling
Automatic rescheduling across hosts
Cluster-level service discovery
Horizontal autoscaling
Rolling deployments across many machines
Fine-grained workload identity
Complex network policy
Large multi-team platform operations

At that point, Kubernetes, Nomad, Swarm, or a managed platform may be a better fit.

My practical rule is to avoid using Kubernetes just to skip learning systemd, and to avoid using Compose when the workload clearly needs orchestration across multiple hosts.

The Basic Architecture

A clean setup separates project files, the systemd unit, and persistent data on the host. The Compose project lives under /opt/myapp/ with compose.yaml, .env, data/, backups/, and optional scripts such as scripts/update.sh. The systemd unit file sits at /etc/systemd/system/myapp.service.

flowchart TB
  subgraph host["Linux host"]
    systemd["systemd unit\n/etc/systemd/system/myapp.service"]
    compose["Docker Compose\n/opt/myapp/compose.yaml"]
    docker["Docker Engine"]
    fs["Persistent data\n/opt/myapp/data/"]
  end
  systemd -->|"ExecStart: docker compose up -d"| compose
  compose --> docker
  docker --> fs

Each layer has a clear job: Docker runs containers, Compose defines the application stack, systemd starts and stops the Compose project on boot and shutdown, the host filesystem stores persistent data, backups stay explicit, and updates go through scripted, reviewable steps. This layout is deliberately boring, because boring infrastructure is easier to repair when something breaks at 2 a.m.

Prepare the Compose Project Directory

Create a directory under /opt:

sudo mkdir -p /opt/myapp
sudo chown -R "$USER":"$USER" /opt/myapp
cd /opt/myapp

Create a Compose file:

nano compose.yaml

Example:

services:
  web:
    image: nginx:stable
    restart: unless-stopped
    ports:
      - "8080:80"
    volumes:
      - ./html:/usr/share/nginx/html:ro
    healthcheck:
      test: ["CMD-SHELL", "nginx -t || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

volumes: {}

Create the content directory:

mkdir -p html
echo "Hello from Docker Compose" > html/index.html

Test manually first:

docker compose up -d
docker compose ps
docker compose logs --tail=50

Then stop it before handing lifecycle to systemd:

docker compose down

Do not create a systemd service until the Compose project works manually. While you test, keep the Docker Compose Cheatsheet nearby for ps, logs, pull, and project structure.

Use the Modern `docker compose` Command

Docker Engine and the Compose plugin must be installed before you write a unit file. On Ubuntu, Install Docker on Ubuntu walks through APT, Snap, rootless mode, and post-install security so you end up with a working docker compose command.

Use this:

docker compose version

Not this:

docker-compose version

The old docker-compose binary still exists on many machines, but modern Docker uses Compose as a Docker CLI plugin.

In service files and scripts, prefer:

/usr/bin/docker compose

You can find the Docker path with:

command -v docker

Usually it is:

/usr/bin/docker

Create a systemd Service for Docker Compose

If unit files are new to you, Run any Executable as a Service in Linux explains Type, ExecStart, systemctl, and the general systemd workflow. This section applies those patterns specifically to a Compose stack.

Create the service file:

sudo nano /etc/systemd/system/myapp.service

Use this unit:

[Unit]
Description=MyApp Docker Compose stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Reload systemd:

sudo systemctl daemon-reload

Start the service:

sudo systemctl start myapp.service

Enable it on boot:

sudo systemctl enable myapp.service

Check status:

systemctl status myapp.service

Check containers:

cd /opt/myapp
docker compose ps

Why Type=oneshot and RemainAfterExit=yes?

This is the part many guides get subtly wrong.

docker compose up -d starts containers in detached mode and exits, so there is no long-running foreground Compose process for systemd to supervise. The systemd unit should not pretend that docker compose up -d is a long-running daemon.

Use:

Type=oneshot
RemainAfterExit=yes

This tells systemd:

Run the start command.
Consider the unit active after the command exits successfully.
Run ExecStop when the service is stopped.

That matches the actual behavior of detached Compose, which is why Type=oneshot with RemainAfterExit=yes is the right default for most stacks.

Why Not Type=simple?

With Type=simple, systemd expects the ExecStart process to keep running, but docker compose up -d exits after starting containers. That can make systemd think the service ended, then call stop logic or mark the unit inactive depending on configuration.

If you want Type=simple, you would usually run Compose in the foreground:

ExecStart=/usr/bin/docker compose up

That can work, but I usually do not prefer it for Compose stacks on servers. Detached containers plus explicit ExecStop are easier to operate.

A More Production-Friendly Unit

For a real server, I prefer a slightly stricter unit:

[Unit]
Description=MyApp Docker Compose stack
Documentation=https://example.com/docs/myapp
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
EnvironmentFile=-/opt/myapp/.env.systemd
ExecStartPre=/usr/bin/docker compose config --quiet
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecReload=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Important details:

WorkingDirectory points to the Compose project.
ExecStartPre validates the Compose config.
ExecReload recreates changed services.
ExecStop stops and removes the Compose project containers and default network.
EnvironmentFile=-... means the file is optional.

Create the optional systemd environment file:

nano /opt/myapp/.env.systemd

Example:

COMPOSE_PROJECT_NAME=myapp

Then reload systemd:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

Compose .env vs systemd EnvironmentFile

Compose and systemd each have their own environment mechanism, and mixing them up causes confusing "variable not set" failures at boot.

Compose automatically reads a .env file in the project directory for variable substitution in the Compose file.

Example .env:

APP_TAG=1.2.3
WEB_PORT=8080

Example compose.yaml:

services:
  web:
    image: nginx:${APP_TAG}
    ports:
      - "${WEB_PORT}:80"

A systemd EnvironmentFile sets environment variables for the docker compose command itself.

Example:

EnvironmentFile=-/opt/myapp/.env.systemd

For many projects, you only need Compose .env.

Use a systemd environment file when you want to define things such as:

COMPOSE_PROJECT_NAME=myapp
COMPOSE_FILE=compose.yaml
DOCKER_HOST=unix:///var/run/docker.sock

Do not use either file as a casual secrets vault. If secrets matter, use Docker secrets, an external secret manager, encrypted files, or at least strict permissions.

Set restrictive permissions:

chmod 600 /opt/myapp/.env
chmod 600 /opt/myapp/.env.systemd

Restart Policies: Docker vs systemd

There are two restart layers — container restart policy in Compose and systemd service restart policy — and they should not be mixed blindly.

For long-running containers, set restart policies in Compose:

services:
  web:
    image: nginx:stable
    restart: unless-stopped

Common restart values:

Policy	Meaning
no	Do not restart automatically
always	Restart after exit and daemon restart
on-failure	Restart only after failure
unless-stopped	Restart unless manually stopped

For most persistent services, I prefer:

restart: unless-stopped

It is predictable and respects intentional manual stops.

The systemd unit itself should usually not restart repeatedly, because docker compose up -d is not the running workload. The containers are.

So avoid this unless you have a specific reason:

Restart=always

In most Compose-as-service units, let Docker handle container restarts.

Health Checks

Restart policies restart containers when processes exit. They do not magically fix every unhealthy application.

Add health checks where they are useful:

services:
  app:
    image: example/app:latest
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s

Check health:

docker compose ps

Inspect a container:

docker inspect container-name

Health checks are especially useful for:

Web apps
Reverse proxies
Databases
Queues
Internal APIs
Workers with a health endpoint

They are less useful when they only check that a process exists, because a process that is alive but wedged still looks healthy. A bad health check is just another lie in YAML.

Startup Order and depends_on

Compose can define dependencies:

services:
  app:
    image: example/app:latest
    depends_on:
      db:
        condition: service_healthy

  db:
    image: postgres:16
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

This can help startup ordering, but do not over-trust it. Applications should still handle retries — databases restart, networks flap, DNS takes time, and a resilient app retries connections instead of assuming perfect startup order.

Logs: journalctl and docker compose logs

Two log views cover most debugging: systemd captures the lifecycle of the unit itself, while Compose captures application output from running containers.

systemd service logs:

journalctl -u myapp.service -n 100 --no-pager

Follow systemd logs:

journalctl -u myapp.service -f

Compose service logs:

cd /opt/myapp
docker compose logs --tail=100
docker compose logs -f
docker compose logs -f web

For most app debugging, docker compose logs is more useful; for lifecycle debugging — start failures, unit crashes, permission errors — journalctl is more useful. If systemctl start myapp fails, check journalctl first. If the stack starts but the app is broken, check docker compose logs.

Log Rotation

Docker logs can grow forever if you do not configure them.

For small servers, configure Docker log rotation in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  }
}

Restart Docker:

sudo systemctl restart docker

Then restart the Compose stack:

sudo systemctl restart myapp.service

This applies to newly created containers. Recreate containers if needed:

cd /opt/myapp
docker compose up -d --force-recreate

Log rotation is not glamorous, but it is one of the easiest ways to prevent a disk-full outage on a small server.

Updating a Compose Service

A simple manual update flow:

cd /opt/myapp
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f

If managed by systemd, you can use:

sudo systemctl reload myapp.service

If your unit has:

ExecReload=/usr/bin/docker compose up -d --remove-orphans

But note: ExecReload does not pull images unless you include that step.

For explicit updates, create a script.

mkdir -p /opt/myapp/scripts
nano /opt/myapp/scripts/update.sh

Script:

#!/usr/bin/env bash
set -euo pipefail

cd /opt/myapp

docker compose config --quiet
docker compose pull
docker compose up -d --remove-orphans
docker image prune -f
docker compose ps

Make it executable:

chmod +x /opt/myapp/scripts/update.sh

Run it:

/opt/myapp/scripts/update.sh

Then the service unit can remain focused on lifecycle, while the update script handles deployment.

Safer Update Script with Backup Hook

For stateful services, update only after backup.

#!/usr/bin/env bash
set -euo pipefail

APP_DIR="/opt/myapp"
BACKUP_DIR="/opt/myapp/backups"

cd "$APP_DIR"

mkdir -p "$BACKUP_DIR"

echo "Validating compose file"
docker compose config --quiet

echo "Running backup hook"
if [ -x "$APP_DIR/scripts/backup.sh" ]; then
  "$APP_DIR/scripts/backup.sh"
else
  echo "No backup hook found"
fi

echo "Pulling images"
docker compose pull

echo "Recreating services"
docker compose up -d --remove-orphans

echo "Pruning unused images"
docker image prune -f

echo "Current status"
docker compose ps

This is still simple, but now it encodes an operational habit: backup before change.

Stopping the Service

Stop the stack:

sudo systemctl stop myapp.service

That runs:

docker compose down

By default, docker compose down removes:

Containers for services in the Compose file
Networks defined by the Compose file
The default network

It does not remove named volumes unless you ask it to.

Do not casually use:

docker compose down -v

That removes named volumes declared in the Compose file and anonymous volumes attached to containers. For databases and stateful apps, that can mean deleting real data.

Use down -v only when you mean "destroy this environment".

Restarting the Service

Restart the systemd unit:

sudo systemctl restart myapp.service

This runs the stop command and then the start command.

For only restarting containers without recreating them:

cd /opt/myapp
docker compose restart

Important distinction:

docker compose restart restarts existing containers.
docker compose up -d applies config or image changes by recreating containers when needed.

If you changed compose.yaml, use:

docker compose up -d

Not just:

docker compose restart

Handling Orphan Containers

If you rename or remove a service in compose.yaml, old containers may remain as orphans.

Use:

docker compose up -d --remove-orphans

That is why the systemd service examples in this guide use:

ExecStart=/usr/bin/docker compose up -d --remove-orphans

It keeps the stack closer to the current Compose file.

Backups

Backups depend on the workload, but the principles are stable.

For bind mounts:

/opt/myapp/data/

Back up that directory.

For named volumes:

docker volume ls

Inspect a volume:

docker volume inspect volume-name

For databases, filesystem copies are not always enough. Use application-aware backups:

PostgreSQL example:

docker compose exec -T db pg_dump -U postgres appdb > backups/appdb.sql

MariaDB example:

docker compose exec -T db mariadb-dump -u root -p appdb > backups/appdb.sql

Redis example:

docker compose exec redis redis-cli BGSAVE

A Compose stack without a backup plan is not a service — it is a temporary experiment that happens to have uptime.

Security Baseline

For a small Compose service on Linux, start with this baseline:

Keep the Compose project under /opt/appname.
Use explicit image tags, not only latest, when stability matters.
Use bind mounts or named volumes deliberately.
Do not expose ports you do not need.
Put public services behind a reverse proxy.
Use HTTPS at the edge.
Keep secrets out of Git.
Restrict .env permissions.
Avoid privileged containers unless truly required.
Avoid mounting the Docker socket into containers.
Keep Docker and images updated.
Test firewall behavior from another machine.

A dangerous pattern:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

This gives the container control over Docker. In practice, that can become host-level control. Use it only when you understand the risk.

Resource Limits

On small servers, one bad container can consume the host.

Compose supports resource-related settings, but behavior can depend on Docker Engine and Compose version. For simple protection, start with application-level limits and Docker logging limits.

For some workloads, you can add memory limits:

services:
  app:
    image: example/app:stable
    restart: unless-stopped
    mem_limit: 512m

Also configure app-level worker counts, queue limits, and cache sizes. Container limits are useful, but they are not a substitute for understanding the application.

Example: A Realistic Compose Service

Directory:

/opt/whoami/
  compose.yaml
  .env

Compose file:

services:
  whoami:
    image: traefik/whoami:v1.10
    restart: unless-stopped
    ports:
      - "${WHOAMI_PORT}:80"
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3

.env file:

WHOAMI_PORT=8080
COMPOSE_PROJECT_NAME=whoami

systemd unit:

[Unit]
Description=Whoami Docker Compose stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/whoami
ExecStartPre=/usr/bin/docker compose config --quiet
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecReload=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Install it:

sudo systemctl daemon-reload
sudo systemctl enable --now whoami.service

Test:

curl http://localhost:8080

Check status:

systemctl status whoami.service
cd /opt/whoami
docker compose ps

Troubleshooting

Service Starts but Containers Are Not Running

Check systemd:

journalctl -u myapp.service -n 100 --no-pager

Validate Compose:

cd /opt/myapp
docker compose config

Check Docker:

systemctl status docker
docker info

WorkingDirectory Is Wrong

If systemd cannot find your Compose file, confirm:

WorkingDirectory=/opt/myapp

Then check:

ls -la /opt/myapp
ls -la /opt/myapp/compose.yaml

The service runs from WorkingDirectory, not from your current shell directory.

Docker Permission Denied

If the unit runs as root, it can normally access Docker.

If you set User=someuser, that user must be able to access Docker. Usually that means membership in the docker group, or a rootless Docker setup.

Check:

groups someuser

Add the user if appropriate:

sudo usermod -aG docker someuser

Be careful. The Docker group is effectively privileged.

Compose Command Not Found

Find Docker:

command -v docker

Use the full path in the unit:

ExecStart=/usr/bin/docker compose up -d --remove-orphans

If Compose plugin is missing:

docker compose version

Install it using your Docker package source.

Environment Variables Are Missing

Check the Compose config as systemd would see it:

cd /opt/myapp
docker compose config

If systemd needs extra environment variables, use:

EnvironmentFile=-/opt/myapp/.env.systemd

If Compose needs variables for substitution, use:

/opt/myapp/.env

These are related, but not identical.

Containers Do Not Start After Reboot

Check whether the systemd service is enabled:

systemctl is-enabled myapp.service

Enable it:

sudo systemctl enable myapp.service

Check Docker:

systemctl is-enabled docker
systemctl status docker

Check boot logs:

journalctl -u myapp.service -b --no-pager

App Starts Before Database Is Ready

Add a database health check and depends_on with service_healthy.

Also fix the application. It should retry database connections. Infrastructure startup ordering is helpful, but application retry logic is better.

Disk Filled with Docker Logs

Check Docker disk usage:

docker system df

Check large container logs:

sudo du -h /var/lib/docker/containers | sort -h | tail

Configure Docker log rotation in /etc/docker/daemon.json.

Then recreate containers.

Common Mistakes

Mistake 1: Running docker compose up in rc.local

Running docker compose up from rc.local or a login script works until it does not — use a proper systemd unit instead.

Mistake 2: Using Restart=always in systemd and restart: always in Compose

Usually you only need container restart policies in Compose. Avoid two supervisors fighting each other.

Mistake 3: Forgetting --remove-orphans

Service renames and removals can leave old containers behind. Use:

docker compose up -d --remove-orphans

Mistake 4: Using docker compose restart After Config Changes

restart restarts containers. It does not apply all configuration changes.

Use:

docker compose up -d

Mistake 5: Running down -v Without Thinking

This can delete volumes. For stateful services, that can mean deleting data.

Mistake 6: No Backup Before Pull

New images can break. Databases can migrate. Tags can move. Back up first.

Mistake 7: Publishing Every Port

Only publish what the host needs to expose. Internal service-to-service traffic can stay on the Compose network.

Final Recommended Pattern

For most single-host Linux services, use this pattern:

Compose file:

services:
  app:
    image: example/app:stable
    restart: unless-stopped
    ports:
      - "8080:8080"
    env_file:
      - .env

systemd unit:

[Unit]
Description=MyApp Docker Compose stack
Requires=docker.service
After=docker.service network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/myapp
ExecStartPre=/usr/bin/docker compose config --quiet
ExecStart=/usr/bin/docker compose up -d --remove-orphans
ExecReload=/usr/bin/docker compose up -d --remove-orphans
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0
TimeoutStopSec=120

[Install]
WantedBy=multi-user.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service

Operate it:

sudo systemctl status myapp.service
sudo systemctl restart myapp.service
journalctl -u myapp.service -f
cd /opt/myapp && docker compose logs -f

This pattern is not fancy, and that is the point. Docker Compose is excellent for small, understandable systems, systemd is excellent at starting and stopping host services, and together they give you a reliable single-server deployment model without pretending every project needs a cluster. For container-level commands outside Compose — images, volumes, networks, and cleanup — see the Docker Cheatsheet.

Install Docker on Ubuntu: APT, Snap, Rootless — Complete Guide 2026

Rost — Wed, 08 Jul 2026 09:50:22 +0000

Installing Docker on Ubuntu should be simple, but in practice several Docker-shaped options compete for the same command name, each with different packaging, upgrade behavior, and security implications.

This guide compares every major install path so you can pick the one that fits your machine.

The options you will encounter include:

docker.io from Ubuntu repositories
docker-ce from Docker's official APT repository
Docker from Snap
Docker Desktop
manually downloaded .deb packages
the Docker convenience script
rootless Docker

Although they all provide container tooling, they are not interchangeable packages. The best choice depends on whether the machine is a developer workstation, a CI runner, a small server, a self-hosting box, or a production host. My default recommendation is calm but firm: for most technical users on normal Ubuntu machines, install Docker Engine from Docker's official APT repository. Use Ubuntu's docker.io only when distribution integration matters more than upstream Docker packaging. Avoid the Snap package unless you specifically want Snap behavior and understand its limits. Rootless Docker is worth knowing about, but it is not automatically the best default for every machine.

This guide explains the tradeoffs, covers post-install security, and gives you clean installation paths for each method. Once Docker Engine is running, the Docker Cheatsheet is your daily command reference, and the Docker Compose Cheatsheet covers multi-container setups. Both sit alongside Git, VS Code, and CI/CD guides in Developer Tools: The Complete Guide to Modern Development Workflows.

Quick Recommendation

The table below summarizes which install path fits common scenarios.

Use case	Recommended install
Developer workstation	Docker official APT repo
CI runner	Docker official APT repo, version pinned if needed
Small self-hosted server	Docker official APT repo
Production server	Docker official APT repo, controlled upgrades
Ubuntu-only conservative system	Ubuntu `docker.io` package
Quick desktop experiment	Docker Desktop or official APT repo
Snap-managed Ubuntu setup	Docker Snap, with caution
Strong non-root daemon requirement	Rootless Docker
Air-gapped host	Manual `.deb` packages or internal mirror

If you do not have a special reason to choose otherwise, Docker's official APT repository is the default.

What Gets Installed

A normal Docker Engine setup includes several moving parts:

Docker daemon: dockerd
Docker CLI: docker
container runtime: containerd
low-level runtime: runc
Buildx plugin: docker buildx
Compose plugin: docker compose

Modern Docker Compose is usually installed as a Docker CLI plugin. That means the command is:

docker compose version

Not:

docker-compose version

The old docker-compose command still exists in older guides and older systems, but new Ubuntu setups should generally use the Compose plugin.

Option 1: Install Docker from Docker's Official APT Repository

This is the best default for most developers and DevOps users. You get Docker's upstream packaging, current Docker Engine releases, Buildx, Compose plugin, and a normal APT upgrade path.

Remove Conflicting Packages First

Before installing Docker CE, remove packages that may conflict with the official Docker packages.

sudo apt remove docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc

It is fine if APT says some of these packages are not installed.

This command does not remove Docker images, containers, volumes, or networks stored under /var/lib/docker. If you want a clean reset, that is a separate step and should be done deliberately.

Add Docker's Official APT Repository

Install prerequisites:

sudo apt update
sudo apt install ca-certificates curl

Create the keyring directory:

sudo install -m 0755 -d /etc/apt/keyrings

Download Docker's repository key:

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc

Allow APT to read the key:

sudo chmod a+r /etc/apt/keyrings/docker.asc

Add the Docker repository using the deb822 .sources format:

sudo tee /etc/apt/sources.list.d/docker.sources > /dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

Update APT metadata:

sudo apt update

Install Docker Engine, Buildx, and Compose

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Check the service:

sudo systemctl status docker

If it is not running:

sudo systemctl start docker

Verify the install:

sudo docker run hello-world

Check versions:

docker --version
docker buildx version
docker compose version

At this point Docker works, though you still need sudo for most commands unless you configure non-root access in the post-install section below.

Option 2: Install Docker from Ubuntu Repositories

Ubuntu provides the docker.io package, which you can install with:

sudo apt update
sudo apt install docker.io docker-compose-v2

Start and enable Docker:

sudo systemctl enable --now docker

Verify:

sudo docker run hello-world

When Ubuntu `docker.io` Makes Sense

Ubuntu's package can be a good choice when:

You prefer Ubuntu-maintained packages.
You want a version aligned with Ubuntu's release process.
You are managing many Ubuntu hosts with standard repositories.
You do not need the newest upstream Docker release.
You want fewer third-party APT sources.

This is a reasonable choice. It is not "wrong."

When Ubuntu `docker.io` Is Not Ideal

Use Docker's official repository instead when:

You want the current upstream Docker Engine.
You follow Docker's own documentation.
You rely on current Buildx and Compose behavior.
You want Docker CE package names.
You are debugging issues against upstream Docker docs.
You need predictable Docker-version parity across distributions.

My bias: for developer machines and container-heavy hosts, use Docker's official APT repository. For conservative Ubuntu-managed machines, docker.io is acceptable.

Option 3: Install Docker with Snap

The Docker snap installs with a single command, but simplicity does not always mean predictable behavior on a server or development machine.

sudo snap install docker

Snap packages have their own packaging model, update behavior, confinement assumptions, and filesystem layout. That is fine for many desktop apps, but Docker Engine is already a system-level container runtime, so the extra Snap layer can surprise people. If you manage other software with Snap, the Snap Package Manager Cheatsheet explains channels, confinement, and update behavior in more detail.

When Docker Snap Makes Sense

Docker Snap may be reasonable when:

You intentionally manage software with Snap.
You are using Ubuntu Core or a Snap-heavy environment.
You want snap-style automatic updates.
You are experimenting and do not care about matching Docker's upstream APT instructions.

Why I Usually Avoid Docker Snap

I usually avoid the Docker snap for development and server use because:

Most Docker documentation assumes the standard Docker Engine layout.
Troubleshooting paths can differ from APT installs.
Service management may feel less transparent.
Snap auto-updates can be inconvenient for infrastructure software.
Some bind mounts, sockets, and host integration details may surprise you.

If Docker is central to your workflow, install it like infrastructure rather than a casual desktop app — even when a Snap install looks tempting on the surface.

Option 4: Install Docker from Manual `.deb` Packages

Manual .deb installation is useful when:

The machine cannot use external APT repositories.
You are building an offline install process.
You mirror packages internally.
You need strict change control.

The tradeoff is maintenance, because you must download and install new packages manually whenever you upgrade.

A manual install usually requires these packages:

containerd.io
docker-ce
docker-ce-cli
docker-buildx-plugin
docker-compose-plugin

Install them with:

sudo dpkg -i ./containerd.io_*.deb \
  ./docker-ce_*.deb \
  ./docker-ce-cli_*.deb \
  ./docker-buildx-plugin_*.deb \
  ./docker-compose-plugin_*.deb

Then fix missing dependencies if needed:

sudo apt --fix-broken install

Verify:

sudo systemctl status docker
sudo docker run hello-world

Manual .deb installs are not my first choice, but they remain valid for controlled or air-gapped environments where explicit package approval matters more than convenience.

Option 5: Use Docker's Convenience Script

Docker provides a convenience script:

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

You can preview what it would do:

sudo sh get-docker.sh --dry-run

The convenience script is useful for disposable test machines, demos, labs, and throwaway environments, but I would not use it as the main install method for production systems. A script that configures repositories and installs packages non-interactively is convenient, yet long-lived infrastructure deserves explicit, reviewable steps — so for production hosts, use the APT repository method directly.

Docker Desktop vs Docker Engine on Ubuntu

Docker Desktop for Linux is a different product from Docker Engine. Docker Engine is the server-side runtime and CLI workflow most Linux server users expect, while Docker Desktop adds a GUI, Desktop integrations, and a product experience closer to macOS and Windows Docker usage.

Use Docker Desktop when:

You want a graphical Docker experience.
You want Desktop features.
You are aligning with a team that standardizes on Docker Desktop.
You do not mind the extra layer.

Use Docker Engine when:

You are running a server.
You want a simple Linux-native runtime.
You prefer systemd-managed services.
You are building CI, DevOps, or self-hosted infrastructure.
You do not need the GUI.

For an advanced technical blog audience, Docker Engine is usually the more interesting default on Linux servers and CI hosts.

Post-Install: Run Docker Without Sudo

After installation, this works:

sudo docker ps

But this may fail:

docker ps

That is because the Docker daemon listens on a Unix socket owned by root. The common fix is to add your user to the docker group.

Create the group if needed:

sudo groupadd docker

Add your user:

sudo usermod -aG docker $USER

Apply the new group membership:

newgrp docker

Or log out and log back in.

Test:

docker run hello-world

Important Security Note About the Docker Group

The docker group is not a harmless convenience group. A user who can control the Docker daemon can usually get root-equivalent control of the host, so for a personal developer machine this is often acceptable, but on a shared server it is a serious access-control decision. Treat membership in the docker group like admin access, and if that feels too broad, consider rootless Docker instead.

Rootless Docker on Ubuntu

Rootless Docker runs the Docker daemon and containers as a non-root user. This is not the same as adding your user to the docker group — with the docker group, the daemon still runs as root, whereas in rootless mode the daemon itself runs as your user.

When Rootless Docker Makes Sense

Rootless Docker is useful when:

You want to reduce daemon-level root risk.
You are on a shared development machine.
You run user-owned containers.
You do not need every advanced networking and storage feature.
You want a safer default for experimental workloads.

When Rootless Docker May Be Annoying

Rootless Docker can be less convenient when:

You need privileged containers.
You rely on low port binding without extra setup.
You need some host networking patterns.
You expect behavior identical to rootful Docker.
You are following guides written for normal Docker Engine installs.

Rootless mode improves security posture, but it is not zero friction compared with a standard rootful install.

Install Rootless Docker

Install prerequisites:

sudo apt update
sudo apt install uidmap

If you installed Docker from DEB or APT packages, the rootless setup tool should be available:

dockerd-rootless-setuptool.sh install

If rootful Docker is already running and you want only rootless Docker, disable the system daemon:

sudo systemctl disable --now docker.service docker.socket

You may also need to remove the rootful socket:

sudo rm -f /var/run/docker.sock

After installing rootless Docker, the setup tool usually prints environment variables to add to your shell profile. They commonly look like this:

export PATH=/usr/bin:$PATH
export DOCKER_HOST=unix:///run/user/1000/docker.sock

Your UID may differ. Check the exact output from the setup tool.

Enable lingering if you want the user service to run after logout:

sudo loginctl enable-linger $USER

Check the user service:

systemctl --user status docker

Run a test container:

docker run hello-world

Rootful Docker vs Rootless Docker

Topic	Rootful Docker	Rootless Docker
Daemon user	root	normal user
Command convenience	high	medium
Compatibility	highest	good, but not perfect
Security posture	weaker by default	better by default
Privileged containers	supported	limited
Low ports	simple	needs extra setup
Server usage	common	possible, but plan carefully
Developer workstation	common	good for security-conscious users

My practical recommendation:

Use normal rootful Docker for a personal dev machine or simple server.
Add only trusted users to the docker group.
Use rootless Docker when user isolation matters.
Do not pretend the docker group is a security boundary.

Enable Docker at Boot

On Ubuntu, Docker installed from normal packages usually starts automatically, but it is worth confirming after a fresh install or migration.

Check:

systemctl is-enabled docker
systemctl is-enabled containerd

Enable manually if needed:

sudo systemctl enable docker.service
sudo systemctl enable containerd.service

Start now:

sudo systemctl start docker

Disable auto-start:

sudo systemctl disable docker.service
sudo systemctl disable containerd.service

For rootless Docker, use the user service:

systemctl --user enable docker
systemctl --user start docker

And enable lingering if needed:

sudo loginctl enable-linger $USER

Install or Verify Docker Compose

With Docker's official APT repository, install Compose as a plugin:

sudo apt update
sudo apt install docker-compose-plugin

Verify:

docker compose version

If you installed Ubuntu's package, you may use:

sudo apt install docker-compose-v2

Prefer:

docker compose up -d

Over the old style:

docker-compose up -d

The hyphenated docker-compose command belongs to the older standalone Compose era. Many systems still have it, but new documentation should use docker compose.

Check Which Docker You Have Installed

When you inherit a machine or debug a broken setup, these commands help identify which Docker packaging is actually in use.

Check the Docker binary:

which docker

Check package ownership:

dpkg -S "$(which docker)"

List Docker-related packages:

dpkg -l | grep -E 'docker|containerd|runc'

Check APT policy:

apt-cache policy docker-ce docker.io containerd.io docker-compose-plugin docker-compose-v2

Check Snap:

snap list | grep docker

Check service status:

systemctl status docker
systemctl status containerd

Check Docker server details:

docker info

If docker info fails without sudo, check your group membership:

groups

Migrate from Ubuntu `docker.io` to Docker CE

Stop Docker:

sudo systemctl stop docker

Remove Ubuntu packages:

sudo apt remove docker.io docker-compose docker-compose-v2 docker-doc containerd runc

Add Docker's official APT repository using the steps above.

Install Docker CE:

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Start Docker:

sudo systemctl start docker

Check existing containers:

docker ps -a
docker images
docker volume ls

Normally, removing packages does not delete /var/lib/docker, so your images, containers, and volumes may still exist. Still, back up important data first.

Migrate from Docker Snap to Docker CE

First inspect what exists:

snap list | grep docker
docker info

Stop workloads and back up important volumes or bind-mounted data.

Remove the snap:

sudo snap remove docker

Then install Docker CE from the official APT repository.

Be careful with data locations. Snap packages often use different paths and confinement rules. Do not assume that Docker Snap data will appear automatically under the standard /var/lib/docker path after migration.

For important services, export or back up data explicitly before switching package sources.

Pin a Docker Version with APT

For production-like systems, you may want controlled upgrades.

List available versions:

apt list --all-versions docker-ce

Install a specific version:

VERSION_STRING="5:29.0.0-1~ubuntu.24.04~noble"

sudo apt install docker-ce=$VERSION_STRING \
  docker-ce-cli=$VERSION_STRING \
  containerd.io \
  docker-buildx-plugin \
  docker-compose-plugin

Hold Docker packages:

sudo apt-mark hold docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Remove the hold later:

sudo apt-mark unhold docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Do this when you need predictable change windows. For a personal laptop, it may be unnecessary.

Firewall and Networking Notes

Docker modifies packet filtering rules to make container networking work, which matters if you use UFW, firewalld, nftables, or custom firewall rules.

Important points:

Published Docker ports may bypass naive UFW expectations.
Docker uses iptables integration.
Custom rules should account for Docker-created chains.
Server hardening should be tested with real container port mappings.
Do not assume ufw deny protects a port published by Docker.

Test exposed ports from another machine, not only from localhost.

Example:

docker run --rm -p 8080:80 nginx

Then from another host:

curl http://server-ip:8080

On servers, Docker networking is part of your security model, not an implementation detail you can ignore after install.

Common Errors and Fixes

Permission Denied on Docker Socket

Error:

permission denied while trying to connect to the Docker daemon socket

Fix options:

Use sudo:

sudo docker ps

Or add your user to the docker group:

sudo usermod -aG docker $USER
newgrp docker

Remember that the docker group is root-equivalent in practice, so treat group membership as a privileged access decision.

Docker Daemon Is Not Running

Check status:

sudo systemctl status docker

Start it:

sudo systemctl start docker

Check logs:

journalctl -u docker --no-pager -n 100

Conflicting Docker Packages

If installing Docker CE fails because of conflicts, remove old packages. If APT itself is in a bad state after adding the Docker repository, work through Ubuntu APT troubleshooting for broken packages and GPG errors before retrying.

sudo apt remove docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc

Then retry:

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Compose Command Not Found

Check modern Compose:

docker compose version

Install plugin:

sudo apt install docker-compose-plugin

If you expected the old command:

docker-compose version

You may be following old documentation. Prefer updating the command to docker compose.

Old Root-Owned Docker Config

If you ran Docker with sudo before setting up group access, your user config may be owned by root.

Fix ownership:

sudo chown "$USER":"$USER" "$HOME/.docker" -R
sudo chmod g+rwx "$HOME/.docker" -R

Or remove the config if you do not need it:

sudo rm -rf "$HOME/.docker"

It will be recreated.

Cannot Connect to Rootless Docker

Check environment:

echo "$DOCKER_HOST"

Check user service:

systemctl --user status docker

Set the socket path if needed:

export DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock

Add it to your shell profile only after confirming it is correct.

Uninstall Docker Engine

Remove Docker CE packages:

sudo apt purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras

Remove Docker data only if you really want to delete images, containers, and volumes:

sudo rm -rf /var/lib/docker
sudo rm -rf /var/lib/containerd

Remove Docker repository files:

sudo rm -f /etc/apt/sources.list.d/docker.sources
sudo rm -f /etc/apt/keyrings/docker.asc

Refresh APT:

sudo apt update

For Docker Snap:

sudo snap remove docker

For Ubuntu docker.io:

sudo apt purge docker.io docker-compose-v2

Recommended Install Path for Most Ubuntu Users

For most technical Ubuntu users, this is the clean path:

sudo apt remove docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc

sudo apt update
sudo apt install ca-certificates curl

sudo install -m 0755 -d /etc/apt/keyrings

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc

sudo chmod a+r /etc/apt/keyrings/docker.asc

sudo tee /etc/apt/sources.list.d/docker.sources > /dev/null <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update

sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

sudo docker run hello-world

Then decide whether you want group access for convenience or rootless Docker for tighter isolation:

sudo usermod -aG docker $USER
newgrp docker
docker run hello-world

Final Opinionated Guidance

Docker installation on Ubuntu is an operational choice, not just a package choice. The method you pick affects upgrades, security boundaries, and how closely your host matches upstream Docker documentation.

My practical rules are:

Use Docker's official APT repository for most developer and DevOps machines.
Use Ubuntu docker.io when Ubuntu repository consistency matters more than upstream freshness.
Avoid Docker Snap for serious container workflows unless you intentionally want Snap behavior.
Avoid the convenience script for production hosts.
Use modern docker compose, not old docker-compose.
Treat the docker group as privileged access.
Consider rootless Docker when user isolation matters.
Pin versions on production-like systems.
Test firewall behavior when publishing ports on servers.

The most boring Docker install is usually the best one: official APT repo, explicit keyring, Compose plugin, systemd service, understood permissions, and no mystery packaging layer in the middle. Once the engine is in place, multi-service workloads — including self-hosted LLM stacks — are a natural next step; for example, see Ollama in Docker Compose for running local models in containers.

https://docs.docker.com/engine/install/ubuntu/ "Install Docker Engine on Ubuntu | Docker Docs"
https://packages.ubuntu.com/noble/docker.io?utm_source=chatgpt.com "Ubuntu - Details of package docker.io in noble"

Ubuntu APT Troubleshooting: Fix Broken Packages, Holds, and GPG Errors

Rost — Tue, 07 Jul 2026 09:00:05 +0000

APT failures are common on long-lived Ubuntu machines, and they usually appear after a release upgrade, a third-party repository change, a removed PPA, a manually installed .deb, or an interrupted package installation.

The error message can look dramatic, but most APT problems are not mysterious — they are state problems where APT's view of repositories, versions, and installed packages no longer agrees.

APT is trying to answer four questions:

Which repositories are enabled?
Which package versions are available?
Which packages are already installed?
Which package changes are allowed?

When those answers conflict, you get held packages, broken dependencies, missing public keys, bad PPA errors, or packages kept back during upgrade. This guide gives you a practical troubleshooting sequence for Ubuntu APT, written for developers, DevOps engineers, and Linux users who want to fix the system without blindly pasting random commands from forum threads. Pair it with our Ubuntu APT and dpkg cheatsheet for everyday install and upgrade commands, and browse the wider Developer Tools collection for related Linux workflows.

The Short Version

If your system is only mildly broken, start here:

sudo apt update
sudo dpkg --configure -a
sudo apt --fix-broken install
sudo apt update
sudo apt upgrade

If packages are kept back:

apt list --upgradable
apt-mark showhold
sudo apt full-upgrade

If a PPA or third-party repository is failing:

ls /etc/apt/sources.list.d/
sudo apt update

Read the failing repository line, then disable or fix that repository. If you see NO_PUBKEY, do not blindly import random keys from a keyserver — find the official repository instructions, install the repository key into /etc/apt/keyrings, and bind it to that repository with signed-by.

Before You Fix Anything: Read the APT Error First

Run this first:

sudo apt update

Do not skip it. apt update refreshes package metadata. It does not upgrade packages. It tells you whether Ubuntu can read all configured repositories and verify their metadata.

Then check your Ubuntu version and codename — a stale release name in /etc/apt/sources.list.d/ is a frequent cause of 404 and Release file errors. If you are unsure which release you are on, see how to check your Ubuntu version:

lsb_release -a

Or:

cat /etc/os-release

Also check what is upgradeable:

apt list --upgradable

And check whether any package is held:

apt-mark showhold

This gives you the first split in the decision tree — identifying the class of failure first makes troubleshooting easier because each class has a different first fix:

Repository problem: apt update fails.
Dependency problem: apt update works, but install or upgrade fails.
Held package problem: APT refuses to change specific packages.
Mixed source problem: A PPA, manual .deb, or old repository provides incompatible versions.
Interrupted install problem: dpkg has unpacked packages that were never configured.

Common Ubuntu APT Failure Types

Packages Kept Back

A kept-back package is not always an error; it means APT chose not to upgrade a package using the current command. This often happens when the upgrade requires installing new dependencies, removing old packages, or changing a package in a way that plain apt upgrade will not perform.

Check details:

apt list --upgradable
apt-cache policy package-name

Try a full upgrade only after reading what APT wants to do:

sudo apt full-upgrade

full-upgrade is allowed to install new packages and remove packages if needed to complete the upgrade. That is useful, but it is also why you should read the proposed changes before accepting. On a workstation, full-upgrade is usually fine after review; on a server, read the removals twice.

Held Packages

A held package is different from a package that is merely kept back. A hold is an explicit instruction that prevents APT from automatically installing, upgrading, or removing that package. Holds are useful for pinning a kernel, database, driver, or runtime version, and they are also easy to forget.

List held packages:

apt-mark showhold

Hold a package:

sudo apt-mark hold package-name

Remove a hold:

sudo apt-mark unhold package-name

If you see dependency errors involving a held package, decide whether the hold is still intentional. Do not remove holds automatically. They may be protecting a production service, driver, or compatibility-sensitive toolchain.

Broken Dependencies

Broken dependencies mean APT cannot find a valid package set, which usually points to a version conflict rather than a corrupted download.

Common causes include:

A package requires a version that is not available.
A PPA provides a newer library than Ubuntu expects.
A manually installed .deb depends on packages from another release.
A package install was interrupted.
A repository for the wrong Ubuntu release is enabled.
Package pinning or holds prevent the needed version change.

Start with:

sudo dpkg --configure -a
sudo apt --fix-broken install

Then inspect the package:

apt-cache policy package-name
apt-cache depends package-name
apt-cache rdepends package-name

Then use those commands to find the package version conflict that is blocking APT, rather than running every repair command in sequence.

GPG Key and NO_PUBKEY Errors

A NO_PUBKEY error means APT received repository metadata, but it cannot verify the signature because the required public key is missing — a trust problem, not merely a download problem.

A typical error looks like this:

The following signatures could not be verified because the public key is not available: NO_PUBKEY ABCD1234EF567890

Old instructions often used apt-key add, but avoid that for new repository setup. Modern Ubuntu systems should use a repository-specific keyring and signed-by.

A better pattern looks like this:

sudo install -d -m 0755 /etc/apt/keyrings

curl -fsSL https://example.com/repo-key.gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/example.gpg

echo "deb [signed-by=/etc/apt/keyrings/example.gpg] https://example.com/apt stable main" \
  | sudo tee /etc/apt/sources.list.d/example.list

sudo apt update

Replace the URL and repository line with the official instructions from the vendor.

The important part is this:

signed-by=/etc/apt/keyrings/example.gpg

That signed-by line binds the key to one repository, which is cleaner and safer than putting every third-party key into a global trust store.

Bad PPA or Release File Errors

PPA failures often look like this:

The repository does not have a Release file.

Or:

404 Not Found

Common causes:

The PPA does not support your Ubuntu release.
You upgraded Ubuntu, but the PPA still points to the old codename.
The PPA was deleted.
The maintainer stopped publishing packages.
You added a PPA intended for a different Ubuntu version.

Check your codename:

. /etc/os-release
echo "$VERSION_CODENAME"

List third-party source files:

ls /etc/apt/sources.list.d/

Inspect them:

grep -R "^deb" /etc/apt/sources.list /etc/apt/sources.list.d/

Disable a suspect source by renaming it:

sudo mv /etc/apt/sources.list.d/example.list /etc/apt/sources.list.d/example.list.disabled
sudo apt update

If APT works after disabling the file, you have found the problematic source and can fix or remove it permanently.

A Safe APT Troubleshooting Workflow

Step 1: Refresh Metadata

sudo apt update

If this fails, fix repositories before trying package repair. APT cannot solve dependencies correctly if its package index is stale or incomplete.

Look for lines containing:

NO_PUBKEY
Release file
404 Not Found
does not have a Release file
The repository is not signed

These are repository or trust problems, and they should be fixed before you attempt any package repair.

Step 2: Finish Interrupted Package Configuration

If a package install was interrupted, dpkg may have unpacked files but not configured the package.

Run:

sudo dpkg --configure -a

If this succeeds, continue:

sudo apt --fix-broken install

If it fails, read the package name and error carefully. The package named at the bottom of the error is often more important than the package you originally tried to install.

Step 3: Ask APT to Repair Dependencies

sudo apt --fix-broken install

This command asks APT to correct dependency problems by installing missing dependencies or removing packages that cannot be satisfied. Read the proposed action carefully, especially any removals.

Be cautious if APT wants to remove:

ubuntu-desktop
ubuntu-server
linux-generic
display manager packages
network packages
database packages
container runtime packages
desktop environment packages

Sometimes removing a metapackage is harmless. Sometimes it is a warning sign that your package sources are mixed badly. Do not accept large removals without understanding them.

Step 4: Check Held Packages

apt-mark showhold

If a held package is blocking the upgrade, inspect it:

apt-cache policy package-name

If the hold is no longer needed:

sudo apt-mark unhold package-name
sudo apt update
sudo apt upgrade

If the hold is intentional, do not fight APT — fix the repository or package selection around the hold instead of removing the protection.

Step 5: Inspect Package Versions

For a package with dependency trouble:

apt-cache policy package-name

This shows:

Installed version
Candidate version
Available versions
Which repository provides each version

apt-cache policy is one of the most useful APT debugging commands because it shows where each available version comes from.

Example:

apt-cache policy docker-ce

If the candidate version comes from an unexpected PPA or old repository, you have found the likely cause of the dependency conflict.

Step 6: Look for Mixed Repositories

List enabled sources:

grep -R "^deb" /etc/apt/sources.list /etc/apt/sources.list.d/

Look for:

Old Ubuntu codenames
Debian repositories on Ubuntu
PPAs for another Ubuntu release
Duplicate vendor repositories
Both Snap and APT instructions mixed for the same tool
Old .list files left behind after uninstalling software

A common anti-pattern is installing a tool from a vendor repository, then later adding a PPA or manual .deb that provides overlapping libraries. APT can handle many sources, but it cannot reconcile conflicting intentions unless you align the repositories yourself.

Step 7: Try a Simulated Install or Upgrade

Before making changes, simulate:

apt -s install package-name

Or:

apt -s full-upgrade

The -s option simulates the operation. It is useful when you want to see what APT would do without changing the system.

Fixing Held Packages

List Held Packages

apt-mark showhold

If nothing is printed, no packages are held with apt-mark, and you can move on to dependency or repository checks.

Hold a Package Intentionally

sudo apt-mark hold package-name

Good reasons to hold a package:

A kernel version is known to work with your hardware.
A database upgrade needs planning.
A driver update breaks your GPU.
A runtime version must match production.
A vendor package is not compatible with the latest dependency.

Bad reasons to hold a package:

You copied a command from an old guide.
You forgot why it was held.
You are avoiding a dependency error without understanding it.

Remove a Hold

sudo apt-mark unhold package-name

Then update and upgrade:

sudo apt update
sudo apt upgrade

If the package still will not upgrade, it was not only a hold problem. Check policy:

apt-cache policy package-name

Fixing Broken Dependencies

The Standard Repair Sequence

Use this sequence when package installation or upgrade failed halfway:

sudo apt update
sudo dpkg --configure -a
sudo apt --fix-broken install
sudo apt upgrade

This order matters because each step prepares the ground for the next: apt update refreshes repository metadata, dpkg --configure -a finishes configuring unpacked packages, apt --fix-broken install lets APT reconcile missing or conflicting dependencies, and apt upgrade resumes normal package upgrades.

Remove a Badly Installed Local Package

If the problem started after installing a downloaded .deb, inspect it:

dpkg -l | grep package-name
apt-cache policy package-name

Remove it:

sudo apt remove package-name

If configuration files are also causing problems:

sudo apt purge package-name

Then repair:

sudo apt --fix-broken install

Reinstall a Damaged Package

If a package is installed but broken:

sudo apt install --reinstall package-name

This is useful when files are missing or corrupted, but the package sources are otherwise healthy and you want to refresh the installed files without changing versions.

Fixing PPA and Third-Party Repository Problems

Find PPAs and Third-Party Repositories

ls /etc/apt/sources.list.d/

Show active repository lines:

grep -R "^deb" /etc/apt/sources.list /etc/apt/sources.list.d/

On newer Ubuntu systems, you may also see .sources files using the deb822 format:

ls /etc/apt/sources.list.d/*.sources

Display them:

cat /etc/apt/sources.list.d/*.sources

Disable a PPA Safely

Rename the source file:

sudo mv /etc/apt/sources.list.d/example.list /etc/apt/sources.list.d/example.list.disabled
sudo apt update

For deb822 files:

sudo mv /etc/apt/sources.list.d/example.sources /etc/apt/sources.list.d/example.sources.disabled
sudo apt update

Renaming the source file is reversible and safer than deleting repository configuration immediately, because you can rename it back if you disabled the wrong source.

Remove Packages from a PPA

Disabling a PPA stops future package downloads from it. It does not automatically downgrade packages already installed from that PPA.

If the PPA caused library conflicts, you may need to downgrade packages back to Ubuntu versions.

Install ppa-purge:

sudo apt install ppa-purge

Then purge the PPA:

sudo ppa-purge ppa:owner/name

Use ppa-purge carefully and read the proposed changes before accepting, because it may remove or downgrade several related packages.

After a Release Upgrade

After upgrading Ubuntu, old PPAs are a common source of errors.

Check for old codenames:

grep -R "jammy\|noble\|oracular\|plucky" /etc/apt/sources.list /etc/apt/sources.list.d/

Adjust the codenames for your real system. For example, if you are on Ubuntu 24.04, your codename is noble. If a third-party source still points to an older codename, verify whether that vendor supports your current Ubuntu release. If you are setting up a new machine rather than repairing an upgrade, our Ubuntu 24.04 install guide walks through adding vendor repositories with signed-by from the start.

Do not just edit the codename and hope for the best — some repositories do not publish packages for every Ubuntu version, so verify vendor support for your release first.

Fixing GPG and NO_PUBKEY Errors

What NO_PUBKEY Means

APT repositories publish signed metadata, and your machine needs the matching public key to verify that metadata. If the key is missing, APT refuses to trust the repository, which is the behavior you want — do not disable signature checks just to make the error disappear.

The Modern Keyring Pattern

Create the keyring directory:

sudo install -d -m 0755 /etc/apt/keyrings

Download and dearmor the vendor key:

curl -fsSL https://example.com/repository-key.gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/example.gpg

Set readable permissions:

sudo chmod 0644 /etc/apt/keyrings/example.gpg

Add the repository with signed-by:

echo "deb [signed-by=/etc/apt/keyrings/example.gpg] https://example.com/linux/ubuntu noble stable" \
  | sudo tee /etc/apt/sources.list.d/example.list

Then update:

sudo apt update

Replace noble with your Ubuntu codename if needed:

. /etc/os-release
echo "$VERSION_CODENAME"

Why apt-key Is the Wrong Habit

Old guides often use:

curl -fsSL https://example.com/key.gpg | sudo apt-key add -

Avoid apt-key add for new setups. The old apt-key style adds keys to a broad trust area, which makes it harder to reason about which key is trusted for which repository, whereas the modern signed-by style scopes the key to a specific repository and is basic supply-chain hygiene.

Find Legacy Trusted Keys

You may still have old keys in:

/etc/apt/trusted.gpg
/etc/apt/trusted.gpg.d/

List files:

ls -l /etc/apt/trusted.gpg.d/

Do not delete keys randomly — first map each key to a repository, then migrate one repository at a time to /etc/apt/keyrings and signed-by.

Common GPG Mistakes

Do not use random keyservers as your first choice when fixing NO_PUBKEY errors.

Better order:

Use the vendor's official install documentation.
Download the key from the vendor's official HTTPS URL.
Store it in /etc/apt/keyrings.
Bind it with signed-by.
Run sudo apt update.

Avoid these shortcuts:

sudo apt update --allow-unauthenticated
sudo apt install --allow-unauthenticated package-name

They may work temporarily, but they remove the signature verification that protects you from tampered repository metadata.

Fixing "The Repository Is Not Signed"

This error usually means one of these things:

The repository does not publish signed metadata.
The repository URL is wrong.
The repository no longer supports your Ubuntu version.
A proxy or mirror is returning the wrong content.
You are using HTTP where the vendor now expects HTTPS.
The source file has the wrong suite or component.

Find the failing source:

sudo apt update

APT will print the URL. Then search for it:

grep -R "example.com" /etc/apt/sources.list /etc/apt/sources.list.d/

Disable it temporarily:

sudo mv /etc/apt/sources.list.d/example.list /etc/apt/sources.list.d/example.list.disabled
sudo apt update

If APT works again after disabling the file, reinstall that repository from the vendor's current official instructions rather than re-enabling the old configuration.

Fixing Duplicate Repository Warnings

APT may warn that a target is configured multiple times.

List matching entries:

grep -R "repo-url-or-domain" /etc/apt/sources.list /etc/apt/sources.list.d/

Duplicate repositories often appear after running vendor install scripts multiple times.

Keep one source file. Disable the others:

sudo mv /etc/apt/sources.list.d/duplicate.list /etc/apt/sources.list.d/duplicate.list.disabled
sudo apt update

Duplicate warnings are not always fatal, but they are a sign of sloppy configuration, so keep one source file and disable the duplicates.

Fixing Packages from the Wrong Ubuntu Release

One of the worst APT problems is mixing Ubuntu releases — for example, a machine on Ubuntu 24.04 should not casually pull packages from Ubuntu 22.04 or Debian testing. Sometimes it works for a while, but eventually the dependency graph becomes a puzzle that APT cannot solve cleanly.

Check your release:

. /etc/os-release
echo "$VERSION_CODENAME"

Search sources:

grep -R "^deb" /etc/apt/sources.list /etc/apt/sources.list.d/

Look for foreign codenames in the enabled sources, then inspect the affected package:

apt-cache policy package-name

If the installed version came from an old or foreign repository, disable that repository and downgrade or reinstall the affected package from Ubuntu repositories.

A conservative repair path is:

sudo apt update
sudo apt install --reinstall package-name

For deeper conflicts, you may need to remove the package and reinstall it from the correct source rather than forcing an upgrade over a foreign version.

Cleaning APT Cache and Unused Packages

APT cache cleanup is not a dependency fix on its own, but it can help after many failed installs by reclaiming disk space and clearing stale package files.

Remove packages that were installed automatically and are no longer needed:

sudo apt autoremove

Clean downloaded package files:

sudo apt clean

Or remove only obsolete package files:

sudo apt autoclean

Use autoremove carefully on servers and desktops with manually installed driver stacks, and read the removal list before accepting.

Practical APT Troubleshooting Recipes

Recipe: Package Is Kept Back

sudo apt update
apt list --upgradable
apt-mark showhold
sudo apt full-upgrade

If APT proposes reasonable changes after simulation, accept them. If it proposes large removals, stop and inspect:

apt-cache policy package-name

Recipe: Held Package Blocks Upgrade

apt-mark showhold
apt-cache policy package-name
sudo apt-mark unhold package-name
sudo apt upgrade

Only unhold a package if the hold is no longer intentional, because removing a hold that protects production software can trigger a breaking upgrade.

Recipe: Interrupted Install

sudo dpkg --configure -a
sudo apt --fix-broken install
sudo apt upgrade

Recipe: NO_PUBKEY Error

Identify the repository from sudo apt update.
Find the vendor's current official install instructions.
Install the key into /etc/apt/keyrings.
Use signed-by in the source file.
Run sudo apt update.

Example structure:

sudo install -d -m 0755 /etc/apt/keyrings

curl -fsSL https://example.com/key.gpg \
  | sudo gpg --dearmor -o /etc/apt/keyrings/example.gpg

sudo chmod 0644 /etc/apt/keyrings/example.gpg

echo "deb [signed-by=/etc/apt/keyrings/example.gpg] https://example.com/ubuntu noble main" \
  | sudo tee /etc/apt/sources.list.d/example.list

sudo apt update

Recipe: PPA Does Not Have a Release File

sudo apt update
ls /etc/apt/sources.list.d/
grep -R "ppa.launchpadcontent.net\|launchpad" /etc/apt/sources.list.d/

Disable the PPA:

sudo mv /etc/apt/sources.list.d/example.list /etc/apt/sources.list.d/example.list.disabled
sudo apt update

Then decide whether to remove, replace, or purge packages from that PPA.

Recipe: Manual `.deb` Broke Dependencies

dpkg -l | grep package-name
apt-cache policy package-name
sudo apt remove package-name
sudo apt --fix-broken install

If you still need the software, prefer the vendor's current APT repository over repeated manual .deb installs, which tend to accumulate dependency conflicts over time.

Essential APT Troubleshooting Commands

Repository and Metadata

sudo apt update
grep -R "^deb" /etc/apt/sources.list /etc/apt/sources.list.d/
ls /etc/apt/sources.list.d/

Package State

apt list --installed
apt list --upgradable
apt-mark showhold
dpkg -l | grep package-name

Package Policy and Dependencies

apt-cache policy package-name
apt-cache depends package-name
apt-cache rdepends package-name

Repair

sudo dpkg --configure -a
sudo apt --fix-broken install
sudo apt install --reinstall package-name
sudo apt full-upgrade

Cleanup

sudo apt autoremove
sudo apt autoclean
sudo apt clean

Simulation

apt -s install package-name
apt -s full-upgrade

What Not To Do

Do Not Randomly Delete `/var/lib/dpkg`

If you see advice to delete dpkg state files, be skeptical. The dpkg database is the record of installed packages, and deleting pieces of it can turn a repairable package issue into a full system recovery project.

Do Not Disable Signature Verification

Avoid:

--allow-unauthenticated

If a repository cannot be verified, fix the key or disable the repository rather than bypassing authentication.

Do Not Mix Ubuntu Releases Casually

Do not add repositories for another Ubuntu release unless you understand APT pinning and the dependency consequences.

This applies especially to:

desktop environments
graphics drivers
Python stacks
container runtimes
Kubernetes packages
database packages

Do Not Treat PPAs as Harmless

PPAs are useful, but they are still package repositories that can replace libraries and system packages — which may be exactly what you want, or exactly why your next upgrade breaks. Prefer PPAs for specific applications, not for broad system foundations, unless you trust the maintainer and understand the upgrade path.

APT Troubleshooting Decision Tree

Use this mental model:

flowchart TD
    A["Does sudo apt update fail?"] -->|yes| B["Fix repositories, GPG keys, PPAs, network, or release files"]
    A -->|no| C["Did an install or upgrade get interrupted?"]
    C -->|yes| D["Run dpkg --configure -a, then apt --fix-broken install"]
    C -->|no| E["Are packages held?"]
    E -->|yes| F["Inspect apt-mark showhold and decide whether to unhold"]
    E -->|no| G["Are packages kept back?"]
    G -->|yes| H["Inspect apt full-upgrade simulation and package policy"]
    G -->|no| I["Is a third-party source involved?"]
    I -->|yes| J["Inspect apt-cache policy and source files"]
    I -->|no| K["Inspect the specific package dependency error"]

Most APT problems become manageable once you stop treating them as one big error and start separating repository health, package state, dependency solving, and trust configuration — the decision tree above is a shorthand for that discipline.

Recommended Baseline for Developer Machines

For a clean Ubuntu developer workstation, I prefer this baseline:

Keep Ubuntu repositories standard.
Use vendor APT repositories only when they are official and maintained.
Use /etc/apt/keyrings and signed-by for third-party repositories.
Avoid old apt-key instructions.
Avoid mixing PPAs that replace core system libraries.
Use containers, uv, pipx, asdf, mise, or language-native tools for fast-moving developer dependencies.
Keep APT responsible for the operating system, drivers, services, and stable CLI tools.

For desktop software, prefer Flatpak or Snap over PPAs when a sandboxed universal package fits your needs. APT is excellent when it manages the base system, but it becomes painful when it is forced to behave like a universal developer dependency manager for fast-moving language ecosystems.

Final APT Troubleshooting Checklist

When APT is broken on Ubuntu, work through this checklist:

[ ] Run sudo apt update and read the first real error.
[ ] Check Ubuntu codename with /etc/os-release.
[ ] Finish interrupted installs with dpkg --configure -a.
[ ] Repair dependencies with apt --fix-broken install.
[ ] Check held packages with apt-mark showhold.
[ ] Inspect package versions with apt-cache policy.
[ ] Disable broken PPAs or third-party repositories.
[ ] Replace apt-key style repositories with signed-by keyrings.
[ ] Simulate risky operations with apt -s.
[ ] Read removals before accepting full-upgrade or autoremove.

APT is not fragile, but it is strict, and that strictness is a feature: it prevents unsigned repositories, impossible dependency sets, and accidental package replacements from silently changing your system. The calm way to fix APT is to preserve that strictness, find the conflicting state, and repair the smallest thing that is actually wrong.

Multi-Agent Orchestration Patterns: A Practical Guide

Rost — Mon, 06 Jul 2026 11:25:31 +0000

Single-agent AI systems peaked in 2025 — you gave one LLM a prompt, some tools, and a goal, and it did reasonably well on bounded tasks.

In 2026, multi-agent systems have moved from research demos to production infrastructure. Gartner reports a 1,445% increase in multi-agent system inquiries from Q1 2024 to Q2 2025, while Salesforce's 2026 Connectivity Benchmark Report found organizations use an average of 12 agents, projected to grow 67% within two years. The AI Systems cluster covers the full stack these systems operate on — from inference and memory to routing and observability.

But here's what's less discussed: 40% of multi-agent pilots fail within six months of production deployment. The failure isn't that multi-agent systems don't work. The failure is that teams pick the wrong orchestration pattern for their problem — or pick the right one without understanding how it breaks.

This guide covers the orchestration patterns that hold up in production, the specific ways each one fails, and a decision framework for picking the right architecture.

The Core Problem: Coordination Is Hard

When you move from a single AI agent to multiple agents working together, the first engineering question is: how do they coordinate?

The coordination model — the orchestration pattern — determines your system's latency, fault tolerance, scalability ceiling, and debugging complexity. It is consistently the highest-impact architectural decision in multi-agent design, conditioning every subsequent implementation choice.

Every production multi-agent system maps to one of six canonical patterns, or a hybrid of two or more. The patterns emerge from distributed systems constraints: coordination cost, fault isolation, throughput requirements, and observability.

Pattern 1: Orchestrator-Worker

How It Works

Orchestrator-Worker is the centralized hub-and-spoke model of multi-agent coordination. A single orchestrator agent receives the task, decomposes it into subtasks, delegates each subtask to a specialist worker agent, and aggregates the results. Workers do not communicate directly with each other — all coordination flows through the orchestrator, which holds the full plan and decision-making authority.

graph TD
    O[Orchestrator<br/>planner] --> WA[Worker A]
    O --> WB[Worker B]
    O --> WC[Worker C]

When to Use It

Cross-functional workflows with clear task decomposition
Triage and routing scenarios (customer support, incident classification)
Workloads where a single accountability point is required
Tasks where the orchestrator can use a capable model while workers use cheaper, task-specific models

Real-world example: Salesforce Agentforce 2.0 uses orchestrator-worker to decompose customer inquiries into research, draft, and review stages.

How It Fails

Single point of failure. The orchestrator is both a bottleneck and a failure point. If the orchestrator's LLM call takes 3 seconds and you have 20 workers waiting for assignments, your decomposition throughput ceiling is roughly 6.7 tasks per second. If the orchestrator misclassifies a task, the wrong worker gets it — and misclassification rates compound at scale.

Context overflow. The orchestrator accumulates context from all workers. At 4+ workers, the orchestrator frequently exceeds context limits because it holds the full conversation history for every worker interaction simultaneously.

Cost explosion. Workflows that cost $0.50 in testing can hit $50,000/month at 100K executions. The orchestrator makes multiple LLM calls for decomposition and aggregation on top of every worker call. At scale, the overhead dominates the worker cost.

Mitigations

Set explicit interface contracts between orchestrator and workers
Require structured outputs from workers (JSON schemas, typed responses)
Bound sub-task budgets (token limits, step limits) to prevent runaway costs
Consider a hierarchical variant (see Pattern 4) when worker count exceeds 5

Pattern 2: Sequential Pipeline

How It Works

Sequential Pipeline is the linear chain with shared state — a predefined sequence of agents with deterministic order, where each stage transforms or enriches data and passes it to the next. There is no runtime branching; the execution order is fixed at design time, making the pattern highly predictable but inflexible.

graph LR
    I[Input] --> A1[Agent 1<br/>stage A]
    A1 --> A2[Agent 2<br/>stage B]
    A2 --> A3[Agent 3<br/>stage C]
    A3 --> O[Output]

When to Use It

Document processing workflows (ingest → extract → validate → output)
Content generation pipelines (research → draft → edit → publish)
Compliance verification (generate → check → revise → approve)
Data enrichment and ETL workflows

Real-world example: Microsoft Azure law firm workflow uses sequential pipelines for contract generation: draft → review → redline → final.

How It Fails

Error propagation. Bad output in stage 1 cascades downstream with no backtracking. A hallucination in the research stage produces a flawed draft, which the editor polishes into a confident but incorrect final output.

Coordination overhead. A 4-agent pipeline adds approximately 950ms of coordination overhead versus 500ms of processing time. You're paying 3x for the same result if specialization isn't required. Token consumption compounds: 29,000 tokens across a 4-agent pipeline versus 10,000 for a single agent doing the same work.

No conditional branching. The pipeline cannot adapt based on intermediate results. If stage 2 discovers the input is malformed, it has no mechanism to signal stage 1 to retry — it must either fail or produce degraded output.

Mitigations

Insert quality gates between stages (lightweight validation agents that check output before passing downstream)
Add reprocessing loops for stages that can retry — durable workflow engines such as Temporal handle retry semantics reliably
Keep pipelines to 3-4 stages maximum; beyond that, consider orchestrator-worker for conditional branching

Pattern 3: Fan-Out / Fan-In

How It Works

Fan-Out / Fan-In is parallel execution with aggregation. A dispatcher routes work to multiple agents running simultaneously, then a collector aggregates their results via voting, weighted merging, or LLM synthesis. Agents operate independently throughout execution and do not communicate with each other — the only shared boundary is the collector.

graph TD
    D[Dispatcher] --> AA[Agent A]
    D --> AB[Agent B]
    D --> AC[Agent C]
    AA --> C[Collector<br/>merge]
    AB --> C
    AC --> C

When to Use It

Multi-perspective analysis where diverse viewpoints are valuable
Concurrent code review (multiple reviewers in parallel)
4+ independent tasks that can be decomposed upfront
Workloads where wall-clock time matters more than token efficiency

Key metric: Fan-out cuts wall-clock time by 75% compared to sequential execution. Four agents running in parallel complete in the time of one.

How It Fails

API rate limits. Collective load exceeds capacity even if individual agents stay within limits. Five agents each making 10 requests per minute may exceed a 40 RPM limit that a single agent respects.

Quadratic race conditions. Shared state conflicts scale as N(N-1)/2. With 5 agents, that's 10 potential conflicts. With 10 agents, it's 45. State management becomes the dominant complexity.

Aggregation hallucination. LLM synthesis can invent consensus. If Agent A says "yes" and Agent B says "no," the aggregator might produce "maybe" — a hallucinated middle ground that neither agent suggested. Requires explicit conflict resolution, not just summarization.

Mitigations

Use explicit voting mechanisms rather than freeform synthesis
Implement rate limiting at the dispatcher level
Maintain separate state per worker; merge at the collector
Set a maximum agent count (5-8) to keep race conditions manageable

Pattern 4: Hierarchical

How It Works

Hierarchical is tree-structured delegation with multiple levels — a top-level manager delegates to mid-level supervisors, which delegate to leaf-level workers. Each level adds a layer of abstraction: strategy at the top, tactics in the middle, and execution at the leaves. Context windows are managed at each level independently, so no single agent needs to hold the entire problem in context.

graph TD
    TM[Top Manager] --> SA[Supervisor A]
    TM --> SB[Supervisor B]
    TM --> SC[Supervisor C]
    SA --> W1[Worker 1]
    SB --> W2[Worker 2]
    SC --> W3[Worker 3]

When to Use It

Complex multi-domain enterprise tasks requiring 20+ agents
Large-scale codebase auditing where different modules need different specialists
Massive document processing (thousands of documents across multiple categories)
Tasks where no single agent's context window can hold the full problem

Key advantage: Hierarchical systems scale logarithmically. Each manager handles a bounded number of subordinates, so adding workers doesn't linearly increase coordination overhead.

How It Fails

Latency accumulation. Each level adds latency. A 3-level hierarchy requires at least 6-12 seconds minimum, accumulating per level. The top manager waits for all supervisors, who wait for all workers.

Information loss. Summarization between levels is lossy. A supervisor summarizes worker output for the top manager, losing details that might be critical for the final decision.

Branch failure isolation. A failure in one branch doesn't propagate to others — which is good for fault tolerance but bad for consistency. Different branches might reach contradictory conclusions that the top manager cannot resolve.

Mitigations

Set explicit summarization requirements for each level
Implement cross-branch validation at the top manager
Keep hierarchy depth to 2-3 levels maximum
Use structured outputs at every level to reduce information loss

Pattern 5: Swarm

How It Works

Swarm is decentralized emergent coordination with no central authority. Autonomous agents make local decisions based on shared state (a blackboard) or environment signals, with no orchestrator directing the flow. Agents discover available tasks, claim them, and publish results back to the shared space. Coordination is emergent — the system self-organizes around available work, similar to how bees navigate to a new hive without a central coordinator.

graph TB
    SB[Shared Blackboard<br/>tasks · results · observations]
    AA[Agent A] <--> SB
    AB[Agent B] <--> SB
    AC[Agent C] <--> SB
    AD[Agent D] <--> SB
    AE[Agent E] <--> SB
    AF[Agent F] <--> SB

When to Use It

Research flows where the optimal search path is unknown
Competitive intelligence gathering across multiple sources
Large-scale web scraping with dynamic target discovery
Parallel hypothesis exploration in scientific or analytical domains

Key advantage: A swarm of 50 research agents can explore 50 hypotheses in parallel without any central coordinator planning the search. The system self-organizes around available work.

How It Fails

Debugging nightmare. Without a central control flow, tracing failures requires distributed tracing and blackboard replay. You cannot follow a single execution path — you must reconstruct the emergent behavior from logs.

No transactional guarantees. Swarm patterns cannot enforce strict ordering or transactional consistency. If you need Agent A to complete before Agent B starts, a swarm is the wrong pattern.

Termination conditions. How does the swarm know when to stop? Without explicit termination criteria, agents may continue indefinitely, consuming compute and generating diminishing returns.

Mitigations

Implement explicit termination conditions (time-based, result-count-based, or convergence-based)
Use a blackboard with versioned entries to track state changes
Add a monitoring agent that observes swarm behavior and can intervene
Set agent-level budgets (maximum steps, maximum tokens) to prevent runaway execution — Kanban-style dispatchers provide practical rate-limit and concurrency patterns for self-hosted swarm deployments

Pattern 6: Mesh

How It Works

Mesh is direct peer-to-peer communication with persistent connections — agents communicate with each other through explicit, predefined channels rather than through any central hub. The communication graph is typically defined at deployment time, so Agent A knows it needs Agent B for database queries and Agent C for authentication logic. For cross-team or cross-system agent communication, the A2A protocol provides a standardized discovery and messaging layer for mesh participants that span different frameworks or ownership boundaries.

graph LR
    A[Agent A] --- B[Agent B]
    A --- C[Agent C]
    B --- C

When to Use It

Collaborative reasoning where agents need to share intermediate state
Multi-agent coding systems (planner ↔ coder ↔ tester loops)
Iterative artifact refinement where multiple specialists contribute
Negotiation scenarios where agents represent different stakeholders

Key advantage: Ideal for iterative refinement. Agents can pass partial results back and forth, building on each other's work without a central aggregator.

How It Fails

Combinatorial explosion. Connection count scales as N(N-1)/2. With 3 agents, that's 3 connections. With 8 agents, it's 28. Best limited to 3-8 tightly coupled agents.

Circular dependencies. Agent A calls Agent B, which calls Agent C, which calls Agent A. Without cycle detection, mesh patterns can enter infinite loops.

Debugging complexity. Non-deterministic routing makes tracing failures nearly impossible. When the output is wrong, you need to reconstruct which agents communicated with which, in what order.

Mitigations

Define the communication graph at deployment time (not runtime)
Implement cycle detection with maximum hop limits
Use message passing with explicit acknowledgment
Add a circuit breaker that terminates communication chains after N hops

The Decision Framework

Start with the simplest pattern that fits your problem. Most teams over-architect toward multi-agent topologies long before the single-agent approach has been genuinely exhausted.

Step 1: Characterize Your Problem

Problem Characteristic	Recommended Pattern
Known task decomposition, clear specialists	Orchestrator-Worker
Fixed sequence, no branching needed	Sequential Pipeline
Independent subtasks, need parallelism	Fan-Out / Fan-In
Complex, multi-domain, 20+ agents	Hierarchical
Exploration, unknown search space	Swarm
Collaborative refinement, peer communication	Mesh

Step 2: Estimate Your Constraints

Constraint	Pattern to Avoid
Low latency (< 2 seconds)	Hierarchical, Mesh
Strict ordering required	Swarm, Fan-Out
Single point of accountability	Swarm, Mesh
High fault tolerance needed	Orchestrator-Worker, Sequential
Budget-constrained	Fan-Out (parallel = more tokens)
Complex debugging required	Swarm, Mesh

Step 3: Start Single-Agent

The canonical agent loop — a single agent with tools, reasoning, and iteration — is still the right default for general-purpose agents. AI Assistant Architecture covers the five-layer foundation that single-agent systems build on, and it is worth mastering that foundation before layering in multi-agent coordination. Note that multi-agent systems are also fundamentally different from multi-model routing; for the latter, see Multi-Model System Design, which covers sequential, parallel, and ensemble patterns applied to model selection rather than agent coordination.

Escalate to multi-agent only when measurement says you must:

Single agent context window is insufficient
Task requires genuine parallelism (wall-clock time matters)
Specialization provides measurable quality improvement
Cost of single-agent approach exceeds multi-agent overhead

For background and proactive agent work — scheduling, queue-based execution, durable polling loops — see Polling Agents in AI Assistants: 11 Implementation Patterns, which complements multi-agent orchestration patterns with the scheduling layer underneath them.

Failure Modes: The MAST Taxonomy

Research from NeurIPS 2025 (MAST — Multi-Agent System Failure Taxonomy) analyzed 1,600+ execution traces across seven popular multi-agent frameworks. Failures distribute across three root categories:

1. Specification Ambiguity (33% of failures)

Agents misinterpret roles, duplicate work, or skip verification because their instructions are underspecified.

Fix: Use specification schemas. Define explicit role descriptions, task boundaries, and output formats for every agent. Structured schemas (JSON, Pydantic models) beat natural language instructions.

2. Coordination Breakdowns (33% of failures)

Agents communicate using unstructured protocols, leading to message loss, race conditions, and circular handoffs.

Fix: Implement structured coordination protocols. Use typed message passing, acknowledgment mechanisms, and explicit termination conditions.

3. Verification Gaps (33% of failures)

No independent validation of agent outputs. Agents trust each other's output without verification, allowing errors to propagate.

Fix: Add independent validation agents. Use a separate model or verification step to validate outputs before accepting them. This is the maker-checker pattern.

Cost Control: The Hidden Multiplier

Multi-agent systems have a cost structure that scales non-linearly:

Pattern	Cost Multiplier (vs single agent)
Orchestrator-Worker	2-3x (orchestrator + workers)
Sequential Pipeline	3-4x (each stage pays full token cost)
Fan-Out / Fan-In	4-5x (all agents run fully)
Hierarchical	3-5x (depends on depth)
Swarm	2-10x (depends on convergence)
Mesh	3-6x (depends on iteration count)

Cost optimization strategies:

Use cheaper models for workers. The orchestrator needs reasoning capability; workers can use smaller, faster models.
Bound execution budgets. Set maximum tokens, maximum steps, and maximum time per agent.
Implement early termination. Stop agents that have clearly failed or succeeded.
Cache shared context. Use prefix caching (vLLM, SGLang RadixAttention) to avoid recomputing shared system prompts.
Monitor per-agent cost. Track token consumption per agent, not just total cost. Identify the most expensive agents and optimize first.

For a deeper treatment of token optimization strategies — prompt compression, caching, batching, and smart model selection — see Reduce LLM Costs: Token Optimization Strategies. The techniques apply equally to individual agent calls within a multi-agent system.

Observability: Seeing Inside the Black Box

Multi-agent systems fail in ways that make traditional debugging inadequate. When multiple agents coordinate, issues propagate across agent boundaries, execution paths become unpredictable, and identifying root causes requires visibility into distributed workflows. Observability for LLM Systems covers the full production observability stack — metrics, distributed tracing, logs, SLOs, and tool comparisons — that multi-agent systems rely on. For instrumenting vLLM and llama.cpp inference endpoints with Prometheus and Grafana, see Monitor LLM Inference in Production.

Essential Observability Components

1. Distributed Tracing

Capture the complete interaction graph across all agents. Traditional tools show you whether components are running, but multi-agent debugging requires understanding how components interact and where coordination breaks down.

Key spans to trace:

Orchestrator decomposition step
Each worker's execution
Aggregation step
Cross-agent communication (mesh/swarm)

2. Blackboard Replay

For swarm and mesh patterns, maintain a versioned blackboard that can be replayed. This allows you to reconstruct the emergent behavior that led to a failure.

3. Cost Attribution

Track token consumption per agent, per step. Identify which agents are consuming disproportionate resources.

4. Convergence Monitoring

For swarm and mesh patterns, monitor whether the system is converging or diverging. Set alerts for:

Agent count exceeding expected bounds
Iteration count exceeding thresholds
Output quality degrading over time

Framework Support Matrix

Pattern	LangGraph	AutoGen	CrewAI	OpenAI Agents SDK
Orchestrator-Worker	✅ Native	✅ Native	✅ Native	✅ Native
Sequential Pipeline	✅ Graph edges	✅ Sequential	✅ Agent chains	✅ Handoff
Fan-Out / Fan-In	✅ Superstep	✅ Group chat	✅ Crew	✅ Parallel
Hierarchical	✅ Nested graphs	✅ Hierarchical	❌ Limited	❌ Limited
Swarm	❌ Limited	✅ Swarm	❌ No	❌ No
Mesh	✅ Custom graph	✅ Group chat	❌ No	❌ No

Putting It Together: A Production Example

Real-world systems rarely map cleanly to a single pattern — most production deployments combine two or three approaches, each handling the part of the workflow it is best suited for. Infrastructure patterns like Go Microservices for AI/ML Orchestration describe the service-level choreography and saga patterns that underpin these hybrid architectures at scale.

Consider a customer support system that handles technical inquiries:

Triage (Orchestrator-Worker): Incoming ticket → orchestrator classifies → routes to specialist
Research (Fan-Out): Specialist agent runs parallel queries (knowledge base, ticket history, product docs)
Draft (Sequential): Research → draft response → quality check
Escalation (Hierarchical): If quality check fails, escalate to senior agent → human review

This hybrid approach uses four patterns because no single pattern handles the full workflow optimally. The key insight: compose patterns, don't force one pattern to do everything.

Key Takeaways

Start simple. Single-agent with tools is the default. Escalate to multi-agent only when measurement demands it.
Match pattern to problem. Orchestrator-worker for decomposition, pipeline for fixed sequences, fan-out for parallelism, hierarchical for scale, swarm for exploration, mesh for collaboration.
Expect failure modes. Every pattern has specific ways it breaks. Design mitigations before you deploy.
Cost scales non-linearly. Multi-agent systems multiply token consumption. Budget for 2-5x the cost of a single agent.
Observability is non-negotiable. Without distributed tracing and cost attribution, you cannot debug or optimize multi-agent systems.
Compose patterns. Most production systems use 2-3 patterns combined. Don't force a single pattern to handle everything.

The multi-agent landscape is maturing rapidly. The teams that succeed are those that understand the tradeoffs, pick patterns deliberately, and build observability from day one.

Frequently Asked Questions

What is multi-agent orchestration?
Multi-agent orchestration is the coordination model that governs how multiple AI agents work together on a task. The pattern you choose — hub-and-spoke, pipeline, fan-out, hierarchical, swarm, or mesh — determines your system's latency, fault tolerance, scalability ceiling, and debugging complexity. Each pattern makes different tradeoffs and breaks in different ways.

Which multi-agent pattern is best for production AI systems?
Most production systems start with orchestrator-worker. It provides clear accountability, debuggable control flow, and predictable costs. Escalate to hierarchical when worker count exceeds 5-8 and to fan-out when independent parallel tasks dominate the workload. Swarm and mesh remain niche patterns reserved for exploration workflows and tight peer collaboration respectively.

Why do 40% of multi-agent pilots fail?
The three root causes according to the MAST taxonomy from NeurIPS 2025 are specification ambiguity (agents misinterpret roles or skip verification steps), coordination breakdowns (unstructured messaging leads to message loss and circular handoffs), and verification gaps (no independent validation of agent outputs, allowing errors to propagate unchecked). Each category accounts for roughly a third of all failures across 1,600+ analyzed execution traces.

How much more does a multi-agent system cost than a single agent?
Expect 2 to 10 times the token cost depending on the pattern. Orchestrator-worker is cheapest at 2-3x. Fan-out and swarm are most expensive at 4-10x because agents run in parallel and each consumes a full token budget independently. These multipliers compound at scale — a workflow costing $0.50 in testing can reach $50,000 per month at 100K executions.

How do you debug a multi-agent system when something goes wrong?
Start with distributed tracing — one trace per execution, with spans for each agent call, tool invocation, and aggregation step. For swarm and mesh patterns, implement blackboard replay so you can reconstruct the emergent behavior from logs. Per-agent cost attribution helps identify which agents are triggering cascading failures or runaway spend before they reach production scale.

Speculative Decoding: 20-50% Faster LLM Inference

Rost — Mon, 06 Jul 2026 07:11:15 +0000

A 70B model generates one token per forward pass, and each pass reloads weights from VRAM, computes attention across the context, and synchronizes memory. Between tokens, the GPU sits idle while it waits for sequential dependencies to resolve.

On an H100, a 70B model produces one token every 30-50ms. The GPU has enough compute capacity to process multiple tokens in parallel, but the sequential dependency prevents it — each token depends on the previous one, and the pipeline stalls.

Speculative decoding breaks that bottleneck by letting you generate multiple tokens in the time it normally takes to generate one, without changing the output distribution. The tokens you get are statistically identical to what you'd get from standard autoregressive decoding; the only difference is how fast you get them.

This guide covers the mechanics, the variants available in 2026, acceptance rate tradeoffs, and practical setup across llama.cpp, vLLM, SGLang, and TensorRT-LLM.

How Autoregressive Decoding Works (and Why It's Slow)

Before you can understand speculative decoding, you need to understand the autoregressive constraint it works around. Standard autoregressive generation processes tokens sequentially:

Run a forward pass through the model with the current context.
Sample the next token from the output distribution.
Append the token to the context.
Repeat.

Each step requires a full forward pass — loading weights from VRAM, computing attention across the entire context, and producing a single token. For a model with 70B parameters, this takes roughly 30-50ms per token on an H100. The GPU has compute capacity to spare — it could process more work in parallel — but the sequential dependency prevents it.

The Compute-VRAM Gap

Modern GPUs have more FLOPs than they need for single-token generation, so the real bottleneck is memory bandwidth — weights must be streamed from VRAM to the compute units for each forward pass. When generating one token at a time, the GPU spends most of its time waiting for memory transfers rather than doing useful compute.

Speculative decoding addresses this by giving the GPU more work per memory transfer. Instead of one token per forward pass, it generates K tokens per forward pass, amortizing the memory cost across multiple outputs.

The Draft-Verify Mechanism

Speculative decoding works in repeating draft-verify cycles. A fast draft mechanism proposes K candidate tokens — from a small draft model, an n-gram lookup, or a prediction head attached to the target model — and the target model verifies all K in a single forward pass. The draft phase is cheap, typically 5-20% of the target model's forward pass time, while verification compares each drafted token against what the target would have generated, accepting the longest matching prefix and resampling from the first rejection onward.

sequenceDiagram
    participant Draft as Draft mechanism
    participant Target as Target model
    Draft->>Draft: Propose K candidate tokens
    Draft->>Target: Send draft prefix for verification
    Target->>Target: Single forward pass over K positions
    alt Draft matches target distribution
        Target->>Target: Accept longest matching prefix
    else Draft diverges at position i
        Target->>Target: Accept tokens 1..i-1, resample from i
    end
    Target->>Draft: Append accepted tokens, start next cycle

Verifying K tokens costs roughly the same as generating one token autoregressively, so when the draft is correct you get K tokens for the price of one verification step.

A Concrete Example

Suppose the draft model proposes 5 tokens: ["I", " like", " cooking", " and", " traveling"]. The target model verifies them in a single forward pass:

Token	Draft	Target agrees?
1	"I"	✓
2	" like"	✓
3	" cooking"	✗ (target would say " playing")
4	" and"	— (not evaluated)
5	" traveling"	— (not evaluated)

The target accepts tokens 1 and 2, then generates " playing" for token 3, producing three tokens in one cycle instead of three separate forward passes. If the draft had been correct through token 5, you'd get five tokens for the cost of one verification — a 5x speedup on that cycle alone.

The Verification Bottleneck

In practice, verification dominates execution time — 42-95% of the cycle, depending on the method and model size. The target model's forward pass is the bottleneck, and rejected tokens represent wasted compute.

This is why acceptance rate matters so much. Every rejected token after the first is wasted verification work. The best speculative decoding methods maximize the expected accepted tokens per cycle, not just the raw acceptance rate.

The Mathematical Guarantee

One of the most important properties of speculative decoding is that it produces tokens from the exact same distribution as standard autoregressive sampling from the target model. The verification step uses rejection sampling — when the draft proposes token x, the target model computes its own probability p(x) and the draft computes p_draft(x). The acceptance probability is:

min(1, p(x) / p_draft(x))

When the target agrees (p(x) ≥ p_draft(x)), the token is always accepted. When the target disagrees, the token is accepted with probability proportional to the ratio, and rejected tokens are resampled from a residual distribution:

r(x) = max(0, p(x) - p_draft(x)) / Σ max(0, p(y) - p_draft(y))

This procedure guarantees that the output sequence follows the target model's distribution exactly, which is why speculative decoding is lossless. The draft model influences speed, not quality — the tokens you get are statistically indistinguishable from standard decoding, with the same perplexity and distribution. The only difference is latency.

Draft Model Strategies

The draft mechanism is the variable that matters most. Different approaches have different tradeoffs between setup complexity, acceptance rate, and speedup.

Standalone Draft Models

The simplest approach loads a smaller model alongside the target — typically a 1B-3B model drafting for a 7B-70B target.

Pros:

Conceptually straightforward
Works with any target model
Draft model can be tuned to match target's distribution

Cons:

Requires loading a second model into VRAM (1-4 GB depending on size)
Draft model quality directly determines acceptance rate
Cross-family drafts (e.g., Qwen drafting for Llama) typically perform poorly

Rule of thumb: Use models from the same family. Gemma 2 2B drafts well for Gemma 2 27B. Llama 3.2 1B drafts well for Llama 3.1 70B. Cross-family drafts tend to have low acceptance rates because the token distributions diverge.

Finding Compatible Draft Models

Not all small models work as draft models for a given target. The critical factor is distribution alignment — how closely the draft model's output probabilities match the target's.

Target Model	Recommended Draft	Family Match
Llama 3.1 70B	Llama 3.2 1B-3B	Same
Llama 3.1 8B	Llama 3.2 1B	Same
Qwen 3 27B	Qwen 3 0.6B-1.8B	Same
Gemma 2 27B	Gemma 2 2B	Same
Mixtral 8x7B	Phi-3 4B (trained on Mixtral data)	Cross (careful)

The golden rule: if the draft model's acceptance rate drops below 50%, speculative decoding may actually slow you down. The overhead of running the draft model plus verification outweighs the benefit when most proposals are rejected.

EAGLE and EAGLE-3: Prediction Heads

EAGLE (Efficient Architecture Guided Language Model Estimation) eliminates the need for a separate draft model. Instead, it attaches lightweight autoregressive prediction heads to the target model's internal layers.

How EAGLE Works

EAGLE trains prediction heads that take hidden states from the target model's intermediate layers and predict future tokens. During inference:

The target model runs a forward pass through its layers.
At each layer, the EAGLE head reads the hidden state and proposes tokens for future positions.
Multiple heads operate in parallel, each predicting a different future timestep.
The target model verifies all proposals in a single pass.

The advantage: EAGLE heads are trained specifically to match the target model's distribution. They see the target's internal representations directly, which gives them much better alignment than a standalone draft model.

EAGLE-3 Improvements

EAGLE-3 (2025) refines the approach with three key changes:

Layer selection: Instead of attaching heads to every layer, EAGLE-3 uses Bayesian optimization to select the optimal exit layer, reducing overhead.
Multi-token prediction: Each head predicts multiple tokens simultaneously, increasing the draft depth without proportional compute cost.
Training efficiency: EAGLE-3 trains on the target model's own generation data, improving acceptance rates on in-distribution workloads.

Acceptance rates: EAGLE-3 typically achieves 60-80% acceptance rates on in-distribution workloads, compared to 40-60% for standalone draft models. On code generation workloads with high repetition, acceptance can exceed 85%.

Setup: EAGLE-3 requires pre-trained heads for your target model. NVIDIA provides EAGLE-3 heads for several popular models through TensorRT-LLM and the Speculative Decoding Modules collection on HuggingFace. Third-party implementations exist for vLLM and SGLang.

P-EAGLE: Parallel Drafting (March 2026)

EAGLE-3's main limitation is autoregressive drafting — each draft token depends on the previous one, so generating K draft tokens requires K sequential forward passes through the draft head, and draft overhead grows linearly with K. P-EAGLE removes this ceiling by generating all K draft tokens in a single forward pass through a lightweight 4-layer drafter trained to predict up to 10 tokens in parallel.

The result: P-EAGLE delivers up to 1.69x speedup over vanilla EAGLE-3 on real workloads on NVIDIA B200. The advantage widens at higher K values — where EAGLE-3's sequential drafting becomes a bottleneck, P-EAGLE's parallel drafting incurs no additional cost.

Setup in vLLM: Download a pre-trained P-EAGLE head from HuggingFace, set "parallel_drafting": true in your vLLM config, and use the same --speculative-model flag — vLLM handles the rest. P-EAGLE is the current state-of-the-art for EAGLE-based speculative decoding, and if you're deploying EAGLE in 2026, P-EAGLE is the variant to use.

n-gram Speculative Decoding

n-gram speculative decoding replaces a neural draft with pattern matching against prompt history. The algorithm looks for repeated n-gram sequences in the context, and when the current token sequence matches a previously seen pattern, it proposes the tokens that followed that pattern earlier — for example, if the model has already generated def calculate_total(items): and encounters def calculate_total( again, it knows the next tokens will likely be items): based on the previous occurrence.

The n-gram map variants (ngram-map-k, ngram-map-k4v) use hash tables for faster lookups instead of linear scanning, with the hash key as the current n-gram of size N and the value as the token sequence that followed.

Pros:

Zero VRAM overhead — no additional model to load (~16 MB for the hash table)
Extremely fast for repetitive workloads (code editing, refactoring, template generation)
Acceptance rates can reach 90%+ on workloads with high self-similarity

Cons:

Useless for novel generation — if the pattern hasn't appeared before, n-gram has nothing to propose
Acceptance rate drops to near zero on creative or diverse workloads
Limited draft depth (typically 2-4 tokens per match)

Best for: Code refactoring, template filling, repetitive documentation, and any workload where the model revisits similar patterns. Worst for: creative writing, open-ended chat, and reasoning tasks.

Parameter Tuning

The n-gram parameters matter more than you'd expect. The defaults work for code, but text workloads need adjustment:

Parameter	Default	Code	Text	Notes
`size-n` (lookup length)	12	12-16	8-10	Longer n-grams reduce false positives but miss shorter patterns
`size-m` (draft length)	48	48	32	Longer drafts mean more tokens per match, but also more rejections
`min-hits`	1	1	2	Higher min-hits reduces false positives at the cost of fewer matches

For text workloads, reduce size-n to 8-10 and increase min-hits to 2. This trades off match frequency for higher acceptance rates per match.

Self-Speculative Decoding

Self-speculative decoding (also called LayerSkip or self-speculation) uses the model's own partial computation as the draft, so no separate model is needed.

How It Works

Instead of running the full model for each token, self-speculative decoding runs a truncated version — skipping some transformer layers — to generate draft tokens cheaply, and the full model then verifies the proposals.

For example, a 32-layer model might run with only 16 layers for drafting, then verify with all 32 layers. The truncated forward pass is faster because it processes fewer layers, and the draft tokens benefit from seeing the same initial layers as the target.

Pros:

No additional model weights to load
Naturally aligned with the target distribution (same architecture, partial layers)
Works well for models with significant redundancy in deeper layers

Cons:

Requires modifying the inference engine to support partial forward passes
KV cache complications — the draft uses partial KV cache that must be reconciled with the full model's cache
Acceptance rates are typically lower than EAGLE or well-tuned draft models

llama.cpp implementation: PR #18471 introduced self-speculative decoding using context history as draft. The model reuses tokens from its own generation history to propose continuations, particularly effective for coding workloads where patterns repeat within the same context window.

MTP (Multi-Token Prediction)

MTP is a specialized form of speculative decoding built directly into certain model checkpoints. Qwen 3.6 ships both standard and MTP-enabled GGUF variants.

How it differs: MTP heads are baked into the model architecture during training. The model carries extra prediction heads that propose multiple future tokens in a single forward pass. There's no separate draft model — the MTP heads are part of the target model itself.

Tradeoffs:

No draft model to manage — MTP is activated with --spec-type draft-mtp --spec-draft-n-max N
MTP heads add ~1-2 GB VRAM overhead
Works best on MoE architectures (Qwen 3.6 35B-A3B) where sparse routing keeps MTP heads cheap

For detailed benchmarks on MTP vs standard decoding across Qwen 3.6 27B and 35B, see Qwen 3.6 MTP vs Standard on 16GB GPU.

Acceptance Rates: What They Mean in Practice

Acceptance rate (α) is the single most important metric for speculative decoding performance. It determines whether you're getting a speedup or paying overhead.

The Speedup Formula

Expected accepted tokens per verification pass:

E[accepted] = α × K

Where K is the number of draft tokens proposed per cycle. If α = 0.7 and K = 5, you accept 3.5 tokens per pass — a 3.5x speedup over standard decoding (which produces 1 token per pass).

Acceptance Rate by Method

Method	Typical α Range	Best Workload
Draft model (same family)	40-60%	General chat, reasoning
Draft model (cross-family)	20-40%	Rarely recommended
EAGLE-3	60-80%	General workloads, code
P-EAGLE	65-85%	General workloads, deeper speculation
n-gram	10-90%+	Workload-dependent (high on repetitive, near zero on novel)
MTP	50-70%	Qwen 3.6 models specifically
Self-speculative	30-50%	Coding, repetitive patterns

When Acceptance Rate Drops

Acceptance rate is not constant across a generation. It varies by:

Token position: Early tokens tend to have higher acceptance (more context, less uncertainty). Later tokens drop as the model explores more diverse continuations.
Workload type: Code editing with repeated patterns sees α > 80%. Open-ended creative writing sees α < 40%.
Temperature: Higher temperature increases divergence between draft and target, lowering acceptance. Speculative decoding works best at low temperature (0.0-0.7).

Critical threshold: If your effective acceptance rate (α × K) drops below 1.0, speculative decoding is slower than standard decoding. The draft overhead plus verification time exceeds the cost of a single autoregressive step.

Speculative Decoding in Production: What Actually Happens

Research papers report 2-4x speedups, but production benchmarks tell a more nuanced story — speedups shrink with batch size, verification dominates cycle time, and no single method wins on every workload.

SpecDecode-Bench Findings (2026)

A systematic evaluation of five SD variants (n-gram, EAGLE, EAGLE-3, Draft-Model, MTP) on vLLM across four models and six workloads revealed:

SD works, but speedups shrink with batch size. At batch size 1, EAGLE achieves up to 1.96x on Llama-3-70B. By batch size 128, this drops to 1.21x. The system becomes compute-bound at high concurrency, and the GPU has less idle capacity to spare for speculation.
Verification dominates execution time (42-95%). The target model's forward pass is the bottleneck. Reducing wasted verification on rejected tokens is the most promising avenue for improvement.
No single method wins everywhere. EAGLE-3 is the best all-around choice. Draft-model methods excel when the target model is large (70B+). n-gram is optimal for code editing and high-overlap tasks.
Oracle analysis reveals a gap. The theoretical upper bound for combined n-gram + EAGLE strategies reaches ~4.9x on code editing workloads, but current implementations achieve 2-3x. There's room for optimization.

Practical Speedup Expectations

Scenario	Expected Speedup
70B model, single request, EAGLE-3	1.5-2.0x
70B model, batch 32, EAGLE-3	1.2-1.5x
8B model, single request, draft model	1.3-1.8x
Code editing, n-gram	2.0-4.0x (workload dependent)
Creative writing, any method	1.0-1.3x (often not worth it)
MTP on Qwen 3.6 27B, 16GB GPU	1.5-1.7x
P-EAGLE on B200, single request	2.0-3.0x

The batch size effect is critical. At small batches, the GPU has idle compute to spare for speculation. At large batches, the system is already saturated, and speculative decoding adds overhead without proportional benefit.

Monitoring in Production

You should track acceptance rate in production. A declining acceptance rate signals that your draft model is diverging from the target — either because the workload changed, or because the draft model needs retraining.

Key metrics to monitor:

Acceptance rate per request (should be stable around your baseline)
Tokens per second with vs without speculative decoding (the actual speedup)
Verification time as percentage of cycle time (should be 42-95%)
Draft model forward pass time (should be < 20% of target model time)

If your acceptance rate drops below 40%, disable speculative decoding for that request. The overhead is not worth it.

Practical Setup

Engine choice matters as much as draft strategy — see Ollama vs vLLM vs LM Studio and other local runtimes for how each runtime handles batching, API compatibility, and throughput before you pick a speculative decoding path.

llama.cpp

For general server setup and GGUF loading, start with the llama.cpp quickstart; the flags below add speculative decoding on top.

llama.cpp supports multiple speculative decoding methods through the --spec-type flag:

# Draft model (standalone)
llama-server \
  --model target-model.gguf \
  --draft-model draft-model.gguf \
  --spec-draft-n-max 4 \
  --parallel 1  # Mandatory: --parallel 1 for speculative decoding

# n-gram
llama-server \
  --model target-model.gguf \
  --spec-type ngram-simple \
  --spec-ngram-simple-size-n 12 \
  --spec-ngram-simple-size-m 48

# n-gram (text workload tuning)
llama-server \
  --model target-model.gguf \
  --spec-type ngram-simple \
  --spec-ngram-simple-size-n 8 \
  --spec-ngram-simple-size-m 32 \
  --spec-ngram-simple-min-hits 2

# MTP (Qwen 3.6)
llama-server \
  --model Qwen3.6-27B-MTP.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2

# Self-speculative (coding workloads)
llama-server \
  --model target-model.gguf \
  --spec-type draft-self

Critical flags:

--parallel 1 — Speculative decoding in llama.cpp requires single-batch mode. This is a current limitation.
--spec-draft-n-max — Number of draft tokens per cycle. Start with 3-5; higher values increase VRAM pressure.
--spec-ngram-simple-size-n — Lookup n-gram length. Default 12 works well for code; reduce to 8 for text.

Common pitfalls:

Forgetting --parallel 1 — the server will silently ignore speculative decoding.
Using cross-family draft models — acceptance rates collapse, negating any speedup.
Setting --spec-draft-n-max too high — each extra draft token costs VRAM for the draft buffer. Diminishing returns kick in around 5-8.

vLLM

The vLLM quickstart covers base deployment; the flags below enable speculative decoding on an existing vLLM server.

vLLM supports speculative decoding through the --speculative-model and --speculative-num-steps flags:

# Draft model
vllm serve target-model \
  --speculative-model draft-model \
  --speculative-num-steps 5 \
  --speculative-accept-length 5

# EAGLE-3
vllm serve target-model \
  --speculative-model EAGLE-target-model/ \
  --speculative-num-steps 7 \
  --speculative-draft-tensor-parallel-size 1

# P-EAGLE (parallel drafting)
vllm serve target-model \
  --speculative-model P-EAGLE-target-model/ \
  --speculative-num-steps 7 \
  --speculative-parallel-drafting true

# n-gram
vllm serve target-model \
  --speculative-method ngram \
  --speculative-num-steps 5 \
  --ngram-context-size 12

vLLM's speculative decoding is integrated with continuous batching, so it works under concurrent workloads. The scheduler handles multiple token slots within a single forward pass, and the memory manager handles KV cache for both draft and target models.

SGLang

SGLang supports speculative decoding through its --speculative-algorithm flag:

python -m sglang.launch_server \
  --model-path target-model \
  --speculative-algorithm ngram \
  --ngram-context-size 12 \
  --ngram-max-candidate-tokens 6

SGLang's RadixAttention architecture pairs well with speculative decoding because prefix caching reduces verification cost — the target model reuses cached attention for shared prefixes, making each verification pass cheaper than a cold forward pass.

TensorRT-LLM

TensorRT-LLM provides production-grade speculative decoding with Triton Inference Server. The setup is more involved but offers the best performance on NVIDIA hardware:

Build the TensorRT engine for both target and draft models.
Configure the model repository with model.yaml specifying the speculative decoding configuration.
Launch Triton with the LLM API / PyTorch backend.

TensorRT-LLM supports both draft-model and EAGLE-3 variants. For code generation workloads, TensorRT-LLM with n-gram speculative decoding has demonstrated 2-3x latency reduction in production deployments.

When to Use Speculative Decoding

Use It When

Large target models (7B+): The overhead of the draft mechanism is amortized across the target's compute. Speculative decoding shines when the target model is slow — the larger the target, the more valuable the speedup.
Low-temperature workloads: Speculative decoding works best at temperature 0.0-0.7, where the target model's distribution is concentrated and the draft has a better chance of matching.
Interactive applications: Latency-sensitive workloads (chat, code completion, agent tool calls) benefit most. Batch processing where you're already saturating the GPU sees less benefit.
Code generation and editing: High repetition in code patterns makes n-gram and self-speculative decoding particularly effective.

Skip It When

Small target models (< 3B): The draft model's overhead approaches the target's forward pass time. The speedup is marginal or negative.
High-temperature sampling: At temperature > 0.7, the target model's distribution is too broad for the draft to match reliably.
Creative writing and open-ended generation: Low acceptance rates on novel content make the overhead not worth it.
High batch sizes (> 32): The system becomes compute-bound, and speculative decoding adds overhead without proportional benefit. SpecDecode-Bench shows speedup dropping from 1.96x to 1.21x as batch size goes from 1 to 128.

Combining Methods

Advanced setups combine multiple speculative decoding strategies. The SpecDecode-Bench oracle analysis showed that adaptively combining n-gram and EAGLE can push speedup to 4.9x on code editing workloads.

The idea is to use n-gram for patterns that have appeared before, where acceptance is high and overhead is near zero, and fall back to EAGLE for novel tokens. In practice this requires engine support for multi-method speculation — vLLM and TensorRT-LLM have experimental support, but production-grade implementations are still maturing.

For now, the most practical combination is MTP + n-gram in llama.cpp. MTP handles the neural speculation, while n-gram catches repetitive patterns that MTP misses. On Qwen 3 27B, this combination achieves 120 tokens/sec compared to 67 tokens/sec standard — a 1.8x speedup.

Cost Considerations

Speculative decoding trades compute for latency. The total compute per token is roughly the same — you're just doing more work in parallel rather than sequentially.

GPU cost impact:

Single-request latency improves by 20-50%, which matters for interactive applications.
Throughput (tokens/sec across many requests) improves less — the GPU is already saturated at high batch sizes.
VRAM usage increases by the draft model's footprint (1-4 GB for standalone drafts, minimal for n-gram/EAGLE).

Cloud inference: At $2-4/hr per H100, speculative decoding reduces per-request latency without increasing per-token cost. For batch processing where you're already saturating the GPU, the cost benefit is minimal — you're paying for the same GPU time either way.

When speculative decoding saves money: Interactive applications where you charge per request and want to reduce time-to-first-token. A 2x speedup means your users wait half as long, and you can serve more requests per second on the same hardware.

When it doesn't: Batch processing where you're already maximizing GPU utilization. The extra compute from speculative decoding doesn't increase throughput — it just changes the latency profile.

What's Next

Speculative decoding is maturing from research novelty to production standard. The frontier is pushing beyond the current limitations:

Speculative Speculative Decoding (SSD): Parallelizes the drafting and verification stages across separate hardware. The draft model runs asynchronously, pre-speculating for multiple likely verification outcomes. Early results show up to 2x speedup over optimized speculative decoding, and 5x over autoregressive decoding. Not production-ready yet, but the direction is clear.
SpecSA (Sparse Speculative Verification): Combines speculative decoding with dynamic sparse attention. Turns sparse attention into a verification-oriented workload, achieving up to 3.49x end-to-end throughput over autoregressive sparse decoding. Relevant for long-context models where sparse attention is already in use.
Adaptive speculation: Automatically switching between n-gram, EAGLE, and draft-model methods based on workload characteristics. The oracle analysis shows significant untapped potential — current implementations achieve 2-3x, but the theoretical bound is 4.9x.
Multimodal speculative decoding: Extending draft-verify to vision-language models and video generation. Early surveys show the same principles apply, but verification strategies need adaptation for non-text modalities.

Decision Framework

Question	Answer	Recommendation
Target model size?	< 3B	Skip speculative decoding
Target model size?	7-13B	Use n-gram or self-speculative (low overhead)
Target model size?	30B+	Use draft model or EAGLE-3 (larger target = more benefit)
Workload type?	Code editing/refactoring	n-gram + EAGLE combination
Workload type?	General chat	EAGLE-3 or P-EAGLE
Workload type?	Creative writing	Skip speculative decoding
Batch size?	1-4 (interactive)	Speculative decoding helps most
Batch size?	32+ (throughput)	Speculative decoding helps less
Temperature?	0.0-0.7	Good for speculative decoding
Temperature?	> 0.7	Skip speculative decoding
Hardware?	16GB GPU	Use n-gram or MTP (low VRAM overhead)
Hardware?	24GB+ GPU	Draft model or EAGLE-3 feasible
Engine?	vLLM	EAGLE-3 or P-EAGLE (best integration)
Engine?	llama.cpp	n-gram or MTP (simplest setup)
Engine?	TensorRT-LLM	EAGLE-3 or draft model (production-grade)

Transactional Outbox Pattern in Go with PostgreSQL

Rost — Sun, 05 Jul 2026 08:48:20 +0000

Two writes that should succeed together will eventually fail separately.
Your order service saves the order to the database, then publishes an order.created event to a message broker.

These two operations run one after the other.

Between them, things go wrong: the broker is down, the network times out, the process restarts, or the container is evicted. The database write succeeded. The publish did not. The downstream service that needs to know about the new order never finds out. Nobody noticed until a customer called.

This is the dual-write problem, and it is one of the most common sources of silent data loss in distributed systems. The transactional outbox pattern is the standard fix.

The dual-write problem

The failure mode is easy to reason about once you see it:

BEGIN;
  INSERT INTO orders ...   -- succeeds
COMMIT;

PUBLISH order.created ...  -- fails, crashes, or is never reached

The database and the message broker do not share a transaction boundary. There is no rollback that covers both. Every service that does save -> publish in sequence has this gap. The pattern shows up in many forms:

db.Save(order) followed by events.Publish(OrderCreated{...})
HTTP handler that commits a transaction and then calls an external webhook
Worker that processes a record from one queue and writes results to another

The outcome in all cases is the same: one side succeeds while the other fails, and the system ends up in a state that is invisible to monitoring because both individual operations returned success at some point.

A retry loop does not fix this. Retrying the publish after the database commit only works if the retry itself is reliable -- which requires exactly the durability guarantee you do not have.

What the transactional outbox pattern does

The outbox pattern eliminates the gap by removing the direct publish entirely. Instead of calling the broker from within your business logic, you write an event record into an outbox table in the same database transaction as the business data. A separate background process -- the relay -- reads from the outbox table and publishes to the broker.

BEGIN;
  INSERT INTO orders ...         -- business data
  INSERT INTO outbox_events ...  -- event record
COMMIT;

-- Relay process (separately):
SELECT ... FROM outbox_events FOR UPDATE SKIP LOCKED;
PUBLISH order.created ...
UPDATE outbox_events SET processed_at = NOW() WHERE id = $1;

Both writes succeed or both fail. The transaction guarantee you already have from PostgreSQL now covers the event record too. The relay can retry publishing as many times as needed because the event sits in durable storage. If the relay crashes mid-flight, it restarts and retries. The worst outcome is that the event is published more than once -- which is handled by making consumers idempotent (see Idempotency in Distributed Systems).

PostgreSQL schema for the outbox table

The schema is intentionally simple:

CREATE TABLE outbox_events (
    id             UUID         PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id   VARCHAR(100) NOT NULL,
    event_type     VARCHAR(100) NOT NULL,
    payload        JSONB        NOT NULL,
    attempts       INT          NOT NULL DEFAULT 0,
    created_at     TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    processed_at   TIMESTAMPTZ
);

-- Partial index: only indexes unprocessed rows, stays small as rows are marked done
CREATE INDEX idx_outbox_unprocessed
    ON outbox_events (created_at)
    WHERE processed_at IS NULL;

The partial index on created_at WHERE processed_at IS NULL is important. Without it, the index grows with every event ever written and the relay's polling query gets slower over time. With it, the index covers only the pending rows, which in steady state is a small, bounded set regardless of how many events have been published.

Key field choices:

aggregate_type and aggregate_id describe which entity the event belongs to. Useful for ordering guarantees and routing.
event_type is the event name your consumers expect.
payload JSONB stores the event body. Use JSONB rather than TEXT so you can query it if needed.
attempts tracks how many times the relay has tried to publish this row. Used for retry limits and dead-letter handling.
processed_at is NULL for pending rows and set when the relay successfully publishes.

Writing business data and outbox event in one transaction

The business logic writes both records inside a single BeginTx / Commit call. There is no publish call here -- only database writes.

type OrderService struct {
    db *sql.DB
}

func (s *OrderService) CreateOrder(ctx context.Context, order Order) error {
    tx, err := s.db.BeginTx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback()

    if _, err := tx.ExecContext(ctx, `
        INSERT INTO orders (id, customer_id, total, created_at)
        VALUES ($1, $2, $3, NOW())
    `, order.ID, order.CustomerID, order.Total); err != nil {
        return fmt.Errorf("insert order: %w", err)
    }

    payload, err := json.Marshal(map[string]any{
        "order_id":    order.ID,
        "customer_id": order.CustomerID,
        "total":       order.Total,
    })
    if err != nil {
        return fmt.Errorf("marshal payload: %w", err)
    }

    if _, err := tx.ExecContext(ctx, `
        INSERT INTO outbox_events
            (aggregate_type, aggregate_id, event_type, payload)
        VALUES ($1, $2, $3, $4)
    `, "order", order.ID, "order.created", payload); err != nil {
        return fmt.Errorf("insert outbox event: %w", err)
    }

    return tx.Commit()
}

If tx.Commit() fails, neither the order row nor the outbox row is persisted. If it succeeds, both are guaranteed to be in the database. The relay can publish the event at any point after that -- immediately, in one second, or after the relay restarts following a crash.

This is the only code change required in your business layer. The rest of the pattern lives in the relay.

Go relay implementation

The relay is a background worker that polls the outbox table on a timer. It fetches a batch of unprocessed rows, publishes each one, and marks it done. Keep it in the same binary as your application or run it as a separate process -- either works, but the same binary is simpler to operate.

type OutboxRelay struct {
    db          *sql.DB
    publisher   Publisher
    logger      *slog.Logger
    batchSize   int
    pollInterval time.Duration
    maxAttempts  int
}

func (r *OutboxRelay) Run(ctx context.Context) error {
    ticker := time.NewTicker(r.pollInterval)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-ticker.C:
            if err := r.processBatch(ctx); err != nil {
                r.logger.Error("outbox relay batch failed", "err", err)
            }
        }
    }
}

The relay respects context cancellation, which makes it straightforward to integrate with graceful shutdown. For a detailed treatment of context lifetime and cancellation patterns, see Go context.Context Done Right.

FOR UPDATE SKIP LOCKED: the concurrent worker pattern

The processBatch function uses FOR UPDATE SKIP LOCKED to safely handle concurrent relay workers:

func (r *OutboxRelay) processBatch(ctx context.Context) error {
    tx, err := r.db.BeginTx(ctx, nil)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback()

    rows, err := tx.QueryContext(ctx, `
        SELECT id, aggregate_type, aggregate_id, event_type, payload
        FROM outbox_events
        WHERE processed_at IS NULL
          AND attempts < $1
        ORDER BY created_at
        LIMIT $2
        FOR UPDATE SKIP LOCKED
    `, r.maxAttempts, r.batchSize)
    if err != nil {
        return fmt.Errorf("query outbox: %w", err)
    }
    defer rows.Close()

    type row struct {
        id            string
        aggregateType string
        aggregateID   string
        eventType     string
        payload       json.RawMessage
    }

    var batch []row
    for rows.Next() {
        var e row
        if err := rows.Scan(
            &e.id, &e.aggregateType, &e.aggregateID, &e.eventType, &e.payload,
        ); err != nil {
            return fmt.Errorf("scan row: %w", err)
        }
        batch = append(batch, e)
    }
    if err := rows.Err(); err != nil {
        return err
    }

    for _, e := range batch {
        if err := r.publisher.Publish(ctx, e.eventType, e.aggregateID, e.payload); err != nil {
            r.logger.Error("publish failed", "event_id", e.id, "err", err)
            if _, err := tx.ExecContext(ctx,
                `UPDATE outbox_events SET attempts = attempts + 1 WHERE id = $1`, e.id,
            ); err != nil {
                r.logger.Error("increment attempts failed", "event_id", e.id, "err", err)
            }
            continue
        }

        if _, err := tx.ExecContext(ctx,
            `UPDATE outbox_events SET processed_at = NOW() WHERE id = $1`, e.id,
        ); err != nil {
            return fmt.Errorf("mark processed: %w", err)
        }
    }

    return tx.Commit()
}

FOR UPDATE SKIP LOCKED does two things. First, FOR UPDATE locks the selected rows for the duration of the transaction, preventing any other transaction from selecting them. Second, SKIP LOCKED means that if a row is already locked by another transaction, the query skips it instead of waiting. The result is that multiple relay workers can run in parallel and each will pick up a non-overlapping subset of rows.

Without SKIP LOCKED, a second worker would block until the first transaction commits before seeing the same rows -- at which point they would already be marked done. With SKIP LOCKED, the second worker immediately picks up different rows instead of waiting, giving you safe horizontal scaling.

Note the scan-then-publish separation in the code above: all rows are scanned into a slice before the publish loop starts. This avoids holding an open *sql.Rows cursor across network calls to the broker, which would hold the transaction open longer than necessary.

Idempotency and deduplication

The relay publishes at least once. If it publishes an event and then crashes before committing the processed_at update, it will publish the same event again on restart. This is unavoidable -- exactly-once delivery across a database and a message broker without a distributed transaction coordinator requires this trade-off.

Consumers must be idempotent. The simplest approach is to track processed event IDs in a processed_events table:

CREATE TABLE processed_events (
    event_id   UUID PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

func (h *OrderHandler) HandleOrderCreated(ctx context.Context, eventID string, payload []byte) error {
    // Deduplicate using the event ID as the natural key
    _, err := h.db.ExecContext(ctx, `
        INSERT INTO processed_events (event_id) VALUES ($1)
        ON CONFLICT (event_id) DO NOTHING
    `, eventID)
    if err != nil {
        return fmt.Errorf("dedup check: %w", err)
    }

    // Check if the insert actually happened (1 row) or was a no-op (0 rows)
    // A simpler approach: use RETURNING or check rows affected
    // If 0 rows affected, this is a duplicate -- skip it
    ...
}

In practice, many teams rely on the broker's own deduplication headers (such as Kafka's key field for log-compacted topics, or RabbitMQ's message-id header) and treat database-level deduplication as a fallback. Both are valid layers to apply.

Include the outbox event id (a UUID) in the published message as the deduplication key. Consumers can then use it regardless of which deduplication mechanism they prefer.

Retry policy and poison messages

The attempts column drives the retry policy. The relay skips rows where attempts >= maxAttempts and treats those rows as dead letters. A separate process or operator alert handles them.

A simple dead-letter view:

CREATE VIEW outbox_dead_letters AS
SELECT *
FROM outbox_events
WHERE attempts >= 5
  AND processed_at IS NULL
ORDER BY created_at;

A good production retry policy:

Set maxAttempts to 5-10 depending on how expensive retries are.
Consider exponential backoff: include a retry_after column and skip rows where retry_after > NOW().
Alert on COUNT(*) FROM outbox_dead_letters exceeding a threshold.
Provide a manual retry path: an admin endpoint or script that resets attempts = 0 and retry_after = NULL for specific rows.

Poison messages -- rows that consistently fail due to a bug in the consumer or a schema mismatch -- should not block healthy messages. Since the relay processes a batch per tick and marks failures with an attempt increment rather than removing them from the queue, healthy rows proceed normally while poisoned ones accumulate attempts until they hit the dead-letter threshold.

Event ordering and partitioning

The polling query orders by created_at, which gives first-in-first-out ordering within a batch. For most use cases that is enough. When strict per-entity ordering matters -- for example, ensuring that order.updated is never published before order.created for the same order -- you need per-aggregate ordering.

Add aggregate_id to the ORDER BY clause and use it as the message key when publishing to a partitioned topic like Apache Kafka. Kafka routes all messages with the same key to the same partition, and partitions are consumed in order. This gives you per-aggregate ordering guarantees without global ordering, which would require a single relay instance.

ORDER BY aggregate_id, created_at

For brokers that do not support partitioned ordering (such as basic AMQP queues), single-instance relay or application-level ordering checks in the consumer are the practical alternatives.

Reduce polling latency with LISTEN/NOTIFY

A polling interval of one second means average event latency of 500 milliseconds. For most workloads that is fine. For cases where you need near-zero latency, PostgreSQL's LISTEN/NOTIFY mechanism lets the relay wake up immediately when a new outbox row is inserted.

Add a trigger to the outbox table:

CREATE OR REPLACE FUNCTION notify_outbox_insert() RETURNS trigger AS $$
BEGIN
    PERFORM pg_notify('outbox_event', NEW.id::text);
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER outbox_insert_notify
AFTER INSERT ON outbox_events
FOR EACH ROW EXECUTE FUNCTION notify_outbox_insert();

In the relay, listen on the channel and wake up on notifications while still falling back to periodic polling:

func (r *OutboxRelay) Run(ctx context.Context) error {
    listener := pq.NewListener(r.dsn, 10*time.Second, time.Minute, nil)
    defer listener.Close()

    if err := listener.Listen("outbox_event"); err != nil {
        return fmt.Errorf("listen: %w", err)
    }

    ticker := time.NewTicker(5 * time.Second) // fallback poll
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-listener.Notify:
            if err := r.processBatch(ctx); err != nil {
                r.logger.Error("outbox batch failed (notify)", "err", err)
            }
        case <-ticker.C:
            if err := r.processBatch(ctx); err != nil {
                r.logger.Error("outbox batch failed (poll)", "err", err)
            }
        }
    }
}

The fallback ticker handles any notifications missed during a relay restart or network hiccup. Keep the fallback interval at a few seconds rather than milliseconds -- its job is recovery, not low latency.

Observability: metrics, logs, and alerts

The outbox is infrastructure. Treat it like infrastructure and instrument it accordingly.

Key metrics:

var (
    outboxPublished = prometheus.NewCounter(prometheus.CounterOpts{
        Name: "outbox_events_published_total",
        Help: "Total outbox events successfully published.",
    })
    outboxFailed = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "outbox_events_failed_total",
        Help: "Total outbox publish failures by event type.",
    }, []string{"event_type"})
    outboxPending = prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "outbox_events_pending",
        Help: "Current number of unprocessed outbox events.",
    })
    outboxBatchDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
        Name:    "outbox_batch_duration_seconds",
        Help:    "Duration of each outbox processing batch.",
        Buckets: prometheus.DefBuckets,
    })
)

Gauge refresh: run a periodic query to keep outbox_events_pending accurate:

SELECT COUNT(*) FROM outbox_events WHERE processed_at IS NULL;

Alert thresholds to consider:

outbox_events_pending > 1000 for more than two minutes: relay is falling behind or stuck.
outbox_events_pending growing monotonically: broker is down or relay has crashed.
Dead-letter count non-zero: schema or consumer bug needs investigation.
outbox_batch_duration_seconds p95 > 5s: database is slow or batch size is too large.

Structured log fields: include event_id, event_type, aggregate_id, and attempt in every log line from the relay. These fields let you correlate a failed publish with the specific outbox row and the downstream consumer trace.

Outbox vs. direct queue vs. saga

The outbox pattern is not the right tool for every coordination problem. Here is the comparison:

Approach	Atomicity	Complexity	When to use
Direct publish	None	Low	Acceptable to occasionally lose events
Transactional outbox	Strong	Medium	Reliable event delivery from a single service
Saga pattern	Eventual	High	Multi-service transactions that span multiple databases
Two-phase commit	Strong	Very high	Rarely practical; avoided in most distributed systems

The outbox pattern guarantees that a single service reliably emits events that reflect its own state changes. It does not coordinate state changes across multiple services -- that is what the Saga pattern is for. The choice of broker -- whether RabbitMQ, SQS, or Kafka -- is independent of the outbox pattern itself; the relay publishes to whichever broker your system uses.

If you are building a saga, the outbox pattern is still useful: each participant in the saga writes its local state change and its saga event in one transaction using the outbox, then the saga orchestrator or choreography reads those events reliably.

WAL-based CDC as an alternative relay

Instead of polling, you can tail PostgreSQL's Write-Ahead Log (WAL) and read outbox inserts directly from the replication stream. Tools like Debezium do this. The advantages are lower latency and no lock pressure on the outbox table. The disadvantages are operational complexity, a dedicated PostgreSQL replication slot, and an external service to run and monitor.

For most teams, the polling relay described above is the right starting point. WAL tailing makes sense when you have high outbox insert rates (tens of thousands per second), need sub-100ms event latency, or are already running Debezium for other change-capture needs.

sqlc integration

If you use sqlc for type-safe Go database code, the outbox queries fit naturally:

-- name: InsertOutboxEvent :exec
INSERT INTO outbox_events (aggregate_type, aggregate_id, event_type, payload)
VALUES (@aggregate_type, @aggregate_id, @event_type, @payload);

-- name: FetchOutboxBatch :many
SELECT id, aggregate_type, aggregate_id, event_type, payload
FROM outbox_events
WHERE processed_at IS NULL
  AND attempts < @max_attempts
ORDER BY created_at
LIMIT @batch_size
FOR UPDATE SKIP LOCKED;

-- name: MarkOutboxProcessed :exec
UPDATE outbox_events SET processed_at = NOW() WHERE id = @id;

-- name: IncrementOutboxAttempts :exec
UPDATE outbox_events SET attempts = attempts + 1 WHERE id = @id;

-- name: OutboxPendingCount :one
SELECT COUNT(*) FROM outbox_events WHERE processed_at IS NULL;

sqlc generates type-safe functions for each query, which avoids string interpolation errors and keeps the outbox query logic co-located with the rest of your database access layer.

Production checklist

Use this before shipping an outbox implementation:

Database

[ ] Outbox table has the partial index on created_at WHERE processed_at IS NULL
[ ] attempts column present with a default of 0
[ ] Dead-letter view or query defined
[ ] Old processed rows are periodically archived or deleted (a nightly cleanup job suffices)

Relay

[ ] FOR UPDATE SKIP LOCKED used in the polling query
[ ] Relay runs inside a transaction (begin before query, commit after all updates)
[ ] Batch size is bounded (50-200 rows is typical)
[ ] Relay respects context cancellation for graceful shutdown
[ ] Failed publishes increment attempts rather than causing the batch to abort

Idempotency

[ ] Published message includes the outbox id as a deduplication key
[ ] Consumers are idempotent or the broker provides deduplication
[ ] See Idempotency in Distributed Systems for deduplication patterns

Observability

[ ] outbox_events_pending gauge is monitored and alerted on
[ ] Dead-letter count is alerted on
[ ] Relay batch duration is tracked
[ ] Structured logs include event_id, event_type, and aggregate_id

Operations

[ ] Manual retry path exists for dead-letter rows
[ ] Relay restart behavior is tested (does it re-publish correctly?)
[ ] Broker outage behavior is tested (does the outbox grow and drain correctly?)

Final thoughts

The dual-write problem is easy to dismiss as an edge case until it causes an incident. The transactional outbox pattern solves it with tools you already have: a PostgreSQL transaction, a background goroutine, and one extra table. The relay is simple to build, simple to operate, and simple to reason about.

The cost is that consumers must be designed for at-least-once delivery. That is a reasonable trade-off. Exactly-once delivery across a database and a broker without distributed transactions is not achievable in practice -- and pretending otherwise leads to systems that silently drop or double-process events under failure conditions.

Write the event with the data. Relay it reliably. Make consumers idempotent. That is the whole pattern.

This article is part of the App Architecture in Production cluster.

Sources

Go context.Context Done Right: Cancellation, Timeouts, and Values

Rost — Sun, 05 Jul 2026 00:39:11 +0000

Go's context.Context is simple enough to use badly — and that is the problem.

Most Go developers learn the surface rules quickly: pass context as the first argument, check ctx.Done(), use context.WithTimeout, and never pass nil.

func DoSomething(ctx context.Context) error {
    // ...
}

Those rules are useful, but they cover the easy part. In production services, context is not just a parameter convention — it is the control plane for request lifetime.

Context tells work when to stop, how long it has left, which cancellation path was taken, and which request-scoped values need to travel across API boundaries. Used well, it prevents goroutine leaks, avoids wasted work, propagates deadlines, and makes services easier to shut down. Used badly, it becomes a bag of hidden dependencies, fake globals, forgotten timeouts, leaked timers, and confusing cancellation behavior.

The slightly opinionated version is this: use context for cancellation, deadlines, and request-scoped metadata, and do not use it as a dependency container.

What context is for

The context package has three main jobs — cancellation, deadlines and timeouts, and request-scoped values — and those three jobs cover everything it is designed to do.

A context should answer questions like:

Has this request been canceled?
How much time does this operation have left?
What request ID should be attached to logs?
Which authenticated user is associated with this request?

A context should not answer questions like:

Where is my database connection?
Where is my logger?
Where is my configuration?
Which service implementation should I use?

Those are dependencies — pass them explicitly through function parameters (see Dependency Injection in Go for patterns on doing this cleanly). Context is for request lifetime and request metadata, not application wiring.

The basic context shape

The core interface is small:

type Context interface {
    Deadline() (deadline time.Time, ok bool)
    Done() <-chan struct{}
    Err() error
    Value(key any) any
}

The important parts are:

Done() is closed when the context is canceled or its deadline expires.
Err() explains why the context ended.
Deadline() tells you whether the context has a deadline.
Value() stores request-scoped data.

Most code does not implement this interface. It receives a context and passes it down.

The first rule: pass context explicitly

For functions that do request-scoped or cancelable work, pass context as the first parameter — this is the standard Go convention and what every library and tool in the ecosystem expects:

func GetUser(ctx context.Context, id string) (*User, error) {
    // ...
}

Do this for functions that may:

Call a database
Call another service
Wait on a queue
Start background work
Block on I/O
Use a timeout
Need request-scoped values
Need cancellation

Do not add context to tiny pure functions that do not need it.

This is fine:

func NormalizeEmail(email string) string {
    return strings.ToLower(strings.TrimSpace(email))
}

Not every function needs a context. Adding context everywhere makes code noisy.

Do not store context in structs

Storing a context in a struct is one of the most common mistakes in Go codebases, and it is worth calling out explicitly. Do not do this:

type UserService struct {
    ctx context.Context
    db  *sql.DB
}

Do this instead:

type UserService struct {
    db *sql.DB
}

func (s *UserService) GetUser(ctx context.Context, id string) (*User, error) {
    // ...
}

A context belongs to a request, operation, or task, while a service struct usually lives much longer than any single request. Mixing those lifetimes makes cancellation unclear and makes it hard to reason about which operation a context belongs to.

There are rare exceptions for types that genuinely represent a single operation lifetime, but they are rare enough that the default rule should be simple:

Pass context. Do not store it.

Do not pass nil context

Never pass nil as a context.

Bad:

err := svc.DoWork(nil)

Use context.Background() when there is no existing context:

err := svc.DoWork(context.Background())

In tests, use the test context when possible:

func TestDoWork(t *testing.T) {
    err := svc.DoWork(t.Context())
    if err != nil {
        t.Fatal(err)
    }
}

A nil context can panic when code calls methods on it. A background context is explicit and safe.

Background, TODO, and request contexts

There are three common starting points.

context.Background

Use context.Background() at the top level of a program when no parent context exists — it is the root context from which all child contexts are derived:

func main() {
    ctx := context.Background()
    _ = run(ctx)
}

or:

func TestSomething(t *testing.T) {
    ctx := context.Background()
    _ = ctx
}

context.TODO

Use context.TODO() when you know a context should be used but have not decided which one yet.

ctx := context.TODO()

This is useful during migration, but it should not become permanent if a real context exists.

Request context

In HTTP servers, use the request context:

func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    _ = ctx
}

The request context is canceled when the client connection closes, the request is canceled, or the server finishes handling the request.

For web services, this is usually the context you should pass down to application code.

Cancellation with context.WithCancel

Use context.WithCancel when you want to stop work explicitly.

ctx, cancel := context.WithCancel(parent)
defer cancel()

The returned cancel function cancels the child context and releases resources associated with it. Always call it when you are done — even if the context will eventually time out, calling cancel early avoids keeping resources alive longer than necessary.

Example:

func RunWorker(parent context.Context) error {
    ctx, cancel := context.WithCancel(parent)
    defer cancel()

    done := make(chan error, 1)

    go func() {
        done <- doBackgroundWork(ctx)
    }()

    select {
    case <-ctx.Done():
        return ctx.Err()
    case err := <-done:
        return err
    }
}

The pattern is simple:

Derive a child context.
Defer cancel.
Pass the child context to work that should stop together.
Watch ctx.Done().

Timeouts with context.WithTimeout

Use context.WithTimeout when an operation has a maximum duration.

ctx, cancel := context.WithTimeout(parent, 2*time.Second)
defer cancel()

Example with an HTTP client:

func FetchUser(ctx context.Context, client *http.Client, url string) (*http.Response, error) {
    ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
    if err != nil {
        return nil, err
    }

    return client.Do(req)
}

This makes the timeout part of the operation, not a hidden global setting.

Always call cancel

When you call WithCancel, WithTimeout, or WithDeadline, always call the returned cancel function — this matters for correctness.

Good:

ctx, cancel := context.WithTimeout(parent, 5*time.Second)
defer cancel()

Bad:

ctx, _ := context.WithTimeout(parent, 5*time.Second)

Failing to call cancel can keep timers and child contexts alive longer than needed.

Deadlines vs timeouts

A timeout is relative:

ctx, cancel := context.WithTimeout(parent, 2*time.Second)
defer cancel()

A deadline is absolute:

deadline := time.Now().Add(2 * time.Second)
ctx, cancel := context.WithDeadline(parent, deadline)
defer cancel()

Most application code uses timeouts. Deadlines are useful when a request has a fixed end time that should be shared across multiple operations — for example, if a request has 900 milliseconds left, do not give each downstream call a fresh 1-second timeout; propagate the remaining budget instead.

Timeout budgets across service layers

A common mistake is stacking timeouts blindly.

func Handler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
    defer cancel()

    _ = service.DoWork(ctx)
}

func (s *Service) DoWork(ctx context.Context) error {
    ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
    defer cancel()

    return s.repo.Query(ctx)
}

This looks harmless, but it hides the real budget. The service layer should usually respect the caller's deadline instead of resetting the timer to the same value.

A better pattern is:

func Handler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
    defer cancel()

    if err := service.DoWork(ctx); err != nil {
        // handle error
        return
    }
}

Then inside the service:

func (s *Service) DoWork(ctx context.Context) error {
    return s.repo.Query(ctx)
}

Add a child timeout only when a sub-operation needs a smaller budget:

func (s *Service) DoWork(ctx context.Context) error {
    queryCtx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)
    defer cancel()

    return s.repo.Query(queryCtx)
}

The right mental model is straightforward: the whole request has one outer budget, specific sub-operations may have smaller budgets carved out of that budget, and no layer silently extends the request beyond what the caller intended.

Check ctx.Err() to distinguish cancellation from timeout

When a context ends, ctx.Err() returns the reason.

Usually it is one of:

context.Canceled
context.DeadlineExceeded

Example:

select {
case <-ctx.Done():
    return ctx.Err()
case result := <-resultCh:
    return handle(result)
}

This lets callers distinguish cancellation from timeout, and that distinction matters in practice. A canceled request often means the client disconnected, while a deadline-exceeded error usually means your service was too slow — they should not always be logged, retried, or reported the same way.

Use context.Cause for better cancellation reasons

Modern Go also supports cause-aware cancellation.

The useful functions include:

context.WithCancelCause
context.WithTimeoutCause
context.WithDeadlineCause
context.Cause

Plain ctx.Err() tells you the broad reason: canceled or deadline exceeded.

context.Cause(ctx) can tell you the more specific cause.

Example:

var ErrShutdown = errors.New("server shutting down")

func Run(ctx context.Context) error {
    ctx, cancel := context.WithCancelCause(ctx)
    defer cancel(nil)

    go func() {
        // Some shutdown signal arrived.
        cancel(ErrShutdown)
    }()

    <-ctx.Done()

    return context.Cause(ctx)
}

Use cause-aware cancellation when the reason matters to callers, logs, or cleanup behavior, and avoid it where a plain ctx.Err() is enough — the extra detail is only worth it when diagnosis genuinely requires it.

HTTP server example

A normal HTTP handler should start from r.Context(). For a full walkthrough of structuring Go HTTP services, see Building REST APIs in Go.

func GetUserHandler(svc *UserService) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()

        id := r.PathValue("id")

        user, err := svc.GetUser(ctx, id)
        if err != nil {
            writeError(w, err)
            return
        }

        writeJSON(w, http.StatusOK, user)
    }
}

The service should accept and propagate the context:

type UserService struct {
    repo *UserRepository
}

func (s *UserService) GetUser(ctx context.Context, id string) (*User, error) {
    return s.repo.GetUser(ctx, id)
}

The repository should use context-aware database methods:

type UserRepository struct {
    db *sql.DB
}

func (r *UserRepository) GetUser(ctx context.Context, id string) (*User, error) {
    const query = `
        select id, email, name
        from users
        where id = $1
    `

    var user User

    err := r.db.QueryRowContext(ctx, query, id).Scan(
        &user.ID,
        &user.Email,
        &user.Name,
    )
    if err != nil {
        return nil, err
    }

    return &user, nil
}

The important thing is the chain — each layer passes the same context down to the next:

flowchart TD
    A[HTTP request context] --> B[Handler]
    B --> C[Service]
    C --> D[Repository]
    D --> E[Database query]

Do not break the chain by creating context.Background() in the middle.

The context.Background() mistake: breaking the cancellation chain

This is a common bug:

func (s *UserService) GetUser(ctx context.Context, id string) (*User, error) {
    return s.repo.GetUser(context.Background(), id)
}

This discards all cancellation and deadline information from the caller. If the client disconnects, the database query keeps running. If the request times out, the downstream work may still be in flight. If the server is shutting down, this code ignores it entirely. Replacing the received context with context.Background() inside business logic is almost always wrong.

Use the context you were given:

func (s *UserService) GetUser(ctx context.Context, id string) (*User, error) {
    return s.repo.GetUser(ctx, id)
}

Only use context.Background() at the edge where no parent context exists.

HTTP client example

For outbound HTTP requests, attach the context to the request.

func CallAPI(ctx context.Context, client *http.Client, endpoint string) (*http.Response, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
    if err != nil {
        return nil, err
    }

    return client.Do(req)
}

Do not do this:

req, err := http.NewRequest(http.MethodGet, endpoint, nil)

That creates a request without the operation context.

Also avoid relying only on http.Client.Timeout. It can be useful as a safety limit, but request contexts give you better propagation across the call chain.

A common pattern is:

func CallAPI(ctx context.Context, client *http.Client, endpoint string) (*http.Response, error) {
    ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
    if err != nil {
        return nil, err
    }

    return client.Do(req)
}

Use this when the downstream API call has a specific budget inside a larger request.

Database example

Most Go database APIs have context-aware methods. For a broader look at how Go data access libraries handle context — including GORM, Ent, Bun, and sqlc — see Comparing Go ORMs for PostgreSQL.

Use them.

Good:

rows, err := db.QueryContext(ctx, query, args...)

Good:

err := db.QueryRowContext(ctx, query, id).Scan(&name)

Good:

result, err := db.ExecContext(ctx, query, args...)

Bad:

rows, err := db.Query(query, args...)

The context-aware forms allow database operations to stop when the request is canceled or times out, which is especially important for slow queries, overloaded databases, and user-facing APIs where latency directly affects user experience.

Transactions and context

Transactions need careful context handling.

A transaction should usually begin with the operation context:

tx, err := db.BeginTx(ctx, nil)
if err != nil {
    return err
}
defer tx.Rollback()

Then use the same context for transaction operations:

if _, err := tx.ExecContext(ctx, query, args...); err != nil {
    return err
}

if err := tx.Commit(); err != nil {
    return err
}

Be careful with timeouts around transactions. If the context is canceled before Commit, the transaction may be rolled back. That may be what you want, but it should be intentional.

For long transactions, the better answer is usually not a longer timeout — it is a shorter transaction that does less work per unit.

Background workers and context

Background workers should receive a context that represents their lifetime.

Example:

type Worker struct {
    logger *slog.Logger
}

func (w *Worker) Run(ctx context.Context) error {
    ticker := time.NewTicker(10 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()

        case <-ticker.C:
            if err := w.doOnce(ctx); err != nil {
                w.logger.Error("worker iteration failed", "err", err)
            }
        }
    }
}

This worker stops cleanly when the context is canceled, and its ticker is properly cleaned up via defer ticker.Stop(). In main, you would create a root context tied to OS signals:

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    worker := &Worker{logger: slog.Default()}

    if err := worker.Run(ctx); err != nil && !errors.Is(err, context.Canceled) {
        slog.Error("worker stopped", "err", err)
    }
}

This is context used correctly: it describes the lifetime of the process work, and when the OS sends a signal, the whole tree of goroutines that share this context will stop together.

Preventing goroutine leaks with context cancellation

A goroutine leak happens when a goroutine remains blocked forever after it is no longer useful.

Context helps prevent this.

Bad:

func StartWorker() {
    go func() {
        for {
            doWork()
            time.Sleep(time.Second)
        }
    }()
}

This goroutine has no shutdown path.

Better:

func StartWorker(ctx context.Context) {
    go func() {
        ticker := time.NewTicker(time.Second)
        defer ticker.Stop()

        for {
            select {
            case <-ctx.Done():
                return
            case <-ticker.C:
                doWork()
            }
        }
    }()
}

Any goroutine that loops should almost always have a cancellation path.

That does not mean every goroutine must receive context directly, but the system should have a clear way to stop it.

context.AfterFunc

context.AfterFunc runs a function after a context is canceled.

It can be useful for cleanup, unblocking operations, or bridging APIs that do not natively support context.

Example:

func waitWithContext(ctx context.Context, ch <-chan struct{}) error {
    stop := context.AfterFunc(ctx, func() {
        // Wake up or clean up if needed.
    })
    defer stop()

    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-ch:
        return nil
    }
}

Use AfterFunc carefully — it starts logic when cancellation happens, which can make control flow harder to follow. For most application code, a normal select on ctx.Done() is clearer and easier to reason about. AfterFunc is most valuable when you need to adapt context cancellation to an API that does not already accept context.

context.WithoutCancel

context.WithoutCancel creates a context that is not canceled when the parent is canceled.

This is useful, but it is also easy to misuse.

Example use case:

func Handler(audit *AuditLog) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()

        // Handle request...
        _ = ctx

        auditCtx := context.WithoutCancel(ctx)

        go func() {
            ctx, cancel := context.WithTimeout(auditCtx, 2*time.Second)
            defer cancel()

            _ = audit.Write(ctx, "request completed")
        }()
    }
}

The idea is that the audit write may need to continue briefly even after the request context is canceled. This should be rare and deliberate — do not use WithoutCancel as a way to avoid dealing with cancellation. Use it only when the child work genuinely must outlive the parent cancellation, and always add a new timeout: a context that ignores cancellation but carries no deadline can easily create background goroutine leaks.

Context values done right

Context values are for request-scoped data that crosses API boundaries.

Good examples:

request ID
trace ID
authenticated user ID
tenant ID
locale
security principal
correlation metadata

Bad examples:

database connection
logger as a hidden dependency
feature flags for ordinary control flow
optional function parameters
configuration
service clients

A useful rule: if the value is part of the request's identity or observability context, it may belong in context. If it is a dependency your code needs to do its job, pass it explicitly.

Use typed keys for context values

Do not use plain strings as context keys.

Bad:

ctx = context.WithValue(ctx, "userID", "123")

This can collide with other packages.

Use an unexported custom key type:

type userIDKey struct{}

func WithUserID(ctx context.Context, userID string) context.Context {
    return context.WithValue(ctx, userIDKey{}, userID)
}

func UserIDFromContext(ctx context.Context) (string, bool) {
    userID, ok := ctx.Value(userIDKey{}).(string)
    return userID, ok
}

This pattern gives you type safety at the package boundary, avoids key collisions with other packages, and keeps the context API surface clean with typed accessor functions.

Do not use context values for optional parameters

This is bad:

ctx = context.WithValue(ctx, "pageSize", 100)
users, err := repo.ListUsers(ctx)

This hides the function contract.

Prefer explicit parameters:

users, err := repo.ListUsers(ctx, ListUsersOptions{
    PageSize: 100,
})

Context values should not replace function arguments. Hidden input makes code harder to understand, test, and review — and anyone reading the function signature will have no idea the parameter even exists.

Logging and context

There are two common approaches to logging with context. The examples here use Go's log/slog package — for a deeper dive into structured logging with slog in production services, see Structured Logging in Go with slog.

Approach 1: Extract values and attach them to logs

func LogRequest(ctx context.Context, logger *slog.Logger, msg string) {
    if requestID, ok := RequestIDFromContext(ctx); ok {
        logger = logger.With("request_id", requestID)
    }

    logger.Info(msg)
}

This keeps the logger explicit as a proper dependency and uses context only for request-scoped values that legitimately need to cross API boundaries.

Approach 2: Store logger in context

Some codebases store a logger in context.

This can be convenient, but I do not recommend it as a default. It turns context into a dependency container.

My preference:

Pass logger dependencies explicitly.
Store trace IDs and request IDs in context.
Add those values to logs at boundaries or middleware.

This keeps dependencies visible.

Context and tracing

Tracing is one of the strongest use cases for context values, and it is a genuinely good fit. OpenTelemetry and similar systems use context to propagate trace spans across function calls and process boundaries, because trace data is exactly the kind of request-scoped metadata context was designed to carry.

A typical pattern looks like:

func (s *Service) DoWork(ctx context.Context) error {
    ctx, span := s.tracer.Start(ctx, "Service.DoWork")
    defer span.End()

    return s.repo.Query(ctx)
}

The context carries the active trace span, and the repository can create a child span from it. Each layer adds its own span without any explicit passing of tracer objects — the context does that work transparently across the entire call tree.

Error handling with context

When an operation stops because of context cancellation, preserve that information. The patterns here complement the broader error design strategies covered in Go Error Handling Architecture.

Example:

err := svc.DoWork(ctx)
if err != nil {
    if errors.Is(err, context.Canceled) {
        // Client canceled or caller stopped the work.
        return err
    }

    if errors.Is(err, context.DeadlineExceeded) {
        // Timeout.
        return err
    }

    return err
}

Do not blindly wrap context errors in a way that hides them.

Wrapping with %w preserves errors.Is, so callers can still detect cancellation or timeout:

if err != nil {
    return fmt.Errorf("query user: %w", err)
}

Replacing the error entirely discards that information and breaks any caller that checks for specific context error types:

if err != nil {
    return errors.New("query user failed")
}

Mapping context errors to HTTP responses

Context errors often map to different HTTP outcomes.

Example:

func writeError(w http.ResponseWriter, err error) {
    switch {
    case errors.Is(err, context.Canceled):
        // The client likely went away.
        // Some systems log this as a client closed request.
        return

    case errors.Is(err, context.DeadlineExceeded):
        http.Error(w, "request timed out", http.StatusGatewayTimeout)
        return

    default:
        http.Error(w, "internal server error", http.StatusInternalServerError)
        return
    }
}

Do not treat client cancellation as an application failure — if the user closed the browser tab, that is not your service misbehaving, and logging it as an error adds noise without signal.

Context in middleware

HTTP middleware is a common place to add request-scoped values.

Example request ID middleware:

type requestIDKey struct{}

func WithRequestID(ctx context.Context, requestID string) context.Context {
    return context.WithValue(ctx, requestIDKey{}, requestID)
}

func RequestIDFromContext(ctx context.Context) (string, bool) {
    requestID, ok := ctx.Value(requestIDKey{}).(string)
    return requestID, ok
}

func RequestIDMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        requestID := r.Header.Get("X-Request-ID")
        if requestID == "" {
            requestID = newRequestID()
        }

        ctx := WithRequestID(r.Context(), requestID)

        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

This is a good use of context. The request ID belongs to the request, it should travel through the full call chain, and attaching it to logs and traces at every layer is exactly the kind of cross-cutting observability concern that context values are designed to support.

Context in tests

In tests, avoid using context.Background() blindly.

Prefer t.Context() when the work belongs to the test lifetime:

func TestService(t *testing.T) {
    ctx := t.Context()

    err := service.DoWork(ctx)
    if err != nil {
        t.Fatal(err)
    }
}

For timeout behavior, test with a real timeout only if the timeout is small and meaningful.

For concurrent and time-dependent code, consider using testing/synctest — Testing Concurrent Go Code with synctest covers this tool in depth:

func TestTimeout(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
        defer cancel()

        time.Sleep(30 * time.Second)

        if !errors.Is(ctx.Err(), context.DeadlineExceeded) {
            t.Fatalf("got %v, want deadline exceeded", ctx.Err())
        }
    })
}

This lets you test real timeout values without waiting for real time.

Context and errgroup

For groups of goroutines that should cancel together, errgroup is often a good fit.

Example:

func FetchAll(ctx context.Context, ids []string, client *Client) error {
    g, ctx := errgroup.WithContext(ctx)

    for _, id := range ids {
        id := id

        g.Go(func() error {
            _, err := client.Fetch(ctx, id)
            return err
        })
    }

    return g.Wait()
}

If one goroutine returns an error, the group context is canceled and other goroutines that respect ctx.Done() can stop early. This is far cleaner than manually managing multiple goroutines, channels, and cancellation paths. The key phrase here is "respect the context" — errgroup cannot stop work that ignores ctx.Done().

Graceful shutdown

Context is central to graceful shutdown.

A typical server setup has:

a root context canceled by OS signals
an HTTP server
background workers
a shutdown timeout
cleanup logic

Example:

func main() {
    root, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    server := &http.Server{
        Addr:    ":8080",
        Handler: routes(),
    }

    go func() {
        <-root.Done()

        shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        defer cancel()

        if err := server.Shutdown(shutdownCtx); err != nil {
            slog.Error("server shutdown failed", "err", err)
        }
    }()

    if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
        slog.Error("server failed", "err", err)
        os.Exit(1)
    }
}

Notice that the shutdown context is not the same as the root context — the root is already canceled when the OS signal arrives. A separate timeout context gives the shutdown process a bounded amount of time to drain in-flight requests before force-quitting, which is the subtle but important distinction that makes graceful shutdown actually work.

Common anti-patterns

Anti-pattern 1: Using context as a dependency container

Bad:

ctx = context.WithValue(ctx, "db", db)
ctx = context.WithValue(ctx, "logger", logger)
ctx = context.WithValue(ctx, "config", cfg)

Pass dependencies explicitly.

Anti-pattern 2: Creating context.Background inside business logic

Bad:

func (s *Service) DoWork(ctx context.Context) error {
    return s.repo.Save(context.Background())
}

This breaks cancellation propagation.

Anti-pattern 3: Forgetting cancel

Bad:

ctx, _ := context.WithTimeout(parent, time.Second)

Good:

ctx, cancel := context.WithTimeout(parent, time.Second)
defer cancel()

Anti-pattern 4: Putting optional parameters in context

Bad:

ctx = context.WithValue(ctx, "includeDeleted", true)

Use explicit options structs.

Anti-pattern 5: Passing context too deep into pure code

Bad:

func Add(ctx context.Context, a, b int) int {
    return a + b
}

Pure computation does not need context unless it is long-running or cancelable.

Anti-pattern 6: Ignoring cancellation in loops

Bad:

for item := range items {
    process(item)
}

Better:

for item := range items {
    select {
    case <-ctx.Done():
        return ctx.Err()
    default:
    }

    if err := process(ctx, item); err != nil {
        return err
    }
}

Anti-pattern 7: Swallowing context errors

Bad:

if err != nil {
    return errors.New("operation failed")
}

Good:

if err != nil {
    return fmt.Errorf("operation failed: %w", err)
}

Preserve cancellation and deadline errors.

A practical context checklist

Use this checklist for Go backend code.

Function signatures

Context is the first parameter.
Context is not stored in long-lived structs.
Context is not passed to pure helper functions unless needed.
Nil context is never used.

Cancellation

Long-running loops check ctx.Done().
Goroutines have a shutdown path.
Worker lifetimes are tied to a parent context.
Context cancellation is propagated to downstream calls.

Timeouts

Outer request timeouts are set at the boundary.
Sub-operation timeouts are smaller than the outer budget.
Cancel functions are always called.
Timeouts are not blindly stacked at every layer.

Values

Context values are request-scoped.
Keys use custom types, not plain strings.
Dependencies are not stored in context.
Optional parameters are not stored in context.

Errors

context.Canceled and context.DeadlineExceeded are preserved.
Context errors are mapped correctly at API boundaries.
Cause-aware cancellation is used only when the reason matters.

Tests

Tests use t.Context() where appropriate.
Timeout tests avoid slow real sleeps.
Concurrent timeout behavior is tested with testing/synctest when useful.
Goroutine leaks are checked by ensuring shutdown paths exist.

How to audit context usage in a Go codebase

Search for these patterns:

grep -R "context.Background()" .
grep -R "context.TODO()" .
grep -R "WithTimeout" .
grep -R "WithCancel" .
grep -R "WithValue" .
grep -R "type .* struct" .

Then ask:

Is context.Background() only used at top-level boundaries?
Are cancel functions always called?
Are timeouts placed at sensible boundaries?
Are context values really request-scoped?
Are dependencies hidden in context values?
Are goroutines stoppable?
Are context errors preserved?

This is a good code review habit, because many context bugs are not syntax bugs — they are lifetime bugs that only surface under cancellation, load, or shutdown conditions.

My opinionated rules

These rules are boring, but they work.

Rule 1: Context is control flow

Use context to control cancellation, deadlines, and request metadata.

Do not use it to smuggle dependencies.

Rule 2: The caller owns the budget

A function should usually respect the context it receives.

Only create a shorter child timeout when the sub-operation needs a specific smaller budget.

Rule 3: Background belongs at the edge

Use context.Background() in main, tests, and top-level setup.

Do not use it inside service and repository methods to escape cancellation.

Rule 4: Values should be boring

Request ID, trace ID, user ID, and tenant ID belong in context. Database connections, loggers, config structs, and service clients do not — they are dependencies and should be passed explicitly.

Rule 5: Every goroutine needs a lifetime

If a goroutine starts, you should know exactly how it stops. Context is often the right answer, and if it is not context, there should be some other clear mechanism — a channel, a sync primitive, or an explicit signal.

Final thoughts

context.Context is not complicated because the API is large — the API is small. It is complicated because it represents lifetime, and lifetime is architecture. Every decision about where context flows, where it is derived, and where it stops is a decision about how your service handles failure, load, and shutdown.

A well-used context makes Go services easier to cancel, easier to shut down, easier to observe, and less likely to leak goroutines. A badly used context hides dependencies, discards deadlines, and makes code harder to reason about under pressure.

The practical takeaway is simple:

Pass context down.
Do not store it.
Do not replace explicit parameters with values.
Respect cancellation.
Use timeouts at boundaries.
Always call cancel.

That is Go context done right.

This article is part of the App Architecture in Production cluster, which covers code structure, data access, integration patterns, and testing architecture for production Go and Python systems.

Sources

What Is Spec-Driven Development? The Spec as Source of Truth

Rost — Sat, 04 Jul 2026 07:48:40 +0000

Spec-Driven Development is one of those ideas that software engineers have reached for before
and then set aside when the effort stopped paying.

What changed in 2025 is that AI coding
agents arrived and made the absence of explicit intent expensive. Prompts are ephemeral.
Agent sessions reset. Code changes but the reasoning behind it disappears. The spec is the
artifact that stops that from happening.

The Spec Is Becoming the Source of Truth

For most of software development history, the spec was either a temporary planning artifact
or an afterthought. Requirements lived in tickets, design decisions lived in chat threads,
and the code was the ground truth. Documentation described what existed after the fact.

Spec-Driven Development inverts that relationship. The specification becomes the primary
artifact. Code is what gets generated or verified against the spec, not the other way around.

This is not a new idea. Formal methods, design-by-contract, and BDD all contain versions of
it. What is new is the practical motivation: AI coding agents need explicit, durable context
to produce correct and consistent output. Prompts are too ephemeral. The spec is the only
artifact that can carry intent across agent sessions, across team members, and across time.

What Spec-Driven Development Actually Means

Spec-Driven Development, usually shortened to SDD, is a workflow where a versioned
specification guides or generates implementation. The specification is written and reviewed
before the agent writes code. It captures:

What to build -- user problem, goals, and non-goals
What correct behavior looks like -- acceptance criteria, edge cases, error states
How to build it -- architecture decisions, data model, API contracts, security constraints
How to verify it -- test strategy, validation rules, traceability back to requirements

The spec is not a one-time document. It is updated when reality differs from the design.
When the agent discovers something during implementation that the spec got wrong, the spec
is corrected before continuing. The spec stays honest because it is treated like code.

Recent academic work formalizes this framing: researchers describe SDD as treating
specifications as the source of truth and code as generated or verified against them. The
practical interpretation is that the spec is the reviewed, durable record of intent that
any human or AI tool can read and trust.

Three terms capture different points on the spec-use spectrum:

Spec-first means writing the full specification before any implementation begins. This
is the strictest interpretation and the one closest to waterfall if not done carefully.

Spec-anchored means keeping a specification in sync with implementation throughout the
feature lifecycle. The spec is updated as decisions change. This is the most practical
version for most teams.

Spec-as-source means generating or validating implementation from the spec, either
through AI agents or through tooling that checks code against spec constraints. This is the
direction tools like GitHub Spec Kit and Kiro are moving toward.

Why SDD Matters Now

The honest answer is that SDD is not compelling for a solo developer building a one-day
script. The overhead is not worth it.

SDD becomes valuable when three conditions are present: the feature is large enough to span
multiple sessions, the agent needs to make decisions that affect the architecture, and the
work will be reviewed or continued by someone else.

All three conditions are increasingly common with AI-assisted development.

LLMs need context, not just prompts. A model that receives a vague prompt makes vague
decisions. A model that receives a reviewed specification with explicit constraints, non-goals,
and acceptance criteria makes better decisions and is easier to course-correct when it drifts.
This connects to how retrieval and representation work: giving an agent a versioned spec is a form of structured retrieval of project intent.

Code generation is cheap; deciding what to build is still hard. The bottleneck in AI-assisted
development is no longer typing -- it is knowing what to build and how to constrain the
agent. SDD shifts the effort to where it matters: specifying intent clearly before generation
begins.

Prompts are ephemeral. The agent does not remember what you told it in the last session.
A versioned specification stored in the repository does. Every new session can read the same
spec and implement against the same intent without re-establishing context from scratch.

Vibe coding works until it does not. For prototypes and throwaway work, ad-hoc prompting is
faster and appropriate. As soon as a feature requires security constraints, multi-file
architecture decisions, or a team handoff, the absence of a spec becomes the main source of
drift and defects. See the comparison in SDD vs Vibe Coding for when each approach applies.

Core Artifacts

A practical SDD workflow produces four main artifacts. Each one reduces a specific kind of
ambiguity before it reaches the agent.

Requirements specification. The problem statement, the users affected, the goals, the
explicit non-goals, and the acceptance criteria. Non-goals are as important as goals -- they
tell the agent what not to build and prevent scope creep. Acceptance criteria are precise
enough that each one maps to at least one test.

Design specification. The architectural decisions relevant to this feature: affected
modules, data model changes, API contracts, migrations, security constraints, observability
requirements, and known failure modes. This is not a full system design document -- it is the
subset of architecture decisions needed to implement this feature correctly.

Task plan. A sequence of small implementation tasks, each with explicit dependencies,
expected file changes, and validation criteria. Tasks are small enough to implement in a
single agent session and verify with a focused diff. Each task has a human review checkpoint.

Traceability record. A mapping from acceptance criteria to tests, from design decisions
to affected files, and from tasks to commits. This is what separates SDD from documentation
that becomes stale: traceability makes it possible to verify that the spec was actually
implemented, not just written.

These artifacts do not have to be heavyweight. A simple feature might produce a single
two-page markdown document covering all four areas. The format matters less than the habit
of writing intent down before implementation begins.

How SDD Differs from Documentation

The most common confusion is treating SDD artifacts as documentation. They are not
documentation in the conventional sense.

Documentation describes. It tells you what the system does, how to use it, and what
it contains. It is written after the fact and updated when the system changes.

Specs constrain. A spec tells the agent what it is allowed to build and what it is
not allowed to do. It is authoritative before implementation begins. It is validated after
implementation completes. A spec that describes what was actually built -- rather than
constraining what should be built -- has already failed its purpose.

Executable specs guide generation and validation. The best SDD specs are close enough
to machine-readable that an agent can implement against them and a test suite can verify
them. Acceptance criteria written as "the endpoint must reject unauthenticated requests with
a 401 response" is an executable spec; "the endpoint is secure" is documentation.

Decision Records -- ADRs, PDRs, and DDRs -- are complementary to SDD artifacts but serve a different
purpose. Decision records capture why a choice was made and what was rejected. SDD
specifications capture what to build and how to verify it. Both belong in the repository.
Together they give AI agents the full picture: the current intent and the reasoning behind it.

How SDD Differs from TDD

Test-Driven Development and Spec-Driven Development are often confused because both produce
explicit artifacts before code exists. The difference is the starting point.

TDD starts with tests. You write a failing test that describes the behavior you want,
then write the minimum code to make it pass. TDD is a feedback loop at the unit level. It
produces good tests but does not answer the question of whether you are building the right
thing.

SDD starts with intent. Before tests exist, before the architecture is decided, the
spec answers: who has this problem, what does correct behavior look like, what is explicitly
out of scope. The spec then informs what tests to write, which is why good SDD and good TDD
are complementary rather than competing.

A practical way to think about it: SDD drives TDD. The acceptance criteria in the spec
become the test scenarios. The design spec identifies the integration boundaries that need
contract tests. The task plan identifies which unit behaviors need test coverage before the
agent implements them.

How SDD Differs from BDD

Behavior-Driven Development uses natural-language scenarios -- typically in Gherkin format --
to describe expected behavior from the user's perspective. These scenarios bridge the gap
between business intent and technical implementation.

SDD is broader. It includes behavior descriptions (which can use BDD-style language or plain
prose) but also covers architecture decisions, data models, security constraints, task
planning, and traceability. BDD can be a useful format for writing acceptance criteria inside
an SDD requirements spec. The spec is the container; BDD scenarios are one way to write what
goes inside it.

The distinction matters in practice: BDD tooling focuses on making scenarios executable.
SDD practice focuses on making intent durable -- across tools, across sessions, and across
team members.

How SDD Differs from Formal Methods

Formal methods use mathematical notation and automated verification to prove properties of
software systems. They are extremely rigorous and extremely expensive for most production
development contexts.

SDD does not require formal notation. A markdown file with acceptance criteria and architecture
decisions is a specification. It constrains without being mathematically formal. The level of
rigor scales with the stakes: a spec for a billing service should be more precise and more
carefully reviewed than a spec for a documentation page.

The relationship is a spectrum:

Informal prose spec (minimum viable SDD)
Structured markdown with acceptance criteria and non-goals
Machine-readable spec with schema validation
Contract tests derived directly from the spec
Formal spec with automated proof

Most teams operate in the middle of that spectrum. The goal is not mathematical rigor -- it
is making intent explicit enough that an AI agent can implement against it and a human
reviewer can verify the result.

Benefits of Spec-Driven Development

Less intent drift. The spec is the reference. When the agent drifts -- and it will -- the
reviewer has something to compare the implementation against. Without a spec, drift is
invisible until something breaks.

Better AI outputs. Agents given explicit constraints, non-goals, and acceptance criteria
produce implementations that are closer to what was intended and easier to correct when they
miss. Context quality directly determines output quality.

Easier review. A pull request attached to a spec is easier to review than a pull request
that requires the reviewer to reconstruct the intent from the code. The spec is the review
checklist.

Team alignment. When multiple people or agents are working on the same feature, the spec
is the shared contract. Without it, each contributor optimizes locally and the pieces may
not fit.

Better test planning. Acceptance criteria in the spec map directly to test cases. Test
coverage becomes a spec coverage question: is every acceptance criterion covered by at least
one test?

Durable handoff. When a feature changes hands -- between engineers, between agent sessions,
between sprints -- the spec is the handoff artifact. It captures what was decided, what was
out of scope, and what remains to be validated.

Costs of Spec-Driven Development

Upfront effort. Writing a good spec before writing any code takes time. For small
features, this overhead is real and sometimes not worth it.

False confidence. A spec that exists but is not validated against the implementation
gives a false sense of correctness. Stale specs are sometimes worse than no spec: they
mislead reviewers and agents that read them.

Stale specs. Specs drift when the team treats them as planning artifacts rather than
living documents. Updating the spec when implementation differs from design is not optional
-- it is what separates SDD from documentation that accumulates and rots.

Generated bureaucracy. AI agents can generate exhaustive task lists and verbose
specifications quickly. A 200-task spec generated in thirty seconds is not a useful spec --
it is a bureaucracy generator. Good SDD requires judgment about what to specify and what to
leave implicit.

Tool lock-in. Some SDD tools are opinionated about format, file structure, and workflow.
A spec written in a proprietary format is harder to carry across tools than a markdown file
with clear headers and acceptance criteria.

Conclusion

Spec-Driven Development is not a new methodology. It is an old discipline becoming practical
again because the cost of implicit intent is now visible in AI-generated code.

The discipline is simple: write down what you intend to build, reviewed and versioned, before
the agent builds it. Keep that record honest by updating it when reality differs. Use it as
the reference for review, testing, and handoff.

The spec is not magic. A spec that is not validated becomes the most expensive kind of
documentation: one that misleads confidently. Good SDD is the practice of keeping specs
honest -- small enough to maintain, precise enough to constrain, and durable enough to
outlast any single agent session.

Useful Links

Decision Records for AI-Driven Software Development -- ADRs, PDRs, and DDRs that complement SDD specs by capturing why decisions were made
Spec-Driven Development vs Vibe Coding: Waterfall? -- when to add specs and when to keep prompting freely
What is Vibe Coding -- Meaning, Tools, Benefits, and Risks -- the vibe coding cluster pillar
App Architecture in Production -- the cluster home for architecture, documentation, testing, and integration patterns
Unit Testing in Go: Structure and Best Practices -- turning SDD acceptance criteria into executable tests
Unit Testing in Python: Complete Guide -- test-writing practices that map to SDD acceptance criteria
Python Design Patterns for Clean Architecture -- code structure practices that SDD helps preserve
Retrieval vs Representation in Knowledge Management -- how explicit specs relate to AI context and retrieval
GitHub Spec Kit documentation -- a portable open-source SDD toolkit
Martin Fowler on Spec-Driven Development tools -- careful analysis of Kiro, Spec Kit, and Tessl

Decision Records for AI-Driven Software Development

Rost — Fri, 03 Jul 2026 09:10:32 +0000

Decision records are the missing memory layer in AI-assisted software development. They capture not just what was built, but why — and that distinction becomes critical when AI tools are writing your code.

Decision records are the missing memory layer

AI-driven programming changes the economics of software development by making code cheaper to generate, easier to refactor, and faster to throw away. That is useful. It is also dangerous, because when code becomes easier to produce, the scarce resource is no longer typing — the scarce resource is judgment.

Why did the team choose PostgreSQL instead of DynamoDB? Why does the product require human review before sending AI-generated emails? Why does the interface show suggestions in a side panel instead of applying them directly? Why was a simpler approach rejected six months ago? The code may show what exists, but it rarely explains why it exists.

Decision records solve this problem by providing a short, version-controlled document that captures an important choice, the context behind it, the alternatives considered, and the consequences the team accepted. In an AI-assisted codebase, these records become more than documentation — they become durable project memory that both humans and AI coding agents can read before making future changes. The practical operating rule is simple: keep decision records as Markdown files in the repository, review them like code, and let future AI tools read them before proposing or implementing changes.

What are decision records?

A decision record is a written record of a meaningful decision, structured to answer four basic questions: what did we decide, why did we decide it, what alternatives did we consider, and what consequences did we accept? The most common form is the Architecture Decision Record, shortened to ADR. ADRs are widely used to document technical decisions, and the same pattern can be extended beyond architecture into product and design work.

For AI-driven programming, three types are especially useful:

Record type	Captures	Example
ADR	Architecture and technical decisions	Use PostgreSQL as the primary database
PDR	Product behavior and scope decisions	AI-generated emails must remain drafts
DDR	Design and interaction decisions	Show AI suggestions in a side panel

Together, ADRs, PDRs, and DDRs describe not only the structure of the system but also the intent of the product and the reasoning behind the user experience. That combination matters because AI agents can read code, but code alone does not contain enough context to make good decisions. Decision records give AI systems a reviewed, durable, human-approved source of project intent.

Architecture Decision Records

Architecture Decision Records capture technical and structural decisions. Use an ADR when a decision affects the shape of the system — its boundaries, dependencies, operational model, or long-term maintainability.

Examples of decisions worth recording as ADRs include:

Choosing PostgreSQL as the primary database
Using an event-driven architecture for background processing
Keeping the application as a modular monolith
Introducing a message queue
Choosing REST instead of GraphQL
Using server-side rendering for the web application
Requiring all background jobs to be idempotent
Adopting a specific authentication and authorization model

An ADR is not a full architecture document — it is intentionally small, recording one important decision at a specific moment in time. A good ADR prevents architectural amnesia: without it, future contributors may rediscover the same tradeoffs, reopen old debates, or accidentally undo important constraints.

In AI-driven programming, ADRs carry even more weight. AI tools are often skilled at local optimization, and they may propose a technically plausible change that violates a larger architectural constraint. An ADR gives the AI a clear boundary: "This is how this system is supposed to be shaped."

Product Decision Records

Product Decision Records capture product behavior, scope, and user-facing intent. This is less common than ADRs, but it is often just as valuable — product decisions are frequently scattered across tickets, roadmap tools, chat threads, meeting notes, and people's memories, making them easy for humans to forget and almost impossible for AI tools to infer reliably.

Use a PDR when a decision affects what the product does, who it serves, what is intentionally out of scope, or how a user-facing feature should behave. Examples include:

AI-generated messages must remain drafts until reviewed by a human
Free-tier users can create up to three projects
Deleted workspaces are recoverable for 30 days
Team billing is out of scope for version 1
Users can export their data without contacting support
Low-confidence AI summaries show a warning instead of being hidden

A PDR is especially useful when a product choice looks arbitrary from the code. The code might contain a limit of three projects for free users, and without a PDR an AI tool may treat that number as a magic constant and suggest changing it. With a PDR, the AI can see that the limit is tied to pricing strategy, onboarding cost, or support load — and that changing it requires a deliberate product decision, not a quick edit.

Design Decision Records

Design Decision Records capture user experience, interaction, visual, and content-design decisions. Use a DDR when a decision affects how users interact with the product, how information is presented, or how a design principle should be applied across future work.

Examples of design decisions worth recording include:

Use inline validation instead of validation only on submit
Put AI suggestions in a side panel instead of directly in the editor
Use progressive disclosure for advanced settings
Require confirmation before destructive actions
Use "Draft" and "Published" instead of "Inactive" and "Active"
Keep primary actions visible on mobile screens

Design intent is easy to lose during implementation. A developer may simplify a flow, or an AI agent may generate a component that technically works but breaks the intended interaction model. For example, a DDR might record: "We show AI writing suggestions beside the document, not inside it, because users need to compare generated text with their own draft before accepting changes." That record gives future contributors a principle to preserve, not just a layout to copy.

Why decision records matter more with AI

AI coding tools are powerful, but they are often stateless or only partially aware of project history. They can inspect files, infer patterns, and generate changes — but they do not automatically know which decisions are intentional, which are accidental, and which were already debated and resolved. This creates several distinct risks.

AI may reopen settled debates

If the team already decided to use a modular monolith, an AI agent may still propose extracting a service because that looks clean in isolation. Without an ADR, the AI has no durable way to know that the team already considered and rejected that path, and the result is wasted effort or a subtle regression in system coherence.

AI may optimize locally and damage globally

A generated refactor may make one file cleaner while violating system boundaries. A UI change may reduce component complexity while weakening the intended user experience. A product change may simplify the implementation while breaking pricing or compliance assumptions. Decision records give the AI a larger frame of reference before it acts on narrowly scoped signals.

AI may preserve code but lose intent

A model can follow existing patterns in the codebase, but patterns are not the same as principles. Sometimes existing code is a compromise. Sometimes it is transitional. Sometimes it exists because of an external constraint that is not visible in the file. Decision records explain the difference between "this is how it works" and "this is why it was built this way."

AI may generate plausible but wrong rationale

AI can draft decision records, but it can also invent confident-sounding explanations that do not match the real decision. This is why human review is non-negotiable: AI may generate the first draft of a record, but a human must verify that it accurately describes the actual decision, alternatives, and consequences before the record is merged.

Decision records as part of a broader methodology

Decision records are not just documentation — they are part of a broader way of working that sits at the intersection of lightweight architecture governance, docs as code, AI-augmented knowledge management workflows, product discovery, design rationale, AI governance, and code review. A useful way to describe the larger process is Decision-Oriented Development.

Most AI-driven programming workflows focus narrowly on the generate-review-commit loop:

flowchart LR
    A[Prompt] --> B[Generate code]
    B --> C[Test]
    C --> D[Commit]

That cycle is too thin for serious systems work. A stronger workflow treats the repository as a store of both code and intent — the diagrams here use Mermaid, a lightweight format that works well inside Markdown decision records too:

flowchart TB
    subgraph top[" "]
        direction LR
        A[Frame the problem] --> B[Identify existing decisions] --> C[Explore options and tradeoffs] --> D[Record the selected decision]
    end
    subgraph bottom[" "]
        direction LR
        E[Generate or modify code] --> F[Review code vs decisions] --> G[Merge implementation and memory] --> H[Use record to guide future work]
    end
    D --> E

This process turns the repository into more than a code store. It becomes the source of truth for implementation, intent, and reasoning — a durable artifact that accumulates value with every decision made.

Decision records and docs as code

Decision records work best when they follow docs-as-code principles, which means they should be stored in the same repository as the code, written in plain Markdown, reviewed in pull requests, versioned with Git, linked to related issues and pull requests, and searchable by both humans and AI tools. This is far more reliable than storing important decisions in chat, wiki pages, slide decks, or meeting notes — those tools may still be useful for discussion, but the accepted decision should always live close to the code.

A well-organized repository structure for decision records might look like this:

docs/
  decisions/
    architecture/
      0001-use-postgresql-for-primary-storage.md
      0002-keep-billing-inside-the-core-app.md
    product/
      0001-ai-generated-email-requires-human-review.md
      0002-free-tier-project-limit.md
    design/
      0001-use-inline-validation.md
      0002-place-ai-suggestions-in-side-panel.md

For smaller projects, a flatter structure works equally well. The exact folder organization matters less than consistency — the records must be easy to find, easy to review, and easy for AI tools to load as context before acting on the codebase. For Go teams, this docs/decisions/ structure fits naturally alongside the cmd/, internal/, and api/ layout described in Go Project Structure: Practices & Patterns, which recommends docs/ as the home for architecture decisions and API references.

A practical decision record template

A useful decision record template should be short enough that people actually use it. Here is a practical Markdown template that includes an optional but valuable AI guidance section:

# Decision: Short title

Status: Proposed | Accepted | Superseded | Deprecated
Date: YYYY-MM-DD
Type: Architecture | Product | Design
Owners: Team or names

## Context

Describe the problem, constraints, goals, user needs, technical facts,
and business factors that led to this decision.

## Decision

State the decision clearly.

## Alternatives considered

### Option 1

Pros:
- ...

Cons:
- ...

## Consequences

Describe what becomes easier, what becomes harder, and what risks
or follow-up work this creates.

## AI guidance

When an AI assistant works in this area, it should:
- Preserve ...
- Avoid ...
- Prefer ...
- Ask for review when ...

## Links

- Related issues:
- Related pull requests:
- Related files:
- Supersedes:
- Superseded by:

The "AI guidance" section is optional, but for AI-driven programming it is extremely valuable — it turns the decision record into a durable instruction for future agents working in the same area of the codebase.

What belongs in a decision record?

Not every choice deserves a record, and if every small implementation detail becomes a decision record the process collapses into noise. Create a decision record when a choice is meaningful and likely to matter later.

Good candidates are decisions that:

Affect multiple parts of the system
Encode a product promise
Resolve a real debate
Introduce a long-term tradeoff
Depend on business, compliance, or operational constraints
Would be expensive to rediscover later
Future AI tools might plausibly get wrong
Future contributors might be tempted to reverse casually

Poor candidates include tiny refactor choices, obvious bug fixes, temporary experiments, local naming decisions, and implementation details with no lasting consequence. A good rule of thumb is straightforward: if reversing the decision would require discussion, record the decision.

Status values and lifecycle

Decision records should have a lifecycle to signal their current standing. The simplest status values are sufficient.

Proposed — The decision is being considered but not yet accepted. Use this when the team wants to discuss a decision in a pull request before committing to it.

Accepted — The decision is active and should guide future work. Most useful decision records will spend most of their life in this state.

Superseded — The decision has been replaced by a newer record. Do not delete old records; keep them for history and link to the newer decision so the evolution of thinking stays visible.

Deprecated — The decision is no longer recommended but may still describe existing parts of the system. This is particularly useful during migrations, when old patterns exist in the codebase alongside newer approaches.

The important principle is that decision records should be append-friendly. When the team changes direction, create a new record and link the old one rather than rewriting history to make the past look cleaner.

How AI should generate decision records

AI can help create decision records, and this is one of the better uses of AI in software development — it is fast at drafting structured documents from context. After a discussion, architecture review, or pull request, you can ask an AI assistant to draft a record:

Draft an Architecture Decision Record for the decision in this pull request.
Include context, alternatives, consequences, and AI guidance.
Save it as Markdown under docs/decisions/architecture.

For product work:

Draft a Product Decision Record explaining why AI-generated messages
must remain drafts until reviewed by the user.
Include user impact, out-of-scope behavior, tradeoffs, and AI guidance.

The AI-generated record should not be trusted automatically, however. Human review should verify that the context is accurate, that the AI has not invented rationale, that the alternatives listed are real, that the consequences are honest, and that the AI guidance matches the team's actual intent. AI is a drafting assistant — it is not the owner of the decision.

How AI should read decision records

The other half of the practice is instructing AI to read the records before acting. Before asking an AI assistant to implement a change, include an instruction like this:

Before modifying this feature, read docs/decisions.
Identify any Architecture, Product, or Design Decision Records that apply.
Follow accepted decisions. If your proposed change conflicts with a decision
record, explain the conflict before changing code.

For larger tasks, reinforce the role of the records as project memory:

Use the decision records as project memory.
Do not reverse accepted decisions without proposing a new superseding decision.
When you generate code, explain which decision records influenced the implementation.

This changes the AI's role from "predict plausible code" to "operate inside a documented system of constraints" — a significant improvement in reliability for complex or long-lived projects.

Decision records in pull requests

Decision records should be part of normal pull request review rather than a separate process. A simple PR checklist entry makes the habit visible:

## Decision record checklist

- [ ] This PR does not introduce a significant architecture, product, or design decision.
- [ ] This PR introduces a significant decision and includes a new decision record.
- [ ] This PR changes a previous decision and includes a superseding record.
- [ ] Relevant existing decision records were considered.
- [ ] AI-generated code follows the accepted decision records.
- [ ] AI-generated decision records were reviewed by a human.

This checklist is simple, but it changes behavior by reminding the team that code is not the only artifact that matters in a pull request. It also makes it natural to catch when an AI-generated change silently violates a prior decision.

Decision records and architecture governance

Traditional architecture governance often fails because it is too heavy, too slow, or too disconnected from implementation — central approval boards, large upfront documents, and gatekeeping processes that block rather than guide. Decision records offer a lighter alternative that integrates directly into the development workflow.

They do not require a central architecture board for every change, nor do they block teams from learning and adapting. Instead, they create a trail of decisions that can be reviewed, referenced, and built on over time. This supports evolutionary architecture: the architecture can change, but it changes with memory rather than in spite of it. The team can revisit old decisions without having to rediscover why they were made, which is a healthier and more honest form of governance:

Small records instead of giant documents
Review near the code instead of separate approval theater
Historical context instead of tribal knowledge
Explicit tradeoffs instead of hidden assumptions

Decision records and product management

Product work also needs decision memory, and this is an area where the value of decision records is often underestimated. A roadmap says what might happen. A ticket says what to build next. Analytics say what users did. None of those fully explain why a product behavior exists.

Product Decision Records fill that gap and are especially useful for pricing and packaging decisions, permission models, limits and quotas, AI safety and review flows, onboarding choices, user role definitions, collaboration rules, data retention policies, and feature scope boundaries. Once implemented, product decisions become invisible in the code — later, someone sees only the code and asks, "Why does it work this way?" A PDR gives the answer in a form that both humans and AI tools can find and use.

Decision records and design systems

Design systems often document components, tokens, and usage rules, but they rarely document why the system works that way. Design Decision Records fill this gap. A component library might say "use the confirmation dialog for destructive actions," while a DDR explains the rationale: "We require confirmation for destructive actions because users often work with shared team data, and accidental deletion has a high recovery cost."

That rationale matters beyond the specific component. It helps future designers, developers, and AI tools apply the principle correctly in new situations. Without the DDR, an AI agent may generate a faster interaction that skips confirmation because it appears more efficient. With the DDR, the agent can recognize that preserving the safety property is intentional and non-negotiable.

How decision records support spec-driven development

Spec-driven development explains what the system should do. Decision records explain why the team chose that direction, and the distinction matters significantly for AI-assisted work.

A feature specification may say that AI-generated emails must be saved as drafts. A Product Decision Record explains why automatic sending was rejected, what risks were considered, and which future changes would require a new decision. A design specification may describe a side-panel interaction, while the corresponding DDR explains why inline AI edits were explicitly rejected and why preserving user control was weighted more heavily than workflow speed. An architecture specification may define a service boundary, and its ADR explains why the team chose that boundary over a simpler or more distributed alternative.

The spec guides implementation. The decision record preserves judgment. Together, they give AI coding agents both instructions and context — the "what" and the "why" — which is what makes the combination so effective for complex, long-lived systems.

Decision records are not specifications

Decision records are related to specifications, but they serve a different purpose. A specification says "the system shall do X," while a decision record says "we chose X instead of Y because of these constraints and tradeoffs." That "instead of Y" is the valuable part. AI tools often generate solutions by finding a plausible path to the requested outcome, but decision records tell them which plausible paths have already been explored, evaluated, and rejected — reducing churn and improving the quality of AI-assisted work.

Decision records are not a replacement for tests

Tests verify behavior; decision records explain intent. Both are necessary and they work together. A test can enforce that AI-generated emails must be saved as drafts, while a Product Decision Record explains that this is required because users must review AI-generated communication before it leaves the system. The test protects behavior. The decision record protects meaning. Together, they make future changes safer and more predictable.

Decision records are not a replacement for code comments

Code comments explain local implementation details, while decision records explain broader decisions. Use comments for surprising lines, edge cases, workarounds, and functions that cannot be simplified. Use decision records for why an architecture exists, why a product behavior exists, why an interaction pattern exists, and why the team chose one direction over another. If the explanation affects only a few lines, a comment is the right tool. If it affects the system's direction, a decision record is the right tool.

Common mistakes

Writing records too late

A decision record should be written when the decision is made, not months later when everyone has forgotten the tradeoffs. It is fine to draft one during a pull request. It is better still to draft it before implementation, while the decision is still being actively discussed and the alternatives are fresh.

Making records too long

A decision record is not an essay. It should be detailed enough to preserve judgment but short enough that people will actually read it. Prefer clarity over completeness — a concise record that gets read is far more valuable than a comprehensive one that gets skipped.

Recording decisions without consequences

The consequences section is the heart of the record. A decision without stated consequences is often just a preference rather than a real decision. Good records admit tradeoffs honestly, including what becomes harder or riskier as a result of the choice.

Editing old records as if history changed

When a decision changes, create a new record and mark the old one as superseded. Silently rewriting an old decision to match the current state destroys the historical context that makes decision records valuable. History is useful precisely because it shows how thinking evolved.

Letting AI-generated records merge without review

AI can produce a polished, well-structured record that is subtly wrong. Treat AI-generated decision records exactly like AI-generated code — review them carefully, verify the rationale is accurate, and ensure the consequences section reflects what the team actually accepted.

Hiding records outside the repository

If decision records live in a separate wiki or documentation system, they are less likely to be updated alongside code changes and far less likely to be read by AI coding tools loading context for a task. Keeping them in the repository is not just a convenience — it is what makes the practice work for AI-assisted development.

A lightweight operating model

A practical team process that adds minimal overhead looks like this:

During planning or implementation, identify whether a meaningful decision is being made.
Ask an AI assistant to draft an ADR, PDR, or DDR based on the discussion.
Review the draft as a team, verifying context, alternatives, and consequences.
Commit the record as Markdown in the repository.
Link it from the related issue or pull request.
Instruct AI coding tools to read relevant records before making future changes in the area.
Supersede records when decisions change, preserving the old record for history.

This does not require a new bureaucracy or a dedicated documentation role. It requires a small habit: preserving important judgment at the moment it is created, close to the code where it will be needed.

Example ADR

# Decision: Use PostgreSQL for primary application storage

Status: Accepted
Date: 2026-06-25
Type: Architecture
Owners: Platform team

## Context

The application needs durable relational storage for accounts, projects,
permissions, and audit events. The team expects frequent reporting queries
and strong consistency requirements for permission checks.

## Decision

We will use PostgreSQL as the primary application database.

## Alternatives considered

### DynamoDB

Pros:
- Operationally scalable
- Good fit for predictable key-value access patterns

Cons:
- More complex for relational queries
- Harder for ad hoc reporting
- Less familiar to the current team

### MySQL

Pros:
- Mature relational database
- Familiar operational model

Cons:
- PostgreSQL better matches the team's needs for JSON support,
  indexing options, and existing expertise

## Consequences

PostgreSQL becomes a core operational dependency. The team must manage
migrations carefully and monitor query performance. In return, the
application gets strong relational modeling, mature indexing, and
flexible reporting support.

## AI guidance

When modifying persistence code, prefer relational modeling in PostgreSQL.
Do not introduce a second primary database without a superseding ADR.

Example PDR

# Decision: AI-generated emails must remain drafts

Status: Accepted
Date: 2026-06-25
Type: Product
Owners: Product team

## Context

The product can generate email replies using AI. Sending email is a
high-trust action because mistakes may reach customers, partners, or
internal teams.

## Decision

AI-generated emails must be created as drafts. A human user must
review and send them.

## Alternatives considered

### Send automatically

Pros:
- Faster workflow
- Less user effort

Cons:
- Higher risk of incorrect or inappropriate messages
- Lower user trust
- Harder to recover from mistakes

### Ask for confirmation only after generation

Pros:
- Keeps the workflow simple
- Provides some user control

Cons:
- Still encourages shallow review
- Does not fit existing email client behavior as well as drafts

## Consequences

The workflow is slightly slower, but safer and more trustworthy.
Future automation can improve review speed, but must not bypass
human approval without a superseding PDR.

## AI guidance

When building email-generation features, create drafts by default.
Do not add automatic sending unless a new accepted PDR explicitly allows it.

Example DDR

# Decision: Show AI writing suggestions in a side panel

Status: Accepted
Date: 2026-06-25
Type: Design
Owners: Design team

## Context

Users need help improving written content, but they also need to stay
in control of the final text. Inline AI edits can make it hard to
distinguish user-written content from generated suggestions.

## Decision

AI writing suggestions will appear in a side panel. Users can accept,
reject, or copy suggestions into the main editor.

## Alternatives considered

### Apply suggestions inline

Pros:
- Fast
- Feels integrated

Cons:
- Blurs authorship
- Makes review harder
- Can surprise users

### Show suggestions in a modal

Pros:
- Focused experience
- Easy to implement

Cons:
- Interrupts writing flow
- Harder to compare suggestion and original text

## Consequences

The side panel takes more screen space, especially on small screens.
However, it preserves user control and makes review clearer.

## AI guidance

When adding writing-assistance features, preserve separation between
user text and AI suggestions. Do not apply generated text directly
into the document without explicit user action.

Suggested prompt library

Use these prompts to make decision records part of daily AI-assisted development.

Find relevant records before working on a feature:

Read docs/decisions and identify any accepted decision records that apply
to this task. Summarize the constraints before proposing code changes.

Draft a new ADR:

Draft an Architecture Decision Record for this technical decision.
Include context, decision, alternatives, consequences, and AI guidance.
Keep it concise and specific.

Draft a new PDR:

Draft a Product Decision Record for this product behavior.
Include user impact, scope, alternatives, consequences, and AI guidance.

Draft a new DDR:

Draft a Design Decision Record for this interaction pattern.
Include user problem, alternatives, tradeoffs, consequences, and AI guidance.

Review a pull request against existing decisions:

Review this pull request against the accepted decision records in docs/decisions.
Identify any conflicts, missing decision records, or decisions that should
be superseded.

Supersede a decision:

Create a new decision record that supersedes the existing one.
Preserve the historical rationale, explain what changed, and link both records.

Spec-Driven Development vs Vibe Coding: Waterfall?

Rost — Thu, 02 Jul 2026 10:55:14 +0000

Spec-Driven Development entered 2026 as the serious developer's answer to vibe coding drift.

The argument is simple: AI agents produce better, more consistent output when they implement
against a reviewed specification rather than an ad-hoc prompt.
Hard to argue with in theory.

In practice, Hacker News called it "Waterfall Strikes Back."

Both sides have a point.

The Case for SDD in a Vibe-Coding World

Vibe coding -- the practice of writing a loose prompt and iterating on whatever the AI agent
produces -- works remarkably well for small, exploratory, and throwaway work. For the first
six months of 2025, it was the dominant AI coding pattern. Developers shipped scripts,
prototypes, and simple tools faster than ever.

Then the projects grew. Multi-file features started drifting. Constraints established in
session one were forgotten by session three. Security assumptions got dropped. Architectural
decisions shifted mid-feature because the agent had no durable memory of intent.

Spec-Driven Development (SDD) appeared as the disciplined response. The core claim: make the
specification the central artifact, not the prompt. Write requirements, a design, and a task
plan first. Let the agent implement against those artifacts one slice at a time. Keep the spec
versioned and updated.

AI coding tools -- GitHub Spec Kit, Kiro, BMAD, and a growing set of Claude Code custom workflows -- are all
implementations of this idea. The tooling is real. The interest is real. The backlash is
also real.

What Vibe Coding Is Good At

Before dismissing vibe coding, it is worth being precise about what it does well.

Exploratory prototypes. When you are not sure what you want to build, the fastest path
is to build something rough and react to it. SDD requires knowing what to specify. If you
do not know yet, specs are premature.

UI experiments. Visual layout and interaction feel are hard to specify in advance. Vibe
coding lets you see options quickly, discard most of them, and converge on something that
actually feels right. A requirements doc does not help you here.

Throwaway automation. One-off scripts, data extraction jobs, migration helpers -- these
rarely need a design document. The cost of getting it slightly wrong is low. The cost of
a slow, ceremonial process is real.

Fast feedback. When you need to learn something quickly -- does this API work the way
I think it does? -- vibe coding cuts the learning loop to minutes. SDD would slow that down
for no benefit.

The mistake is to take the success patterns from these contexts and apply them to production
features with real constraints, real users, and real consequences for getting it wrong.

Where Vibe Coding Breaks Down

Vibe coding degrades predictably as scope and stakes increase.

Multi-file changes. Once a feature touches five or more files, the agent's context window
starts losing track of invariants. Without a design document, each prompt has to re-establish
context that was established and forgotten in a prior session.

Architectural drift. Without explicit non-goals, agents implement things. The agent
adds a caching layer because it seems reasonable. Three sessions later, the caching
assumption is baked into the data model and removing it is expensive.

Forgotten constraints. "Only authenticated users can trigger this" is a sentence in a
requirements doc. In a vibe coding session, it is something you mentioned once in session one
and the agent does not remember in session four when it writes the new endpoint.

Hidden security assumptions. Authorization rules, input validation boundaries, secrets
handling -- these are exactly the kind of implicit requirements that get missed when the
agent is optimizing for plausible working code rather than correct, constrained code.

Team handoff. If you built it through iterative prompting, the artifact that records
what was decided and why is... the git log. Good luck with that.

What Spec-Driven Development Changes

SDD does not claim to eliminate iteration. The good versions of SDD are explicitly
iterative. What they change is where iteration happens.

Instead of iterating on code and inferring intent from diffs, you iterate on the spec and
then implement. The spec becomes the artifact that records what was decided, why, and what
is out of scope -- serving a similar function to
Architecture Decision Records
but oriented around feature intent rather than system-level choices. The code implements that intent.

A typical SDD workflow runs through five phases:

Specify. Problem statement, users, goals, non-goals, acceptance criteria, open questions.

Plan. Architecture decisions, affected modules, data model, API contracts, security
concerns, test strategy.

Task breakdown. Small implementation slices with explicit validation criteria and human
review checkpoints.

Implement. One task at a time, with context reset between tasks, applying constraints
from the spec, and updating the plan when reality differs from the design.

Validate. Tests, lint, type checks, acceptance criteria, spec-to-code diff.

The agent participates in most phases -- drafting the spec, generating the design, proposing
tasks -- but humans review the artifacts before implementation starts. That review step is
the central difference between SDD and vibe coding.

Why Developers Call It Waterfall

The waterfall critique is not wrong. It is just aimed at bad SDD, not SDD itself.

The specific failure mode is long upfront planning. Waterfall's defining characteristic is
a feedback loop that stretches to weeks or months: requirements phase, design phase, build
phase, test phase, ship. Feedback arrives late. By the time you discover that the design
assumption was wrong, you have built on top of it for weeks.

When a developer uses Spec Kit and generates a 200-line task list before writing a single
line of code, and then spends two days polishing the requirements document before the agent
touches anything, that is waterfall. It is waterfall with markdown instead of UML, but
the failure mode is identical.

One HN commenter described using Spec Kit for a small CLI tool and finding it "too slow,
too much tweaking before seeing code." That is the bad version. That user was right to
reject it for that task.

The useful critique is not "specs are bad." It is "long upfront planning before feedback
is bad." Those are different claims.

The Useful Middle Ground

Good SDD avoids the waterfall trap by keeping the spec small and starting implementation
early.

Small specs. A requirements document for a single feature should fit on one screen.
If the spec is ten pages, it is either a platform design or it needs to be broken into
smaller features. Specs that are too large take too long to review and become stale quickly.

Short task slices. Each task should be implementable in a single agent session,
reviewable as a small diff, and testable in isolation. If tasks are too large, the
implementation loop stretches and the spec-to-code mapping becomes hard to verify.

Early implementation. Spec the first task, implement it, validate it, then move to the
next task. Do not spec everything before implementing anything. The first implementation
will reveal things your spec got wrong. Update the spec before continuing.

Living spec. When reality differs from the design -- and it will -- update the spec,
not just the code. The spec is only useful if it reflects what was actually built.

Tests as executable feedback. Every acceptance criterion should map to at least one
test. The test suite is the machine-readable version of the spec. If the spec says "only
authenticated users can trigger this," there should be a test that verifies unauthenticated
requests are rejected.

This hybrid -- small specs, short tasks, early implementation, living documents -- is what
actually works. It is not vibe coding and it is not waterfall. It is controlled iteration
with durable artifacts.

When SDD Beats Vibe Coding

Use SDD -- even lightweight SDD -- when the cost of getting it wrong is real.

Risky business logic. Billing, permissions, data migrations, idempotency -- any logic
where incorrect behavior is expensive or hard to reverse. Vibe coding leaves these kinds
of requirements implicit. SDD makes them explicit and reviewable before implementation.

Production API changes. Any change to a public or internal API contract should have a
design document. The design document is what you review before the agent writes code that
breaks callers.

Multi-agent workflows. When multiple agents are implementing different parts of a
feature, the spec is the shared source of truth. Without it, each agent optimizes locally
and the pieces may not fit.

Team handoff. If another developer or another agent will continue this work, the spec
is the handoff artifact. A git log and a README are not enough.

Significant refactors. Refactors that touch core abstractions need an explicit statement
of what must stay the same (behavior) and what is allowed to change (structure). Without
that, the agent may break contracts that you thought were preserved.

When Vibe Coding Is Still Better

SDD is overhead. Sometimes overhead is not worth it.

Quick scripts. A 50-line script to rename files or transform JSON does not need a
requirements doc. Write the prompt, check the output, ship it.

Experiments. If you are learning whether an approach is feasible -- exploring an API,
testing a library, validating a hypothesis -- you need speed, not structure. Experiment
first, specify if the experiment succeeds.

UI sketches. Interaction design benefits from seeing rather than specifying. Build
several rough variations quickly, react to what you see, and only spec what you are
actually going to ship.

Disposable automation. One-time scripts, data imports, migration helpers -- the cost
of a slightly wrong result is usually low, and the artifact is going to be deleted after
use anyway.

Solo prototypes. If you are the only person who will ever see this code and the goal
is learning rather than production, vibe coding is faster and the downsides are contained.

A Simple Decision Framework

The practical question is not "SDD or vibe coding?" It is "how much spec do I need for
this specific task?"

Use vibe coding when:

The task takes less than a day
You are exploring or learning
The artifact is throwaway or low-stakes
You are the only person who will touch this
Feedback speed matters more than correctness

Use lightweight SDD when:

The task takes two or more days
Multiple files are affected
There are explicit security or correctness requirements
Another person or agent will continue the work
You need to write tests that map to requirements

Use full SDD when:

The feature touches a public interface or data contract
Multiple agents or team members are involved
The organization requires a design review before implementation
Compliance or audit trails are required

The most common mistake is applying full SDD to tasks that only need lightweight SDD, and
applying no spec at all to tasks that need at least a lightweight one.

Bad SDD is waterfall with markdown. Good SDD is controlled iteration with durable artifacts.
Vibe coding is the right tool for the right tasks -- and the wrong tool for the wrong ones.
Knowing the difference is the skill.

Useful Links

GitHub Spec Kit documentation -- the portable SDD toolkit
Martin Fowler on SDD tools -- cautious and useful analysis of Kiro, Spec Kit, and Tessl
HN: Waterfall Strikes Back -- the original waterfall critique thread
HN: GitHub Spec Kit launch thread -- community reaction
AI Coding Assistants Comparison -- tools that support SDD workflows: Cursor, Copilot, Claude Code, Kiro
What is Vibe Coding -- Meaning, Tools, Benefits, and Risks in 2026 -- the full vibe coding cluster pillar
AI Developer Tools: The Complete Guide to AI-Powered Development -- the ai-devtools cluster home
Decision Records for AI-Driven Software Development -- how to keep architectural intent durable alongside your specs
Claude Skills for Developers: SKILL.md for VS Code, JetBrains, Cursor -- reusable SDD-style workflows in Claude Code
Python Design Patterns for Clean Architecture -- architecture practices that SDD helps preserve across agent sessions
Unit Testing in Python: Complete Guide with Examples -- turning SDD acceptance criteria into executable tests

DEV Community: Rost

A2A and MCP Agent Security: Identity, Delegation, and Audit Trails

Guardrails vs Protocol Security vs Runtime Policy

Threat Model for A2A and MCP Agent Systems

Malicious or compromised tools (MCP)

Malicious or impersonated agents (A2A)

Confused deputy

Over-broad permissions and hidden delegation chains

Prompt injection through artifacts and cross-agent messages

Poisoned or misleading Agent Cards

Identity Model for Multi-Agent Systems

Internal vs external views of the same agent

Authentication and Authorization for MCP and A2A

MCP tool access

A2A agent access

Gateway-enforced vs per-agent policy

A2A Gateway as the Control Plane

When a gateway is overkill vs necessary

Partner-facing vs internal Agent Cards

Agent Registry and Discovery Security

Registry vs well-known Agent Card URLs

Versioning, deprecation, and ownership

Enterprise internal networks vs external partners

Delegation Across Agent Boundaries

Propagating user intent vs over-delegation

Separate reasoning from execution permissions

Audit Trails and Answer Provenance

Runtime Policy, Egress, and Secrets

Reference Security Architecture

Production Checklist for A2A and MCP Security

Conclusion

Frequently Asked Questions

Sources

A2A Streaming and Async Tasks for Long-Running Agent Workflows

Why Long-Running A2A Agent Tasks Need Async Design

A2A Task Lifecycle and State Transitions

When input_required pauses versus fails a workflow

Who can cancel an A2A task

Human-in-the-Loop with input_required Task States

Approval flows for risky A2A delegation

UX patterns for paused A2A tasks

How an A2A gateway handles input_required

Choosing Sync, SSE Streaming, Polling, or Push Notifications

Read Agent Card capability flags first

When to use assistant-side polling around A2A

A2A Server-Sent Events (SSE) Streaming

Streaming progress updates and partial artifacts

SSE connection drops and resubscription

A2A Push Notifications and Webhooks

When push beats an open SSE connection

Correlating push notifications to A2A tasks

Push endpoint security overview

Async A2A Workflow Patterns

Fire-and-follow task submission

Resuming a task after input_required

Chained A2A delegation with intermediate artifacts

Durable task storage for delayed completion

Failure Handling for Long-Running A2A Workflows

Per-hop and end-to-end timeout budgets

Retries and idempotency for A2A tasks

Partial artifact recovery policies

Safe cancellation across delegation chains

Observability for Async A2A Workflows

Production Checklist for A2A Streaming and Async Tasks

Conclusion

Frequently Asked Questions

Sources

Run Docker Compose as a Linux Service with systemd

When Docker Compose as a Service Makes Sense

When Docker Compose Is Not Enough

The Basic Architecture

Prepare the Compose Project Directory

Use the Modern docker compose Command

Create a systemd Service for Docker Compose

Why Type=oneshot and RemainAfterExit=yes?

Why Not Type=simple?

A More Production-Friendly Unit

Compose .env vs systemd EnvironmentFile

Restart Policies: Docker vs systemd

Health Checks

Use the Modern `docker compose` Command

When Ubuntu `docker.io` Makes Sense

When Ubuntu `docker.io` Is Not Ideal

Option 4: Install Docker from Manual `.deb` Packages

Migrate from Ubuntu `docker.io` to Docker CE