<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Serif COLAKEL</title>
    <description>The latest articles on DEV Community by Serif COLAKEL (@serifcolakel).</description>
    <link>https://dev.to/serifcolakel</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F886132%2F3fec0e05-3fc6-4d3a-9d77-ab9d203d481c.jpeg</url>
      <title>DEV Community: Serif COLAKEL</title>
      <link>https://dev.to/serifcolakel</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/serifcolakel"/>
    <language>en</language>
    <item>
      <title>Building an AI Gateway: Model Routing, Fallbacks, Budgets and Provider Abstraction</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Thu, 19 Nov 2026 18:13:26 +0000</pubDate>
      <link>https://dev.to/serifcolakel/building-an-ai-gateway-model-routing-fallbacks-budgets-and-provider-abstraction-1idl</link>
      <guid>https://dev.to/serifcolakel/building-an-ai-gateway-model-routing-fallbacks-budgets-and-provider-abstraction-1idl</guid>
      <description>&lt;p&gt;&lt;em&gt;Designing a centralized infrastructure layer to manage multiple LLM providers, route requests intelligently, handle failures, and enforce usage policies.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Why Do We Need an AI Gateway?
&lt;/h2&gt;

&lt;p&gt;As AI adoption grows, applications increasingly depend on multiple model providers and deployment environments.&lt;/p&gt;

&lt;p&gt;One application may use OpenAI for general reasoning, Anthropic for code analysis, Google for multimodal tasks, and internally hosted models for specific workloads.&lt;/p&gt;

&lt;p&gt;Initially, integrating each provider directly into an application seems straightforward.&lt;/p&gt;

&lt;p&gt;We initialize an SDK, configure an API key, select a model, and start making requests.&lt;/p&gt;

&lt;p&gt;However, as the number of applications and teams increases, managing these integrations independently introduces several problems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Every application manages its own provider credentials.&lt;/li&gt;
&lt;li&gt;Model configuration becomes duplicated.&lt;/li&gt;
&lt;li&gt;Rate limits and retry policies are implemented differently.&lt;/li&gt;
&lt;li&gt;Switching providers requires application changes.&lt;/li&gt;
&lt;li&gt;Usage and spending become difficult to attribute.&lt;/li&gt;
&lt;li&gt;Provider failures affect applications independently.&lt;/li&gt;
&lt;li&gt;Security and data-handling policies become inconsistent.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Consider an organization with several AI-powered applications:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An internal developer assistant.&lt;/li&gt;
&lt;li&gt;A document analysis service.&lt;/li&gt;
&lt;li&gt;A financial research agent.&lt;/li&gt;
&lt;li&gt;A customer support assistant.&lt;/li&gt;
&lt;li&gt;A multi-agent orchestration platform.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each application may use different models, but many of their infrastructure requirements are identical.&lt;/p&gt;

&lt;p&gt;Instead of implementing provider management separately in every application, we can introduce a centralized infrastructure layer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;An AI Gateway acts as a unified entry point between applications and model providers.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It centralizes model access, provider routing, authentication, usage policies, and failure handling.&lt;/p&gt;

&lt;p&gt;The objective is not simply to reduce the number of API integrations.&lt;/p&gt;

&lt;p&gt;It is to make model access a shared, governed, and independently manageable platform capability.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. What Is an AI Gateway?
&lt;/h2&gt;

&lt;p&gt;An AI Gateway is an infrastructure component that sits between AI applications and model providers.&lt;/p&gt;

&lt;p&gt;Applications send model requests through a common gateway endpoint. The gateway handles provider-specific integrations and applies the configured routing and access policies.&lt;/p&gt;

&lt;p&gt;A simplified architecture:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 AI Applications
                        |
          +-------------+-------------+
          |             |             |
          v             v             v
       Chat App     Agent Runtime   API Service
          |             |             |
          +-------------+-------------+
                        |
                        v
                   AI Gateway
                        |
          +-------------+-------------+
          |             |             |
          v             v             v
       Identity      Model Router   Policies
       Management       Engine      &amp;amp; Quotas
          |             |             |
          +-------------+-------------+
                        |
              +---------+---------+
              |         |         |
              v         v         v
           OpenAI   Anthropic   Google
              |         |         |
              +---------+---------+
                        |
                        v
                Model Response
                        |
                        v
                   Application
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A gateway can provide several capabilities:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Capability&lt;/th&gt;
&lt;th&gt;Responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Provider abstraction&lt;/td&gt;
&lt;td&gt;Common interface for multiple providers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model routing&lt;/td&gt;
&lt;td&gt;Select the appropriate model or deployment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Provider routing&lt;/td&gt;
&lt;td&gt;Select an upstream provider serving a model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Load balancing&lt;/td&gt;
&lt;td&gt;Distribute traffic across eligible deployments&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failover&lt;/td&gt;
&lt;td&gt;Switch to an alternative after eligible failures&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Authentication&lt;/td&gt;
&lt;td&gt;Centralize application authentication&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Authorization&lt;/td&gt;
&lt;td&gt;Control access to providers and models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rate limiting&lt;/td&gt;
&lt;td&gt;Enforce request and token quotas&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Budget management&lt;/td&gt;
&lt;td&gt;Track and restrict usage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Observability&lt;/td&gt;
&lt;td&gt;Record request, latency, and usage metadata&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Policy enforcement&lt;/td&gt;
&lt;td&gt;Apply organizational model-access rules&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Not every AI Gateway implements all these capabilities in the same way.&lt;/p&gt;

&lt;p&gt;Some focus on request routing and provider normalization, while others offer more comprehensive enterprise governance features.&lt;/p&gt;

&lt;p&gt;It is also important to distinguish an AI Gateway from an agent runtime.&lt;/p&gt;

&lt;p&gt;The gateway manages model access. The agent runtime manages planning, tool execution, memory, and agent orchestration.&lt;/p&gt;

&lt;p&gt;These layers can work together without taking responsibility for each other's operations.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. AI Gateway vs. Model Provider vs. AI SDK
&lt;/h2&gt;

&lt;p&gt;These technologies are complementary.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Model Provider&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A model provider hosts or exposes the actual inference service.&lt;/p&gt;

&lt;p&gt;Examples include OpenAI, Anthropic, Google, Amazon Bedrock, Azure AI, and self-hosted inference endpoints.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AI SDK&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;An AI SDK provides application-level APIs for interacting with models.&lt;/p&gt;

&lt;p&gt;For example, Vercel AI SDK provides &lt;code&gt;generateText&lt;/code&gt;, &lt;code&gt;streamText&lt;/code&gt;, structured output, and tool-calling abstractions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AI Gateway&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;An AI Gateway provides infrastructure-level management of model requests, including routing, provider selection, authentication, and usage policies.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Primary responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;AI SDK&lt;/td&gt;
&lt;td&gt;Application development&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AI Gateway&lt;/td&gt;
&lt;td&gt;Model access infrastructure&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model Provider&lt;/td&gt;
&lt;td&gt;Model inference&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For example, an application can use Vercel AI SDK for its agent implementation while routing all model calls through an AI Gateway.&lt;/p&gt;

&lt;p&gt;The application still owns its business logic. The gateway manages the infrastructure surrounding model access.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Model Routing: Selecting the Right Model for the Right Task
&lt;/h2&gt;

&lt;p&gt;Model routing is one of the most important capabilities in a multi-provider architecture.&lt;/p&gt;

&lt;p&gt;However, the term &lt;em&gt;model routing&lt;/em&gt; can describe several different decisions.&lt;/p&gt;

&lt;p&gt;We should distinguish them before designing the routing layer.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.1. Task-Based Model Selection
&lt;/h3&gt;

&lt;p&gt;This determines which model is appropriate for a particular task.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Model requirements&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Text classification&lt;/td&gt;
&lt;td&gt;Low latency and cost&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Document extraction&lt;/td&gt;
&lt;td&gt;Structured output&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Complex reasoning&lt;/td&gt;
&lt;td&gt;Strong reasoning capabilities&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Code review&lt;/td&gt;
&lt;td&gt;Coding performance and tool calling&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Image understanding&lt;/td&gt;
&lt;td&gt;Vision support&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Embedding generation&lt;/td&gt;
&lt;td&gt;Compatible embedding dimensions&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Using an expensive reasoning model for every classification request may increase cost unnecessarily.&lt;/p&gt;

&lt;p&gt;Likewise, routing every complex analysis task to the cheapest available model may compromise quality.&lt;/p&gt;

&lt;p&gt;This decision is generally owned by application-level routing logic or a dedicated classification layer.&lt;/p&gt;

&lt;p&gt;It should not be confused with provider failover.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.2. Provider Routing
&lt;/h3&gt;

&lt;p&gt;After selecting a model, we may have multiple providers or deployments capable of serving it.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Requested Model: Claude Sonnet
                 |
                 v
              Gateway
                 |
        +--------+--------+
        |        |        |
        v        v        v
     Anthropic  Vertex   Bedrock
        |        |        |
        +--------+--------+
                 |
                 v
              Response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The gateway decides which eligible provider should receive the request.&lt;/p&gt;

&lt;p&gt;Routing policies can consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Provider availability.&lt;/li&gt;
&lt;li&gt;Recent latency.&lt;/li&gt;
&lt;li&gt;Throughput.&lt;/li&gt;
&lt;li&gt;Cost.&lt;/li&gt;
&lt;li&gt;Region.&lt;/li&gt;
&lt;li&gt;Data-retention requirements.&lt;/li&gt;
&lt;li&gt;Provider-specific capabilities.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, Vercel AI Gateway can choose among providers based on recent uptime and latency, with additional controls for provider ordering and filtering.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.3. Deployment Load Balancing
&lt;/h3&gt;

&lt;p&gt;A single logical model may have multiple deployments.&lt;/p&gt;

&lt;p&gt;A gateway can distribute traffic across them using strategies such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Round robin.&lt;/li&gt;
&lt;li&gt;Random selection.&lt;/li&gt;
&lt;li&gt;Weighted distribution.&lt;/li&gt;
&lt;li&gt;Least loaded deployment.&lt;/li&gt;
&lt;li&gt;Latency-aware selection.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The appropriate strategy depends on traffic characteristics and provider capabilities.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.4. Semantic or Complexity-Based Routing
&lt;/h3&gt;

&lt;p&gt;A more advanced approach is to classify incoming requests based on complexity.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Incoming Request
       |
       v
Task Classifier
       |
       +----&amp;gt; Simple Task
       |          |
       |          v
       |      Fast Model
       |
       +----&amp;gt; Complex Task
       |          |
       |          v
       |     Reasoning Model
       |
       +----&amp;gt; Code Task
                  |
                  v
              Coding Model
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This can reduce inference costs when a large proportion of requests are simple.&lt;/p&gt;

&lt;p&gt;However, this classifier is often an application-level component or a dedicated routing service.&lt;/p&gt;

&lt;p&gt;We should not assume that every AI Gateway automatically understands task complexity and selects an appropriate model.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The key architectural distinction:&lt;/strong&gt; Task selection chooses the model. Provider routing chooses where that model runs. Failover handles failures in either layer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://vercel.com/docs/ai-gateway/models-and-providers/provider-options" rel="noopener noreferrer"&gt;Vercel AI Gateway — Provider Options&lt;/a&gt; · &lt;a href="https://openrouter.ai/blog/insights/model-routing/" rel="noopener noreferrer"&gt;OpenRouter — Model Routing&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Provider Abstraction: One Interface, Multiple Providers
&lt;/h2&gt;

&lt;p&gt;One of the most useful properties of an AI Gateway is provider abstraction.&lt;/p&gt;

&lt;p&gt;Instead of making application code depend on specific provider endpoints, we can introduce logical model identifiers.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application
    |
    v
general-reasoning
    |
    v
AI Gateway
    |
    v
Provider Resolution
    |
    +----&amp;gt; OpenAI
    |
    +----&amp;gt; Anthropic
    |
    +----&amp;gt; Azure AI
    |
    +----&amp;gt; Self-hosted Model
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The application requests &lt;code&gt;general-reasoning&lt;/code&gt;, while the gateway resolves that identifier using its model catalog and routing policy.&lt;/p&gt;

&lt;p&gt;This allows the underlying deployment to change without requiring every consuming application to change its integration code.&lt;/p&gt;

&lt;h3&gt;
  
  
  OpenAI-Compatible APIs
&lt;/h3&gt;

&lt;p&gt;Many gateways expose OpenAI-compatible endpoints.&lt;/p&gt;

&lt;p&gt;Common examples include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;POST /v1/chat/completions
POST /v1/embeddings
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This makes migration relatively straightforward for applications already using compatible clients.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'openai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;baseURL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;AI_GATEWAY_URL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;apiKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;AI_GATEWAY_API_KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'general-reasoning'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'user'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Explain the purpose of an AI Gateway.'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The application does not need to know which provider serves &lt;code&gt;general-reasoning&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;However, there is an important limitation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;OpenAI compatibility does not guarantee complete model interoperability.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Different providers have different support for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tool calling.&lt;/li&gt;
&lt;li&gt;Structured output.&lt;/li&gt;
&lt;li&gt;JSON Schema.&lt;/li&gt;
&lt;li&gt;Streaming.&lt;/li&gt;
&lt;li&gt;Reasoning parameters.&lt;/li&gt;
&lt;li&gt;Multimodal input.&lt;/li&gt;
&lt;li&gt;Prompt caching.&lt;/li&gt;
&lt;li&gt;Response metadata.&lt;/li&gt;
&lt;li&gt;Cancellation behavior.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A gateway can normalize common request and response structures, but it cannot guarantee identical model behavior.&lt;/p&gt;

&lt;p&gt;Provider abstraction should therefore be based on explicit capabilities, not the assumption that every model is interchangeable.&lt;/p&gt;

&lt;h3&gt;
  
  
  Define a Capability Contract
&lt;/h3&gt;

&lt;p&gt;A model registry can maintain the capabilities that matter to the application:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;ModelCapabilities&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;streaming&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;toolCalling&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;structuredOutput&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;vision&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;audio&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;embeddings&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;maxContextTokens&lt;/span&gt;&lt;span class="o"&gt;?:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;ModelDefinition&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;capabilities&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ModelCapabilities&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A routing policy can then exclude models that do not satisfy the requirements of a request.&lt;/p&gt;

&lt;p&gt;For example, a workflow that requires tool calling and structured output should not silently fall back to a model that supports neither.&lt;/p&gt;

&lt;p&gt;This is especially important in agentic applications, where a model change can affect the behavior of an entire execution loop.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Model Fallbacks: Handling Provider Failures
&lt;/h2&gt;

&lt;p&gt;Provider outages, rate limits, and network failures are inevitable in distributed systems.&lt;/p&gt;

&lt;p&gt;A multi-provider architecture allows us to introduce fallback strategies.&lt;/p&gt;

&lt;p&gt;Consider an application that requests a model from a primary provider.&lt;/p&gt;

&lt;p&gt;The provider becomes unavailable and returns a temporary error.&lt;/p&gt;

&lt;p&gt;Without failover, the application has to handle the failure itself.&lt;/p&gt;

&lt;p&gt;With a configured gateway fallback, the request can be routed to an alternative deployment or model.&lt;/p&gt;

&lt;h3&gt;
  
  
  Two Different Fallback Strategies
&lt;/h3&gt;

&lt;p&gt;It is important to distinguish between provider fallback and model fallback.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Provider fallback&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The requested model remains the same, but the gateway tries another provider that serves it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Requested Model
       |
       v
Provider A
       |
     Failed
       |
       v
Provider B
       |
       v
Same Model Response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Model fallback&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The primary model fails, so the gateway tries a different model.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Primary Model
       |
     Failed
       |
       v
Fallback Model 1
       |
     Failed
       |
       v
Fallback Model 2
       |
       v
Response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These strategies solve different availability problems.&lt;/p&gt;

&lt;h3&gt;
  
  
  Example Failure Policy
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;fallbackPolicy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;primary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'primary-model'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;providerFallbacks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="s1"&gt;'approved-provider-a'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;'approved-provider-b'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;modelFallbacks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="s1"&gt;'compatible-model-a'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;'compatible-model-b'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;maxAttempts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is an illustrative application-level policy, not a universal gateway configuration format.&lt;/p&gt;

&lt;p&gt;Actual routing and fallback behavior depends on the gateway.&lt;/p&gt;

&lt;h3&gt;
  
  
  Retry Is Not the Same as Fallback
&lt;/h3&gt;

&lt;p&gt;A retry attempts the same logical operation again.&lt;/p&gt;

&lt;p&gt;A fallback changes the eligible provider or model according to a policy.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;Possible strategy&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;HTTP 429&lt;/td&gt;
&lt;td&gt;Respect retry guidance or use another deployment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP 500 / 503&lt;/td&gt;
&lt;td&gt;Bounded retry or failover&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network timeout&lt;/td&gt;
&lt;td&gt;Retry within an execution deadline&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP 400&lt;/td&gt;
&lt;td&gt;Do not blindly retry&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Authentication failure&lt;/td&gt;
&lt;td&gt;Correct credentials or configuration&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context window exceeded&lt;/td&gt;
&lt;td&gt;Reduce input or use a compatible larger-context model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unsupported capability&lt;/td&gt;
&lt;td&gt;Select a model that supports the required capability&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Content policy rejection&lt;/td&gt;
&lt;td&gt;Do not automatically bypass the policy&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Avoid Retry Storms
&lt;/h3&gt;

&lt;p&gt;If multiple application instances retry simultaneously during an outage, they can increase the load on an already degraded provider.&lt;/p&gt;

&lt;p&gt;Use:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Exponential backoff.&lt;/li&gt;
&lt;li&gt;Randomized jitter.&lt;/li&gt;
&lt;li&gt;Maximum retry attempts.&lt;/li&gt;
&lt;li&gt;Request deadlines.&lt;/li&gt;
&lt;li&gt;Provider cooldowns.&lt;/li&gt;
&lt;li&gt;Circuit-breaker patterns.&lt;/li&gt;
&lt;li&gt;Retry-After handling.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There should also be one clearly defined retry owner.&lt;/p&gt;

&lt;p&gt;If both the application and gateway independently retry without coordinated limits, one logical request can generate many more upstream attempts than expected.&lt;/p&gt;

&lt;h3&gt;
  
  
  Streaming Requires a Separate Strategy
&lt;/h3&gt;

&lt;p&gt;Streaming introduces another important limitation.&lt;/p&gt;

&lt;p&gt;If a provider fails before sending any content, a gateway may be able to route the request to another provider.&lt;/p&gt;

&lt;p&gt;But if partial output has already been delivered to the user, switching providers is no longer transparent.&lt;/p&gt;

&lt;p&gt;The application may receive duplicate, contradictory, or incomplete content.&lt;/p&gt;

&lt;p&gt;For that reason, streaming systems should define how to handle mid-stream failures.&lt;/p&gt;

&lt;p&gt;Possible approaches include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Retry before the first emitted content.&lt;/li&gt;
&lt;li&gt;Terminate the stream and expose a recoverable error.&lt;/li&gt;
&lt;li&gt;Restart using an explicit new attempt.&lt;/li&gt;
&lt;li&gt;Use a continuation strategy where supported.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Never assume that every fallback is invisible to the user.&lt;/p&gt;

&lt;h3&gt;
  
  
  Model Compatibility Matters
&lt;/h3&gt;

&lt;p&gt;A successful HTTP response does not necessarily mean the fallback result is equivalent to the primary model's result.&lt;/p&gt;

&lt;p&gt;A fallback might:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Generate a different output structure.&lt;/li&gt;
&lt;li&gt;Use different tool-calling semantics.&lt;/li&gt;
&lt;li&gt;Have a smaller context window.&lt;/li&gt;
&lt;li&gt;Interpret system instructions differently.&lt;/li&gt;
&lt;li&gt;Produce less reliable structured output.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every fallback should pass the application's capability and quality requirements.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://vercel.com/docs/ai-gateway/models-and-providers/model-fallbacks" rel="noopener noreferrer"&gt;Vercel — Model Fallbacks&lt;/a&gt; · &lt;a href="https://docs.litellm.ai/docs/proxy/reliability" rel="noopener noreferrer"&gt;LiteLLM — Routing and Reliability&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Authentication, Authorization and Key Management
&lt;/h2&gt;

&lt;p&gt;In enterprise environments, provider credentials quickly become a governance problem.&lt;/p&gt;

&lt;p&gt;If each application stores its own provider API keys, the organization loses centralized control.&lt;/p&gt;

&lt;p&gt;This can create several risks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Credentials accidentally exposed in application environments.&lt;/li&gt;
&lt;li&gt;Unauthorized provider usage.&lt;/li&gt;
&lt;li&gt;Inconsistent key rotation.&lt;/li&gt;
&lt;li&gt;Shared administrative credentials.&lt;/li&gt;
&lt;li&gt;Difficult usage attribution.&lt;/li&gt;
&lt;li&gt;Inconsistent access policies.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An AI Gateway provides an opportunity to centralize provider credentials and application identities.&lt;/p&gt;

&lt;h3&gt;
  
  
  Application-to-Gateway Authentication
&lt;/h3&gt;

&lt;p&gt;The application authenticates to the gateway with a gateway-specific credential.&lt;/p&gt;

&lt;p&gt;The gateway uses its configured provider credentials to access the upstream inference service.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application
    |
    | Gateway Credential
    v
AI Gateway
    |
    | Provider Credential
    v
LLM Provider
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The application does not need direct access to upstream provider credentials.&lt;/p&gt;

&lt;h3&gt;
  
  
  Virtual Keys
&lt;/h3&gt;

&lt;p&gt;Some gateways provide virtual keys that identify applications, users, teams, or projects.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Key&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;th&gt;Access&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;frontend-assistant&lt;/td&gt;
&lt;td&gt;Chat application&lt;/td&gt;
&lt;td&gt;Approved chat models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;research-service&lt;/td&gt;
&lt;td&gt;Research workflow&lt;/td&gt;
&lt;td&gt;Reasoning and embeddings&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;document-worker&lt;/td&gt;
&lt;td&gt;Document processing&lt;/td&gt;
&lt;td&gt;Extraction models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;development-agent&lt;/td&gt;
&lt;td&gt;Coding assistant&lt;/td&gt;
&lt;td&gt;Approved coding models&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Virtual keys can enforce model access, usage limits, and spending policies.&lt;/p&gt;

&lt;p&gt;However, the exact capabilities depend on the gateway and deployment model.&lt;/p&gt;

&lt;h3&gt;
  
  
  Identity Propagation
&lt;/h3&gt;

&lt;p&gt;Consider a shared backend serving multiple authenticated users.&lt;/p&gt;

&lt;p&gt;The gateway should receive trusted application or user identity metadata derived from the server-side authentication context.&lt;/p&gt;

&lt;p&gt;It should not trust arbitrary user identifiers sent directly from a browser.&lt;/p&gt;

&lt;p&gt;This matters for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Per-user budgets.&lt;/li&gt;
&lt;li&gt;Tenant isolation.&lt;/li&gt;
&lt;li&gt;Audit logs.&lt;/li&gt;
&lt;li&gt;Access control.&lt;/li&gt;
&lt;li&gt;Cost attribution.&lt;/li&gt;
&lt;li&gt;Investigating unexpected usage.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Vercel AI Gateway Authentication
&lt;/h3&gt;

&lt;p&gt;Vercel AI Gateway supports API key authentication and OIDC authentication.&lt;/p&gt;

&lt;p&gt;For applications deployed on Vercel, OIDC tokens can be used without maintaining a static gateway key in the application.&lt;/p&gt;

&lt;p&gt;For other environments, gateway API keys can be configured through environment variables.&lt;/p&gt;

&lt;p&gt;It also supports Bring Your Own Key (BYOK), allowing teams to use their own provider credentials.&lt;/p&gt;

&lt;p&gt;An important operational detail: when BYOK credentials fail, the configured gateway fallback behavior may use Vercel-managed credentials. Teams should understand the resulting billing behavior rather than assuming that every request is always billed directly to their provider account.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://vercel.com/docs/ai-gateway/authentication-and-byok" rel="noopener noreferrer"&gt;Vercel — Authentication and BYOK&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Budgets, Quotas and Rate Limiting
&lt;/h2&gt;

&lt;p&gt;Centralizing model access also makes it possible to manage usage consistently.&lt;/p&gt;

&lt;p&gt;Without a shared gateway, individual applications may implement different rate-limit and budget mechanisms.&lt;/p&gt;

&lt;p&gt;A gateway can provide controls at different scopes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Organization.&lt;/li&gt;
&lt;li&gt;Team.&lt;/li&gt;
&lt;li&gt;Project.&lt;/li&gt;
&lt;li&gt;Application.&lt;/li&gt;
&lt;li&gt;API key.&lt;/li&gt;
&lt;li&gt;User.&lt;/li&gt;
&lt;li&gt;Model.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  RPM and TPM
&lt;/h3&gt;

&lt;p&gt;Two important rate-limit dimensions are:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;RPM — Requests Per Minute&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Controls the number of requests in a defined time window.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TPM — Tokens Per Minute&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Controls token consumption within a defined time window.&lt;/p&gt;

&lt;p&gt;These limits are useful for protecting upstream provider quotas and preventing individual applications from monopolizing shared capacity.&lt;/p&gt;

&lt;h3&gt;
  
  
  Budget Management
&lt;/h3&gt;

&lt;p&gt;Rate limits control request volume.&lt;/p&gt;

&lt;p&gt;Budgets control spending.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AI Platform Budget

Monthly Limit: $1,000

Development Team: $300
Research Team:    $400
Automation Team:  $300
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The gateway can track consumption and enforce configured spending policies.&lt;/p&gt;

&lt;p&gt;However, budget behavior is not identical across platforms.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Vercel AI Gateway&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Vercel supports team, project, API key, and user budget scopes.&lt;/p&gt;

&lt;p&gt;These budgets are soft caps, not exact hard spending limits.&lt;/p&gt;

&lt;p&gt;The check occurs when a request starts. The request that crosses the threshold can still finish, and the actual total may exceed the configured limit slightly.&lt;/p&gt;

&lt;p&gt;BYOK billing also requires separate consideration because usage billed directly through provider credentials may not be covered by the same gateway spending boundary.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;LiteLLM&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;LiteLLM provides key-, user-, team-, and other scope-based usage controls.&lt;/p&gt;

&lt;p&gt;Its budget and virtual-key functionality depends on database-backed usage tracking. Running only the configuration-based proxy without a database does not provide the same budget enforcement.&lt;/p&gt;

&lt;p&gt;This is an important deployment distinction.&lt;/p&gt;

&lt;h3&gt;
  
  
  Budget Enforcement Under Concurrency
&lt;/h3&gt;

&lt;p&gt;Imagine ten concurrent requests arriving when a team has only a small amount of budget remaining.&lt;/p&gt;

&lt;p&gt;Depending on the implementation, multiple requests may pass a check before the usage records are fully updated.&lt;/p&gt;

&lt;p&gt;For strict spending controls, consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Atomic budget reservation.&lt;/li&gt;
&lt;li&gt;Shared quota state.&lt;/li&gt;
&lt;li&gt;Concurrency limits.&lt;/li&gt;
&lt;li&gt;Request admission control.&lt;/li&gt;
&lt;li&gt;Maximum estimated request cost.&lt;/li&gt;
&lt;li&gt;Reconciliation against actual provider usage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A gateway budget is an important control, but it should not be confused with a guarantee that an entire distributed AI workflow cannot exceed a fixed monetary amount.&lt;/p&gt;

&lt;h3&gt;
  
  
  What Should Happen When a Budget Is Exhausted?
&lt;/h3&gt;

&lt;p&gt;Possible policies include:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Reject new requests.&lt;/li&gt;
&lt;li&gt;Require additional authorization.&lt;/li&gt;
&lt;li&gt;Route to an approved lower-cost model.&lt;/li&gt;
&lt;li&gt;Route to an approved self-hosted model.&lt;/li&gt;
&lt;li&gt;Reduce usage for non-critical workloads.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The correct behavior depends on business requirements.&lt;/p&gt;

&lt;p&gt;A critical workflow should not silently switch to a less capable model simply because its budget has been reached.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://vercel.com/docs/ai-gateway/observability-and-spend/budgets" rel="noopener noreferrer"&gt;Vercel — Budgets and Spend Limits&lt;/a&gt; · &lt;a href="https://docs.litellm.ai/docs/proxy/users" rel="noopener noreferrer"&gt;LiteLLM — Budgets and Rate Limits&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Integrating Vercel AI Gateway with AI SDK v7
&lt;/h2&gt;

&lt;p&gt;One of the advantages of Vercel AI Gateway is its direct integration with the AI SDK.&lt;/p&gt;

&lt;p&gt;An important correction to many older examples:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Gateway provider is now part of the &lt;code&gt;ai&lt;/code&gt; package.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You do not need to install a separate provider package for the standard integration.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Install the dependency
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm add ai
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 2: Configure authentication
&lt;/h3&gt;

&lt;p&gt;For local development or external deployments:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;AI_GATEWAY_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;your_gateway_api_key
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On supported Vercel deployments, OIDC authentication can be used automatically.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Make a request
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;generateText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'ai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'openai/gpt-6-astra'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`
    Analyze this application architecture
    and identify its main scalability risks.
  `&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model identifier follows the &lt;code&gt;provider/model-name&lt;/code&gt; format.&lt;/p&gt;

&lt;p&gt;The gateway resolves the model and manages the upstream provider request.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Configure Model Fallbacks
&lt;/h3&gt;

&lt;p&gt;We can specify alternative models through &lt;code&gt;providerOptions.gateway.models&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;generateText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'ai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'openai/gpt-6-astra'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Analyze the architectural risks of this system.'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;providerOptions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s1"&gt;'anthropic/claude-sonnet-5.5'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'google/gemini-4-hydrogen'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The gateway tries the primary model first.&lt;/p&gt;

&lt;p&gt;If it fails, it proceeds through the configured fallback model list and returns the first successful result.&lt;/p&gt;

&lt;p&gt;The model identifiers above are examples from the current AI Gateway catalog documentation. Always verify actual model availability and capability compatibility before deployment.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Control Provider Ordering
&lt;/h3&gt;

&lt;p&gt;Model fallback and provider ordering are separate controls.&lt;/p&gt;

&lt;p&gt;We can specify a preferred provider order for a particular model:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;generateText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'ai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'anthropic/claude-sonnet-5.5'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Explain distributed systems.'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;providerOptions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s1"&gt;'vertex'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'anthropic'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;order&lt;/code&gt; option controls provider preference.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;only&lt;/code&gt; option restricts routing to an explicit provider allowlist.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="n"&gt;providerOptions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;only&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'vertex'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'anthropic'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This distinction is useful when a request is required to use specific provider infrastructure.&lt;/p&gt;

&lt;p&gt;The Gateway provider also exposes model discovery and usage reporting capabilities through the AI SDK interface.&lt;/p&gt;

&lt;p&gt;For example, available models can be inspected programmatically rather than relying on hardcoded model lists.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://ai-sdk.dev/providers/ai-sdk-providers/ai-gateway" rel="noopener noreferrer"&gt;AI SDK — AI Gateway Provider&lt;/a&gt; · &lt;a href="https://vercel.com/docs/ai-gateway/models-and-providers/provider-options" rel="noopener noreferrer"&gt;Vercel — Provider Options&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Building a Self-Hosted AI Gateway with LiteLLM
&lt;/h2&gt;

&lt;p&gt;A managed gateway is not always the right solution.&lt;/p&gt;

&lt;p&gt;Some organizations need to operate the model access layer inside their own infrastructure because of:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Network restrictions.&lt;/li&gt;
&lt;li&gt;Internal model endpoints.&lt;/li&gt;
&lt;li&gt;Data residency requirements.&lt;/li&gt;
&lt;li&gt;Private deployments.&lt;/li&gt;
&lt;li&gt;Custom authentication.&lt;/li&gt;
&lt;li&gt;Infrastructure ownership.&lt;/li&gt;
&lt;li&gt;Provider-specific policies.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;LiteLLM provides an open-source gateway with a unified interface for multiple model providers.&lt;/p&gt;

&lt;p&gt;It supports model routing, retries, fallbacks, virtual keys, budgets, rate limiting, and usage tracking.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Define Logical Models
&lt;/h3&gt;

&lt;p&gt;A simplified configuration:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;model_list&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;model_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;fast-chat&lt;/span&gt;
    &lt;span class="na"&gt;litellm_params&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;openai/gpt-6.1-sol&lt;/span&gt;
      &lt;span class="na"&gt;api_key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;os.environ/OPENAI_API_KEY&lt;/span&gt;

  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;model_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;reasoning-chat&lt;/span&gt;
    &lt;span class="na"&gt;litellm_params&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;anthropic/claude-sonnet-5&lt;/span&gt;
      &lt;span class="na"&gt;api_key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;os.environ/ANTHROPIC_API_KEY&lt;/span&gt;

&lt;span class="na"&gt;router_settings&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;num_retries&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;

  &lt;span class="na"&gt;fallbacks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;fast-chat&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"reasoning-chat"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;fast-chat&lt;/code&gt; is the logical model name exposed by the gateway.&lt;/li&gt;
&lt;li&gt;The OpenAI model is the primary deployment.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;reasoning-chat&lt;/code&gt; is an alternative model group.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;num_retries&lt;/code&gt; controls retry attempts within the configured routing behavior.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;fallbacks&lt;/code&gt; defines a cross-model fallback relationship.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The model names and provider configuration should be validated against the installed LiteLLM version and actual provider availability.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Deploy with PostgreSQL
&lt;/h3&gt;

&lt;p&gt;For a real deployment with virtual keys and budget enforcement, use a database-backed setup.&lt;/p&gt;

&lt;p&gt;LiteLLM's official Docker Compose quickstart starts the proxy together with PostgreSQL.&lt;/p&gt;

&lt;p&gt;The official setup also requires:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;LITELLM_MASTER_KEY&lt;/code&gt; for administration and management API authentication.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;LITELLM_SALT_KEY&lt;/code&gt; for encrypting provider credentials stored in the gateway database.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The official quickstart is a good starting point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sSLO \
  https://github.com/BerriAI/litellm/raw/main/litellm/deploy/docker/docker-compose.quickstart.yml

&lt;span class="nb"&gt;printf&lt;/span&gt; &lt;span class="s1"&gt;'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n'&lt;/span&gt; \
  &lt;span class="s2"&gt;"$(openssl rand -hex 32)"&lt;/span&gt; \
  &lt;span class="s2"&gt;"$(openssl rand -hex 32)"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; .env

docker compose -f docker-compose.quickstart.yml up -d
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Models and virtual keys can then be configured using the Admin UI or version-controlled configuration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Important:&lt;/strong&gt; A standalone LiteLLM proxy running without a database can still provide basic model routing, but virtual keys and database-backed spend tracking and budget enforcement will not work. Do not treat a configuration-only Docker container as a complete budget-managed gateway.&lt;/p&gt;

&lt;p&gt;Also, do not use a floating &lt;code&gt;latest&lt;/code&gt; image tag for a controlled production deployment. Pin a tested version or image digest and review dependency updates.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Connect an application
&lt;/h3&gt;

&lt;p&gt;LiteLLM exposes an OpenAI-compatible API.&lt;/p&gt;

&lt;p&gt;An existing OpenAI SDK client can communicate with the gateway:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'openai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;baseURL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;LITELLM_BASE_URL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;apiKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;LITELLM_VIRTUAL_KEY&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'fast-chat'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'user'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Explain the benefits of model routing.'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The application uses the logical model name &lt;code&gt;fast-chat&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It does not need to know whether the request is handled by OpenAI or another configured provider.&lt;/p&gt;

&lt;p&gt;For production, give each application or workload a dedicated virtual key with the minimum required model access and usage limits.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://docs.litellm.ai/docs/proxy/docker_quick_start" rel="noopener noreferrer"&gt;LiteLLM Quickstart&lt;/a&gt; · &lt;a href="https://docs.litellm.ai/docs/router/api_reference/configuration" rel="noopener noreferrer"&gt;LiteLLM Router Configuration&lt;/a&gt; · &lt;a href="https://docs.litellm.ai/docs/proxy/reliability" rel="noopener noreferrer"&gt;LiteLLM Fallbacks&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Choosing an AI Gateway
&lt;/h2&gt;

&lt;p&gt;Several platforms are worth evaluating, but they do not all have the same deployment model or intended use case.&lt;/p&gt;

&lt;h3&gt;
  
  
  Vercel AI Gateway
&lt;/h3&gt;

&lt;p&gt;A managed model gateway with close integration into the Vercel AI SDK.&lt;/p&gt;

&lt;p&gt;Its capabilities include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Multi-provider model access.&lt;/li&gt;
&lt;li&gt;Provider ordering and filtering.&lt;/li&gt;
&lt;li&gt;Model fallbacks.&lt;/li&gt;
&lt;li&gt;BYOK.&lt;/li&gt;
&lt;li&gt;OIDC authentication.&lt;/li&gt;
&lt;li&gt;Usage reporting.&lt;/li&gt;
&lt;li&gt;Team, project, key, and user budgets.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is attractive when teams want a managed gateway without operating the gateway infrastructure themselves.&lt;/p&gt;

&lt;h3&gt;
  
  
  LiteLLM
&lt;/h3&gt;

&lt;p&gt;An open-source Python-based gateway with self-hosting and managed deployment options.&lt;/p&gt;

&lt;p&gt;Its capabilities include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Broad provider integrations.&lt;/li&gt;
&lt;li&gt;Model aliases.&lt;/li&gt;
&lt;li&gt;Routing and load balancing.&lt;/li&gt;
&lt;li&gt;Retries and fallbacks.&lt;/li&gt;
&lt;li&gt;Virtual keys.&lt;/li&gt;
&lt;li&gt;Team and key budgets.&lt;/li&gt;
&lt;li&gt;Rate limiting.&lt;/li&gt;
&lt;li&gt;Usage tracking.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is particularly useful when an organization needs control over its model access infrastructure.&lt;/p&gt;

&lt;p&gt;The database requirement for features such as virtual keys and spend tracking should be included in capacity and operations planning.&lt;/p&gt;

&lt;h3&gt;
  
  
  Portkey
&lt;/h3&gt;

&lt;p&gt;An AI gateway platform with a self-hostable open-source gateway and managed offerings.&lt;/p&gt;

&lt;p&gt;Its gateway capabilities include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Provider routing.&lt;/li&gt;
&lt;li&gt;Fallbacks and retries.&lt;/li&gt;
&lt;li&gt;Load balancing.&lt;/li&gt;
&lt;li&gt;Guardrails.&lt;/li&gt;
&lt;li&gt;Request management.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Teams should evaluate which advanced logging, retention, and compliance capabilities are available in their chosen deployment and plan.&lt;/p&gt;

&lt;h3&gt;
  
  
  Envoy AI Gateway
&lt;/h3&gt;

&lt;p&gt;An open-source gateway built around Envoy.&lt;/p&gt;

&lt;p&gt;It is particularly relevant to teams already operating Kubernetes, Envoy Proxy, or service mesh infrastructure.&lt;/p&gt;

&lt;p&gt;Its focus includes model-aware routing, upstream authorization, failover, and gateway-level traffic policies.&lt;/p&gt;

&lt;p&gt;It can be a natural choice when AI traffic management needs to become part of an existing platform networking architecture.&lt;/p&gt;

&lt;h3&gt;
  
  
  OpenRouter
&lt;/h3&gt;

&lt;p&gt;OpenRouter is best understood as a managed model-routing and provider aggregation platform.&lt;/p&gt;

&lt;p&gt;It provides access to models through a unified interface and supports provider ordering, filtering, price-oriented routing, latency and throughput preferences, and fallback controls.&lt;/p&gt;

&lt;p&gt;It is useful when broad model access and managed provider routing are the primary requirements.&lt;/p&gt;

&lt;p&gt;It is not the same deployment model as a self-hosted gateway such as LiteLLM or Envoy AI Gateway.&lt;/p&gt;

&lt;h3&gt;
  
  
  High-Level Comparison
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Platform&lt;/th&gt;
&lt;th&gt;Deployment model&lt;/th&gt;
&lt;th&gt;Main strength&lt;/th&gt;
&lt;th&gt;Important consideration&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Vercel AI Gateway&lt;/td&gt;
&lt;td&gt;Managed&lt;/td&gt;
&lt;td&gt;AI SDK integration and managed operations&lt;/td&gt;
&lt;td&gt;Provider and platform dependency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LiteLLM&lt;/td&gt;
&lt;td&gt;Open source / managed&lt;/td&gt;
&lt;td&gt;Self-hosting, provider coverage, virtual keys&lt;/td&gt;
&lt;td&gt;Database and operational ownership&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Portkey&lt;/td&gt;
&lt;td&gt;Open source / managed&lt;/td&gt;
&lt;td&gt;Routing and guardrails&lt;/td&gt;
&lt;td&gt;Hosted versus self-hosted feature boundaries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Envoy AI Gateway&lt;/td&gt;
&lt;td&gt;Self-hosted open source&lt;/td&gt;
&lt;td&gt;Kubernetes and Envoy integration&lt;/td&gt;
&lt;td&gt;More infrastructure to operate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenRouter&lt;/td&gt;
&lt;td&gt;Managed&lt;/td&gt;
&lt;td&gt;Broad model and provider routing&lt;/td&gt;
&lt;td&gt;Less control over the gateway infrastructure itself&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is a deployment-oriented comparison rather than a complete feature checklist.&lt;/p&gt;

&lt;p&gt;Feature availability, licensing, pricing, and hosted-plan boundaries change over time. Validate them against the current documentation and your own requirements.&lt;/p&gt;

&lt;p&gt;I would also benchmark candidate gateways against representative traffic instead of relying on published performance claims.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://vercel.com/i/open-source-ai-gateways" rel="noopener noreferrer"&gt;Vercel — Open Source AI Gateways Compared&lt;/a&gt; · &lt;a href="https://portkey.ai/docs/" rel="noopener noreferrer"&gt;Portkey Documentation&lt;/a&gt; · &lt;a href="https://github.com/envoyproxy/ai-gateway" rel="noopener noreferrer"&gt;Envoy AI Gateway&lt;/a&gt; · &lt;a href="https://openrouter.ai/blog/insights/model-routing/" rel="noopener noreferrer"&gt;OpenRouter Routing&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  12. Why an OpenAI-Compatible Interface Is Not Enough
&lt;/h2&gt;

&lt;p&gt;An OpenAI-compatible API is useful, but compatibility should not be confused with behavioral equivalence.&lt;/p&gt;

&lt;p&gt;Different models and providers may implement the same feature differently.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tool-calling arguments may be represented differently.&lt;/li&gt;
&lt;li&gt;Structured output constraints may vary.&lt;/li&gt;
&lt;li&gt;Streaming events may contain different metadata.&lt;/li&gt;
&lt;li&gt;Reasoning parameters may not map directly.&lt;/li&gt;
&lt;li&gt;Some providers may not support a required input modality.&lt;/li&gt;
&lt;li&gt;Context windows may differ.&lt;/li&gt;
&lt;li&gt;Cancellation and timeout behavior may differ.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A gateway can normalize the common protocol, but application behavior must still account for provider differences.&lt;/p&gt;

&lt;h3&gt;
  
  
  Keep Model Identity Visible
&lt;/h3&gt;

&lt;p&gt;Even when the application uses a logical model alias, it should remain possible to inspect:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Requested model.&lt;/li&gt;
&lt;li&gt;Actual model selected.&lt;/li&gt;
&lt;li&gt;Upstream provider.&lt;/li&gt;
&lt;li&gt;Fallback attempts.&lt;/li&gt;
&lt;li&gt;Provider-specific metadata.&lt;/li&gt;
&lt;li&gt;Usage and pricing information.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is essential for debugging, performance analysis, evaluation, and reproducibility.&lt;/p&gt;

&lt;h3&gt;
  
  
  Test Fallback Compatibility
&lt;/h3&gt;

&lt;p&gt;For every important logical model group, define a compatibility test suite.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Why it matters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Text generation&lt;/td&gt;
&lt;td&gt;Basic response compatibility&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Streaming&lt;/td&gt;
&lt;td&gt;Event and completion behavior&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool calling&lt;/td&gt;
&lt;td&gt;Agent execution compatibility&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Structured output&lt;/td&gt;
&lt;td&gt;Schema compliance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Multimodal input&lt;/td&gt;
&lt;td&gt;Required input support&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Usage reporting&lt;/td&gt;
&lt;td&gt;Cost attribution&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cancellation&lt;/td&gt;
&lt;td&gt;Correct failure handling&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A fallback that produces a successful HTTP response but cannot execute the next step of an agent workflow is not a useful fallback.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The right abstraction hides integration complexity without hiding operational reality.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  13. Cost-Aware Routing and Usage Attribution
&lt;/h2&gt;

&lt;p&gt;An AI Gateway provides a central point for understanding where model consumption originates.&lt;/p&gt;

&lt;p&gt;However, usage tracking alone does not optimize costs.&lt;/p&gt;

&lt;p&gt;We need to associate usage with the workload that generated it.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AI Platform Usage
|
+-- Development Assistant
|   |
|   +-- Code Review
|   +-- Code Generation
|   +-- Test Generation
|
+-- Research Platform
|   |
|   +-- Document Analysis
|   +-- Market Research
|
+-- Automation
    |
    +-- Data Extraction
    +-- Classification
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A useful usage model includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Application.&lt;/li&gt;
&lt;li&gt;Team.&lt;/li&gt;
&lt;li&gt;User or service identity.&lt;/li&gt;
&lt;li&gt;Feature.&lt;/li&gt;
&lt;li&gt;Model.&lt;/li&gt;
&lt;li&gt;Provider.&lt;/li&gt;
&lt;li&gt;Input tokens.&lt;/li&gt;
&lt;li&gt;Output tokens.&lt;/li&gt;
&lt;li&gt;Cached tokens.&lt;/li&gt;
&lt;li&gt;Request count.&lt;/li&gt;
&lt;li&gt;Total cost.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Basic Model Cost Calculation
&lt;/h3&gt;

&lt;p&gt;A simplified model cost formula:&lt;/p&gt;

&lt;p&gt;Model cost = Σ (Input tokens × Input price + Output tokens × Output price)&lt;/p&gt;

&lt;p&gt;Where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;(T) represents token usage.&lt;/li&gt;
&lt;li&gt;(P) represents the applicable price per token.&lt;/li&gt;
&lt;li&gt;(i) represents an individual model operation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A production implementation should also account for cached input, multimodal usage, provider-specific charges, and other billable operations.&lt;/p&gt;

&lt;p&gt;For an agent workflow, we should aggregate usage across all model calls rather than measuring only the final response.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cost Per Successful Task
&lt;/h3&gt;

&lt;p&gt;A model that costs less per request is not necessarily the better option.&lt;/p&gt;

&lt;p&gt;Consider:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Model A&lt;/th&gt;
&lt;th&gt;Model B&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Cost per attempt&lt;/td&gt;
&lt;td&gt;$0.04&lt;/td&gt;
&lt;td&gt;$0.07&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Successful completion rate&lt;/td&gt;
&lt;td&gt;90%&lt;/td&gt;
&lt;td&gt;98%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost per successful task&lt;/td&gt;
&lt;td&gt;~$0.044&lt;/td&gt;
&lt;td&gt;~$0.071&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The cheaper model is more economical in this example, but this does not account for differences in quality, latency, or the business impact of failed tasks.&lt;/p&gt;

&lt;p&gt;A useful metric is:&lt;/p&gt;

&lt;p&gt;Cost per successful task = Total execution cost / Successful tasks&lt;/p&gt;

&lt;p&gt;This should include failed attempts that consumed resources.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cost Optimization Strategies
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Use smaller models for predictable, low-complexity tasks.&lt;/li&gt;
&lt;li&gt;Use stronger models when the task requires them.&lt;/li&gt;
&lt;li&gt;Avoid unnecessary model calls.&lt;/li&gt;
&lt;li&gt;Measure fallback frequency.&lt;/li&gt;
&lt;li&gt;Use prompt caching where appropriate.&lt;/li&gt;
&lt;li&gt;Define per-workload quotas.&lt;/li&gt;
&lt;li&gt;Introduce evaluation-based escalation when appropriate.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Cost-aware routing should never become a reason to silently downgrade critical operations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://vercel.com/kb/guide/cost-aware-model-routing-with-ai-gateway" rel="noopener noreferrer"&gt;Vercel — Cost-Aware Model Routing&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  14. Security and Governance
&lt;/h2&gt;

&lt;p&gt;An AI Gateway becomes a sensitive infrastructure component because it controls access to external model providers and processes potentially confidential application data.&lt;/p&gt;

&lt;p&gt;Several security requirements should be considered.&lt;/p&gt;

&lt;h3&gt;
  
  
  Centralized Credential Management
&lt;/h3&gt;

&lt;p&gt;Applications should authenticate to the gateway using dedicated application credentials.&lt;/p&gt;

&lt;p&gt;Upstream provider keys should be stored in an approved secret-management system or the gateway's secure credential store.&lt;/p&gt;

&lt;h3&gt;
  
  
  Application-Specific Identities
&lt;/h3&gt;

&lt;p&gt;Avoid sharing one unrestricted administrator credential across every application.&lt;/p&gt;

&lt;p&gt;Use separate service identities, virtual keys, or other supported workload credentials.&lt;/p&gt;

&lt;h3&gt;
  
  
  Model Allowlists
&lt;/h3&gt;

&lt;p&gt;Applications should only access approved models and providers.&lt;/p&gt;

&lt;p&gt;A development assistant should not automatically have access to every production model deployment.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data Residency and Provider Policies
&lt;/h3&gt;

&lt;p&gt;Provider selection must consider more than cost and latency.&lt;/p&gt;

&lt;p&gt;Some workloads may require:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Specific regions.&lt;/li&gt;
&lt;li&gt;Approved providers.&lt;/li&gt;
&lt;li&gt;Zero Data Retention arrangements.&lt;/li&gt;
&lt;li&gt;Restrictions on provider data usage.&lt;/li&gt;
&lt;li&gt;Internal-only inference endpoints.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Where supported, apply these constraints at the gateway layer.&lt;/p&gt;

&lt;p&gt;However, a gateway's routing policy does not replace verification of the provider's contractual and data-handling commitments.&lt;/p&gt;

&lt;h3&gt;
  
  
  Protect the Gateway Itself
&lt;/h3&gt;

&lt;p&gt;A gateway should have:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Authenticated administrative endpoints.&lt;/li&gt;
&lt;li&gt;Restricted network access.&lt;/li&gt;
&lt;li&gt;Secure credential storage.&lt;/li&gt;
&lt;li&gt;Request validation.&lt;/li&gt;
&lt;li&gt;Rate limits.&lt;/li&gt;
&lt;li&gt;Availability monitoring.&lt;/li&gt;
&lt;li&gt;Backup and recovery procedures.&lt;/li&gt;
&lt;li&gt;Controlled configuration changes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Since multiple applications depend on it, the gateway is itself a critical infrastructure service.&lt;/p&gt;

&lt;h2&gt;
  
  
  15. Operational Considerations: Avoid Creating a New Bottleneck
&lt;/h2&gt;

&lt;p&gt;Centralizing model access simplifies application integrations, but it also introduces a shared dependency.&lt;/p&gt;

&lt;p&gt;Every application now relies on the gateway to communicate with providers.&lt;/p&gt;

&lt;p&gt;Several operational considerations follow.&lt;/p&gt;

&lt;h3&gt;
  
  
  High Availability
&lt;/h3&gt;

&lt;p&gt;For critical workloads, use multiple gateway instances behind an appropriate load balancer.&lt;/p&gt;

&lt;p&gt;Avoid relying on one process or in-memory state for shared configuration and quota enforcement.&lt;/p&gt;

&lt;h3&gt;
  
  
  Shared Rate-Limit State
&lt;/h3&gt;

&lt;p&gt;If multiple gateway instances enforce the same team or application quota, they must share consistent usage state.&lt;/p&gt;

&lt;p&gt;Otherwise, each instance may independently allow requests and exceed the intended limit.&lt;/p&gt;

&lt;h3&gt;
  
  
  Provider Health Management
&lt;/h3&gt;

&lt;p&gt;Monitor upstream providers and deployments.&lt;/p&gt;

&lt;p&gt;Temporarily exclude unhealthy deployments when appropriate, but avoid aggressive health-check behavior that creates additional provider load.&lt;/p&gt;

&lt;h3&gt;
  
  
  Configuration Management
&lt;/h3&gt;

&lt;p&gt;Model aliases, provider priorities, budgets, and access policies are production configuration.&lt;/p&gt;

&lt;p&gt;Version and review changes where possible.&lt;/p&gt;

&lt;p&gt;A provider change should be treated with the same care as a change to an external service dependency.&lt;/p&gt;

&lt;h3&gt;
  
  
  Gateway Observability
&lt;/h3&gt;

&lt;p&gt;Track:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Gateway request latency.&lt;/li&gt;
&lt;li&gt;Upstream provider latency.&lt;/li&gt;
&lt;li&gt;Provider failure rate.&lt;/li&gt;
&lt;li&gt;Fallback frequency.&lt;/li&gt;
&lt;li&gt;Model selection distribution.&lt;/li&gt;
&lt;li&gt;Budget rejections.&lt;/li&gt;
&lt;li&gt;Queue and connection utilization.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This helps distinguish gateway overhead from upstream inference latency.&lt;/p&gt;

&lt;h3&gt;
  
  
  Understand the Additional Failure Domain
&lt;/h3&gt;

&lt;p&gt;A gateway adds another service to operate.&lt;/p&gt;

&lt;p&gt;Teams should evaluate its:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Availability.&lt;/li&gt;
&lt;li&gt;Throughput.&lt;/li&gt;
&lt;li&gt;Network overhead.&lt;/li&gt;
&lt;li&gt;Connection limits.&lt;/li&gt;
&lt;li&gt;Capacity requirements.&lt;/li&gt;
&lt;li&gt;Failure isolation.&lt;/li&gt;
&lt;li&gt;Scaling behavior.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The gateway should simplify model operations without becoming a single point of failure.&lt;/p&gt;

&lt;h2&gt;
  
  
  16. What Can an AI Chapter Lead Contribute?
&lt;/h2&gt;

&lt;p&gt;When multiple teams use different models and providers independently, infrastructure complexity grows quickly.&lt;/p&gt;

&lt;p&gt;Each team may implement its own credentials, model configurations, retries, usage reporting, and provider policies.&lt;/p&gt;

&lt;p&gt;This creates duplicated engineering effort and inconsistent governance.&lt;/p&gt;

&lt;p&gt;An AI Chapter Lead can turn model access into a reusable platform capability.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Establish a Shared Model Access Layer
&lt;/h3&gt;

&lt;p&gt;Define a common gateway endpoint for AI-powered applications.&lt;/p&gt;

&lt;p&gt;It should standardize:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Provider integration.&lt;/li&gt;
&lt;li&gt;Model aliases.&lt;/li&gt;
&lt;li&gt;Routing.&lt;/li&gt;
&lt;li&gt;Fallback policies.&lt;/li&gt;
&lt;li&gt;Authentication.&lt;/li&gt;
&lt;li&gt;Usage tracking.&lt;/li&gt;
&lt;li&gt;Access control.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Feature teams should focus on application behavior rather than managing provider-specific infrastructure.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Build an Internal Model Catalog
&lt;/h3&gt;

&lt;p&gt;Maintain an approved registry of model deployments.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Logical model&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;th&gt;Capabilities&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;fast-chat&lt;/td&gt;
&lt;td&gt;Classification and simple tasks&lt;/td&gt;
&lt;td&gt;Text, structured output&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;reasoning-chat&lt;/td&gt;
&lt;td&gt;Complex analysis&lt;/td&gt;
&lt;td&gt;Text, tool calling&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;coding-model&lt;/td&gt;
&lt;td&gt;Code generation and review&lt;/td&gt;
&lt;td&gt;Text, tool calling&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;vision-model&lt;/td&gt;
&lt;td&gt;Image analysis&lt;/td&gt;
&lt;td&gt;Multimodal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;embedding-model&lt;/td&gt;
&lt;td&gt;Semantic retrieval&lt;/td&gt;
&lt;td&gt;Embeddings&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each definition should also contain provider information, model version, supported capabilities, ownership, and access restrictions.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Define Shared Routing Policies
&lt;/h3&gt;

&lt;p&gt;Provide reusable routing profiles:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Cost-optimized.&lt;/li&gt;
&lt;li&gt;Latency-sensitive.&lt;/li&gt;
&lt;li&gt;Reasoning-focused.&lt;/li&gt;
&lt;li&gt;Coding-focused.&lt;/li&gt;
&lt;li&gt;Multimodal.&lt;/li&gt;
&lt;li&gt;Region-restricted.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Teams can select an approved routing profile rather than independently implementing provider-specific decisions.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Centralize Access Governance
&lt;/h3&gt;

&lt;p&gt;Introduce application-specific credentials and model allowlists.&lt;/p&gt;

&lt;p&gt;Teams should only access the models and providers approved for their use cases.&lt;/p&gt;

&lt;p&gt;This also creates a foundation for usage attribution and auditing.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Standardize Model Compatibility Testing
&lt;/h3&gt;

&lt;p&gt;Build automated tests that verify:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Streaming behavior.&lt;/li&gt;
&lt;li&gt;Tool calling.&lt;/li&gt;
&lt;li&gt;Structured output.&lt;/li&gt;
&lt;li&gt;Multimodal capabilities.&lt;/li&gt;
&lt;li&gt;Cancellation.&lt;/li&gt;
&lt;li&gt;Usage reporting.&lt;/li&gt;
&lt;li&gt;Failure handling.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This becomes particularly important when introducing new models or changing fallback configurations.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Introduce a Model Promotion Process
&lt;/h3&gt;

&lt;p&gt;A new model should not automatically become the default for every application.&lt;/p&gt;

&lt;p&gt;A promotion process could look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;New Model Available
        |
        v
Capability Validation
        |
        v
Offline Evaluation
        |
        v
Cost / Latency Benchmark
        |
        v
Security Review
        |
        v
Limited Traffic
        |
        v
Production Approval
        |
        v
Model Catalog Update
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This allows the organization to introduce new models without uncontrolled changes to existing applications.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Separate Platform Ownership from Application Ownership
&lt;/h3&gt;

&lt;p&gt;The platform team should own:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Gateway infrastructure.&lt;/li&gt;
&lt;li&gt;Provider integration.&lt;/li&gt;
&lt;li&gt;Credentials.&lt;/li&gt;
&lt;li&gt;Routing engine.&lt;/li&gt;
&lt;li&gt;Global access policies.&lt;/li&gt;
&lt;li&gt;Platform availability.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Application teams should own:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Business logic.&lt;/li&gt;
&lt;li&gt;Prompt and agent behavior.&lt;/li&gt;
&lt;li&gt;Model requirements.&lt;/li&gt;
&lt;li&gt;Application-level evaluation.&lt;/li&gt;
&lt;li&gt;Workflow-specific constraints.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This creates a clearer responsibility boundary.&lt;/p&gt;

&lt;h2&gt;
  
  
  17. When Should We Build Our Own AI Gateway?
&lt;/h2&gt;

&lt;p&gt;Not every organization needs a custom gateway.&lt;/p&gt;

&lt;p&gt;For one application using one provider, adding a separate infrastructure component may not provide enough value.&lt;/p&gt;

&lt;p&gt;A direct provider integration or managed gateway may be simpler.&lt;/p&gt;

&lt;p&gt;A dedicated gateway becomes more attractive when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Multiple applications use different model providers.&lt;/li&gt;
&lt;li&gt;Different teams need centralized access control.&lt;/li&gt;
&lt;li&gt;Provider credentials need centralized management.&lt;/li&gt;
&lt;li&gt;Cost attribution is important.&lt;/li&gt;
&lt;li&gt;Fallback and availability are business requirements.&lt;/li&gt;
&lt;li&gt;Internal model deployments coexist with external providers.&lt;/li&gt;
&lt;li&gt;Data governance requires provider restrictions.&lt;/li&gt;
&lt;li&gt;Model routing policies need to evolve independently of applications.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is also an important distinction between customizing an existing gateway and building one from scratch.&lt;/p&gt;

&lt;p&gt;I would start with an existing gateway, validate its capabilities, and identify the remaining requirements.&lt;/p&gt;

&lt;p&gt;Build custom infrastructure only when there is a clear requirement that existing solutions cannot satisfy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;As AI adoption grows, managing model access independently in every application becomes increasingly difficult.&lt;/p&gt;

&lt;p&gt;An AI Gateway provides a shared infrastructure layer for provider abstraction, model routing, authentication, fallback handling, usage policies, and cost management.&lt;/p&gt;

&lt;p&gt;Combined with an application-level SDK such as Vercel AI SDK, it allows development teams to focus on building AI-powered features while the platform manages model connectivity.&lt;/p&gt;

&lt;p&gt;But a gateway should not create the illusion that all models are interchangeable.&lt;/p&gt;

&lt;p&gt;Provider differences, capability compatibility, streaming behavior, data governance, and failure semantics still matter.&lt;/p&gt;

&lt;p&gt;For engineering teams, the opportunity is to turn model access into a standardized internal platform capability.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;My recommendation:&lt;/strong&gt; Start with a small model catalog, introduce application-specific identities, and define explicit routing and fallback policies. Then validate provider compatibility and budget behavior before expanding the gateway across teams.&lt;/p&gt;

&lt;p&gt;The goal is not to create another proxy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It is to make model access predictable, governed, and independent of individual applications.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  References and Further Reading
&lt;/h2&gt;

&lt;h3&gt;
  
  
  AI Gateway and AI SDK
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://ai-sdk.dev/providers/ai-sdk-providers/ai-gateway" rel="noopener noreferrer"&gt;AI SDK — AI Gateway Provider&lt;/a&gt; — Current provider integration, model discovery, routing, and fallbacks.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/ai-gateway" rel="noopener noreferrer"&gt;Vercel AI Gateway&lt;/a&gt; — Managed model access infrastructure.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/docs/ai-gateway/models-and-providers/provider-options" rel="noopener noreferrer"&gt;Provider Options&lt;/a&gt; — Provider ordering, filtering, and routing.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/docs/ai-gateway/models-and-providers/model-fallbacks" rel="noopener noreferrer"&gt;Model Fallbacks&lt;/a&gt; — Model-level failover.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/docs/ai-gateway/authentication-and-byok" rel="noopener noreferrer"&gt;Authentication and BYOK&lt;/a&gt; — API keys, OIDC, and provider credentials.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/docs/ai-gateway/observability-and-spend/budgets" rel="noopener noreferrer"&gt;Budgets and Spend Limits&lt;/a&gt; — Team, project, API key, and user budgets.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/kb/guide/cost-aware-model-routing-with-ai-gateway" rel="noopener noreferrer"&gt;Cost-Aware Model Routing&lt;/a&gt; — Model selection based on task complexity and cost.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  LiteLLM
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://docs.litellm.ai/" rel="noopener noreferrer"&gt;LiteLLM Documentation&lt;/a&gt; — Gateway architecture and provider support.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.litellm.ai/docs/proxy/docker_quick_start" rel="noopener noreferrer"&gt;LiteLLM Docker Quickstart&lt;/a&gt; — PostgreSQL-backed gateway setup.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.litellm.ai/docs/router/api_reference/configuration" rel="noopener noreferrer"&gt;Router Configuration&lt;/a&gt; — Model groups, retries, and fallbacks.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.litellm.ai/docs/proxy/reliability" rel="noopener noreferrer"&gt;Fallbacks and Reliability&lt;/a&gt; — Provider failover and timeout handling.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.litellm.ai/docs/proxy/users" rel="noopener noreferrer"&gt;Budgets and Rate Limits&lt;/a&gt; — Virtual keys, team budgets, and usage limits.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Other Gateway Platforms
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://portkey.ai/docs/" rel="noopener noreferrer"&gt;Portkey Documentation&lt;/a&gt; — Routing, reliability, and gateway configuration.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/envoyproxy/ai-gateway" rel="noopener noreferrer"&gt;Envoy AI Gateway&lt;/a&gt; — Open-source gateway architecture for Envoy-based infrastructure.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://openrouter.ai/blog/insights/model-routing/" rel="noopener noreferrer"&gt;OpenRouter — Model Routing&lt;/a&gt; — Provider selection, routing, and failover.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://openrouter.ai/docs" rel="noopener noreferrer"&gt;OpenRouter Documentation&lt;/a&gt; — Model and provider integration.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/i/open-source-ai-gateways" rel="noopener noreferrer"&gt;Open Source AI Gateways Compared&lt;/a&gt; — A comparative overview of LiteLLM, Portkey, Envoy AI Gateway, and Bifrost. This is a Vercel-published comparison and should be treated as one input, not an independent benchmark.&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>productivity</category>
      <category>programming</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Transactional Outbox in Go: Reliable Event Publishing Without Distributed Transactions</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Thu, 19 Nov 2026 17:15:20 +0000</pubDate>
      <link>https://dev.to/serifcolakel/transactional-outbox-in-go-reliable-event-publishing-without-distributed-transactions-1dol</link>
      <guid>https://dev.to/serifcolakel/transactional-outbox-in-go-reliable-event-publishing-without-distributed-transactions-1dol</guid>
      <description>&lt;p&gt;Your API creates an order and commits it to PostgreSQL.&lt;/p&gt;

&lt;p&gt;The next step is publishing an &lt;code&gt;OrderCreated&lt;/code&gt; event to Kafka so that inventory, notifications, and analytics services can react.&lt;/p&gt;

&lt;p&gt;You write something like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nc"&gt;OrderService&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;CreateOrder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="nc"&gt;Order&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;publisher&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Publish&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;OrderCreated&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;OrderID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It looks straightforward.&lt;/p&gt;

&lt;p&gt;But there is a failure window.&lt;/p&gt;

&lt;p&gt;What happens if PostgreSQL commits the order successfully and Kafka is unavailable?&lt;/p&gt;

&lt;p&gt;The order exists, but no event is published.&lt;/p&gt;

&lt;p&gt;What if Kafka accepts the event, but your application crashes before recording that it was published?&lt;/p&gt;

&lt;p&gt;Your application might publish the same event again after restarting.&lt;/p&gt;

&lt;p&gt;These are not unusual edge cases. They are consequences of coordinating independent systems without a shared transaction.&lt;/p&gt;

&lt;p&gt;The transactional outbox pattern addresses this problem by storing the business change and the event to be published in the same database transaction.&lt;/p&gt;

&lt;p&gt;Instead of trying to atomically commit to PostgreSQL and Kafka, we make the database transaction the source of truth and publish events asynchronously.&lt;/p&gt;

&lt;p&gt;In this article, we'll build a PostgreSQL-backed transactional outbox in Go, implement concurrent workers, handle retries and leases, and explore the delivery guarantees that matter in production.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. The Dual-Write Problem
&lt;/h2&gt;

&lt;p&gt;A dual write happens when an application needs to update two independent systems as part of one logical operation.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HTTP Request
     |
     v
Create Order in PostgreSQL
     |
     v
Publish OrderCreated to Kafka
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There are two possible execution orders.&lt;/p&gt;

&lt;h3&gt;
  
  
  Database first
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. PostgreSQL commit succeeds
2. Kafka publish fails
3. Application returns an error
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The order exists, but downstream services never receive the event.&lt;/p&gt;

&lt;p&gt;Retrying the entire HTTP request might create another order unless the endpoint is idempotent.&lt;/p&gt;

&lt;h3&gt;
  
  
  Message broker first
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. Kafka publish succeeds
2. PostgreSQL commit fails
3. Application returns an error
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now downstream services have received an event for an order that does not exist.&lt;/p&gt;

&lt;p&gt;Neither order solves the problem.&lt;/p&gt;

&lt;p&gt;A database transaction cannot automatically include a Kafka publish. They are independent systems with independent failure and commit semantics.&lt;/p&gt;

&lt;p&gt;Distributed transactions and two-phase commit are possible in some environments, but they introduce operational and architectural complexity.&lt;/p&gt;

&lt;p&gt;The transactional outbox pattern takes a different approach.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. How the Transactional Outbox Works
&lt;/h2&gt;

&lt;p&gt;The central idea is simple:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Store the business change and the event in the same database transaction.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Instead of publishing directly to the broker, insert an event into an &lt;code&gt;outbox_events&lt;/code&gt; table.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 HTTP Request
                      |
                      v
               PostgreSQL TX
                      |
              +-------+-------+
              |               |
              v               v
          Create Order    Insert Event
              |               |
              +-------+-------+
                      |
                    COMMIT
                      |
                      v
                 Outbox Worker
                      |
                      v
                 Message Broker
                      |
                      v
              Downstream Consumers
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the transaction commits, both the order and the outbox event exist.&lt;/p&gt;

&lt;p&gt;If the transaction rolls back, neither exists.&lt;/p&gt;

&lt;p&gt;This eliminates the inconsistency between the business write and the durable record of the event that must be published.&lt;/p&gt;

&lt;p&gt;It does not make publishing exactly once. We'll discuss that distinction later.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Designing the Outbox Table
&lt;/h2&gt;

&lt;p&gt;Let's start with a PostgreSQL schema.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="kt"&gt;BIGSERIAL&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="n"&gt;event_id&lt;/span&gt; &lt;span class="kt"&gt;UUID&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;UNIQUE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="n"&gt;aggregate_type&lt;/span&gt; &lt;span class="kt"&gt;TEXT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;aggregate_id&lt;/span&gt; &lt;span class="kt"&gt;TEXT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;event_type&lt;/span&gt; &lt;span class="kt"&gt;TEXT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="kt"&gt;JSONB&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="n"&gt;created_at&lt;/span&gt; &lt;span class="kt"&gt;TIMESTAMPTZ&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="kt"&gt;TIMESTAMPTZ&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="n"&gt;attempts&lt;/span&gt; &lt;span class="kt"&gt;INTEGER&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;next_attempt_at&lt;/span&gt; &lt;span class="kt"&gt;TIMESTAMPTZ&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="kt"&gt;UUID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;lease_until&lt;/span&gt; &lt;span class="kt"&gt;TIMESTAMPTZ&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="n"&gt;last_error&lt;/span&gt; &lt;span class="kt"&gt;TEXT&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;idx_outbox_pending&lt;/span&gt;
    &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;next_attempt_at&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Let's look at the important fields.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Field&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;event_id&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Stable identifier used for deduplication&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;aggregate_type&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Business entity type&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;aggregate_id&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Entity identifier&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;event_type&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Event name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;payload&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Serialized event data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;published_at&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Publication completion timestamp&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;attempts&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Number of publication attempts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;next_attempt_at&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Earliest time the event can be retried&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;claim_token&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Identifies the worker currently holding the lease&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;lease_until&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;When another worker can reclaim the event&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;last_error&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Most recent publication error&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Why do we need both &lt;code&gt;claim_token&lt;/code&gt; and &lt;code&gt;lease_until&lt;/code&gt;?&lt;/p&gt;

&lt;p&gt;A worker can crash after claiming an event. Without a lease, the event might remain stuck forever.&lt;/p&gt;

&lt;p&gt;A lease makes the event available again after a bounded period.&lt;/p&gt;

&lt;p&gt;The claim token prevents an old worker from updating a row after another worker has reclaimed it.&lt;/p&gt;

&lt;p&gt;This is a form of ownership fencing for database updates. It does not prevent an expired worker from publishing to the broker, so consumers still need deduplication.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Writing the Business Change and Event Atomically
&lt;/h2&gt;

&lt;p&gt;The outbox only works if the business write and event insertion share the same transaction.&lt;/p&gt;

&lt;p&gt;Let's define an event:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;outbox&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;"context"&lt;/span&gt;
    &lt;span class="s2"&gt;"encoding/json"&lt;/span&gt;
    &lt;span class="s2"&gt;"fmt"&lt;/span&gt;
    &lt;span class="s2"&gt;"time"&lt;/span&gt;

    &lt;span class="s2"&gt;"github.com/google/uuid"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;Event&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="no"&gt;ID&lt;/span&gt;            &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UUID&lt;/span&gt;
    &lt;span class="nc"&gt;AggregateType&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;
    &lt;span class="nc"&gt;AggregateID&lt;/span&gt;   &lt;span class="kt"&gt;string&lt;/span&gt;
    &lt;span class="nc"&gt;Type&lt;/span&gt;          &lt;span class="kt"&gt;string&lt;/span&gt;
    &lt;span class="nc"&gt;Payload&lt;/span&gt;       &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RawMessage&lt;/span&gt;
    &lt;span class="nc"&gt;CreatedAt&lt;/span&gt;     &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Time&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A repository method can insert the event using the same &lt;code&gt;sql.Tx&lt;/code&gt; as the business operation.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nc"&gt;Insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;tx&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Tx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sb"&gt;`
        INSERT INTO outbox_events (
            event_id,
            aggregate_type,
            aggregate_id,
            event_type,
            payload,
            created_at
        )
        VALUES ($1, $2, $3, $4, $5, $6)
    `&lt;/span&gt;

    &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ExecContext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;AggregateType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;AggregateID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;CreatedAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"insert outbox event: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now let's use it in an order service.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nc"&gt;OrderService&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;Create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt; &lt;span class="nc"&gt;CreateOrderRequest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BeginTx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"begin transaction: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Rollback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nc"&gt;Order&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;        &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NewString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;Customer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;CustomerID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;Amount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;    &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;CreatedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UTC&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;orders&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"insert order: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Marshal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;OrderCreated&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nc"&gt;OrderID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;Customer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Customer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;Amount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;    &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;CreatedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;CreatedAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"marshal event: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;outbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;            &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;AggregateType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"order"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;AggregateID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;Type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;          &lt;span class="s2"&gt;"order.created"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;Payload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;       &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nc"&gt;CreatedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;     &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UTC&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;outbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Insert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"insert event: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Commit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"commit order transaction: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is no Kafka call inside this transaction.&lt;/p&gt;

&lt;p&gt;That is intentional.&lt;/p&gt;

&lt;p&gt;The transaction only guarantees that the order and its event record are persisted together.&lt;/p&gt;

&lt;p&gt;The actual event publication happens asynchronously.&lt;/p&gt;

&lt;h3&gt;
  
  
  What if the commit response is lost?
&lt;/h3&gt;

&lt;p&gt;Suppose PostgreSQL commits successfully, but the connection drops before the application receives the commit response.&lt;/p&gt;

&lt;p&gt;The application might not know whether the transaction succeeded.&lt;/p&gt;

&lt;p&gt;This is another reason the HTTP operation should be idempotent.&lt;/p&gt;

&lt;p&gt;A retry should not blindly create another order and another event. Use the idempotency mechanism from the previous article, or a durable business uniqueness constraint, to recognize the logical operation.&lt;/p&gt;

&lt;p&gt;The outbox guarantees atomic persistence of the event record with the business write. It does not replace request-level idempotency.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Claiming Events With Multiple Workers
&lt;/h2&gt;

&lt;p&gt;A single worker can process pending events, but production systems often need multiple workers.&lt;/p&gt;

&lt;p&gt;Imagine three worker instances:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Worker A
Worker B
Worker C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If all three execute the same query:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;they can select the same events.&lt;/p&gt;

&lt;p&gt;That creates duplicate work immediately.&lt;/p&gt;

&lt;p&gt;PostgreSQL provides a useful mechanism for this problem:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It allows workers to claim different rows without waiting for rows already locked by other workers.&lt;/p&gt;

&lt;p&gt;However, we should not hold a database transaction open while publishing to Kafka.&lt;/p&gt;

&lt;p&gt;Instead, we use a short transaction to claim a batch, commit it, and publish outside the transaction.&lt;/p&gt;

&lt;h3&gt;
  
  
  Claiming a batch
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="n"&gt;candidates&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
    &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt;
        &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;
        &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;next_attempt_at&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;lease_until&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;
            &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="n"&gt;lease_until&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
    &lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="no"&gt;SKIP&lt;/span&gt; &lt;span class="no"&gt;LOCKED&lt;/span&gt;
    &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;events&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt;
    &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;lease_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$3&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="kt"&gt;INTERVAL&lt;/span&gt; &lt;span class="s1"&gt;'1 second'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;attempts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;attempts&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;candidates&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;candidates&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
&lt;span class="k"&gt;RETURNING&lt;/span&gt;
    &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;event_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aggregate_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;event_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Parameters:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;$1&lt;/code&gt;: batch size.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;$2&lt;/code&gt;: unique claim token for this batch.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;$3&lt;/code&gt;: lease duration in seconds.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The &lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt; clause prevents workers from claiming the same locked rows.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;UPDATE&lt;/code&gt; records ownership and establishes a lease.&lt;/p&gt;

&lt;p&gt;Once the transaction commits, the worker can publish the returned events.&lt;/p&gt;

&lt;p&gt;A simplified Go representation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;ClaimedEvent&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="no"&gt;ID&lt;/span&gt;            &lt;span class="kt"&gt;int64&lt;/span&gt;
    &lt;span class="nc"&gt;EventID&lt;/span&gt;       &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UUID&lt;/span&gt;
    &lt;span class="nc"&gt;AggregateType&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;
    &lt;span class="nc"&gt;AggregateID&lt;/span&gt;   &lt;span class="kt"&gt;string&lt;/span&gt;
    &lt;span class="nc"&gt;Type&lt;/span&gt;          &lt;span class="kt"&gt;string&lt;/span&gt;
    &lt;span class="nc"&gt;Payload&lt;/span&gt;       &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="kt"&gt;byte&lt;/span&gt;
    &lt;span class="nc"&gt;Attempts&lt;/span&gt;      &lt;span class="kt"&gt;int&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;Claim&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Token&lt;/span&gt;  &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UUID&lt;/span&gt;
    &lt;span class="nc"&gt;Events&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="nc"&gt;ClaimedEvent&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The repository method should execute the claim query in a short transaction and return the rows after committing.&lt;/p&gt;

&lt;p&gt;The key design rule:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Never keep database row locks open while waiting for a message broker.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A slow broker should not hold database transactions and locks indefinitely.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Publishing Events Outside the Transaction
&lt;/h2&gt;

&lt;p&gt;Define a small publisher interface:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;Publisher&lt;/span&gt; &lt;span class="k"&gt;interface&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Publish&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="nc"&gt;ClaimedEvent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The implementation can use Kafka, RabbitMQ, NATS, or another messaging system.&lt;/p&gt;

&lt;p&gt;The outbox worker should not depend on the broker's concrete client.&lt;/p&gt;

&lt;p&gt;Now let's define the worker:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;Worker&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;repo&lt;/span&gt;      &lt;span class="nc"&gt;Repository&lt;/span&gt;
    &lt;span class="n"&gt;publisher&lt;/span&gt; &lt;span class="nc"&gt;Publisher&lt;/span&gt;

    &lt;span class="n"&gt;batchSize&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;lease&lt;/span&gt;     &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A useful repository interface:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;Repository&lt;/span&gt; &lt;span class="k"&gt;interface&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Claim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;lease&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Claim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nc"&gt;MarkPublished&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UUID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;

    &lt;span class="nc"&gt;ScheduleRetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="kt"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;token&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;UUID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;cause&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The worker processes one claimed batch:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nc"&gt;Worker&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;ProcessBatch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;claim&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Claim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;batchSize&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lease&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"claim outbox events: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;claim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Events&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="n"&gt;publishCtx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;WithTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;publisher&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Publish&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;publishCtx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nf"&gt;retryDelay&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Attempts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="n"&gt;scheduleErr&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ScheduleRetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;claim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;scheduleErr&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                    &lt;span class="s2"&gt;"schedule retry for event %s: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;EventID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;scheduleErr&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="k"&gt;continue&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;repo&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;MarkPublished&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;claim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="s2"&gt;"mark event %s as published: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;EventID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a sequential batch processor. You can add bounded parallelism later, but the ownership and completion rules should remain the same.&lt;/p&gt;

&lt;p&gt;Notice that the worker does not delete an event immediately after publishing it.&lt;/p&gt;

&lt;p&gt;It marks the event as published.&lt;/p&gt;

&lt;p&gt;This makes it possible to inspect publication history, measure delivery delays, and troubleshoot failures.&lt;/p&gt;

&lt;p&gt;For large systems, published rows should eventually be archived or removed according to a retention policy.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Marking Events as Published Safely
&lt;/h2&gt;

&lt;p&gt;The claim token must be checked when updating an event.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt;
    &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;lease_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;last_error&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;
    &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$1&lt;/span&gt;
    &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$2&lt;/span&gt;
    &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why is this important?&lt;/p&gt;

&lt;p&gt;Consider this sequence:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Worker A claims an event.&lt;/li&gt;
&lt;li&gt;Worker A pauses for a long time.&lt;/li&gt;
&lt;li&gt;Its lease expires.&lt;/li&gt;
&lt;li&gt;Worker B reclaims the event.&lt;/li&gt;
&lt;li&gt;Worker A resumes.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Without an ownership check, Worker A could update the event even though Worker B now owns it.&lt;/p&gt;

&lt;p&gt;The claim token prevents that stale update.&lt;/p&gt;

&lt;p&gt;Your repository should also check &lt;code&gt;RowsAffected()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;If no row was updated, the claim is no longer owned by that worker, or the event has already been completed.&lt;/p&gt;

&lt;p&gt;However, there is an important limitation.&lt;/p&gt;

&lt;p&gt;The claim token protects database state. It cannot stop Worker A from publishing to Kafka after its lease expires.&lt;/p&gt;

&lt;p&gt;A lease is not a distributed lock over the external broker.&lt;/p&gt;

&lt;p&gt;That is why consumer-side deduplication remains necessary.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Retry Strategy and Backoff
&lt;/h2&gt;

&lt;p&gt;Message brokers can be temporarily unavailable.&lt;/p&gt;

&lt;p&gt;A worker should not retry a failed event continuously.&lt;/p&gt;

&lt;p&gt;That creates unnecessary load and can make an existing outage worse.&lt;/p&gt;

&lt;p&gt;Use exponential backoff with jitter.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nf"&gt;retryDelay&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Second&lt;/span&gt;
        &lt;span class="nb"&gt;max&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Minute&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;exponent&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;base&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;exponent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;jitter&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;rand&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Int64N&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int64&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;jitter&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This implementation assumes a Go version with &lt;code&gt;math/rand/v2&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The delay grows with each failed attempt but is capped to prevent excessively long retry intervals.&lt;/p&gt;

&lt;p&gt;The retry update can look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt;
    &lt;span class="n"&gt;next_attempt_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="kt"&gt;INTERVAL&lt;/span&gt; &lt;span class="s1"&gt;'1 second'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;lease_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;last_error&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$2&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;
    &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$3&lt;/span&gt;
    &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$4&lt;/span&gt;
    &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The repository should pass a bounded, sanitized error message rather than storing arbitrary error output indefinitely.&lt;/p&gt;

&lt;h3&gt;
  
  
  What about permanently failing events?
&lt;/h3&gt;

&lt;p&gt;Some events will never succeed without intervention.&lt;/p&gt;

&lt;p&gt;Examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Invalid payload.&lt;/li&gt;
&lt;li&gt;Unsupported schema version.&lt;/li&gt;
&lt;li&gt;Message exceeds broker limits.&lt;/li&gt;
&lt;li&gt;A downstream system permanently rejects the event.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A production design needs a poison-event policy.&lt;/p&gt;

&lt;p&gt;Possible approaches:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Maximum attempt count.&lt;/li&gt;
&lt;li&gt;Dead-letter table or queue.&lt;/li&gt;
&lt;li&gt;Manual replay.&lt;/li&gt;
&lt;li&gt;Alerting and operational investigation.&lt;/li&gt;
&lt;li&gt;Event quarantine without losing the original payload.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not silently delete failed events.&lt;/p&gt;

&lt;p&gt;A dead-letter policy should preserve enough information to investigate and replay the operation safely.&lt;/p&gt;

&lt;p&gt;Also remember that a maximum retry count is a business and operational decision. Some important events may require indefinite retry with alerting rather than permanent abandonment.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. The Most Important Failure Window: Publish Succeeds, Database Update Fails
&lt;/h2&gt;

&lt;p&gt;Let's examine the most important scenario.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1. Worker claims event
2. Worker publishes event to Kafka
3. Kafka acknowledges the message
4. Worker crashes
5. published_at is never updated
6. Lease expires
7. Another worker claims and publishes the event again
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The event is delivered twice.&lt;/p&gt;

&lt;p&gt;This is expected behavior.&lt;/p&gt;

&lt;p&gt;The outbox pattern provides durable, at-least-once publication when configured with retries and recovery. It does not provide exactly-once delivery across PostgreSQL and an external broker.&lt;/p&gt;

&lt;p&gt;The consumer must be idempotent.&lt;/p&gt;

&lt;p&gt;This is not an optional optimization. It is part of the design.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Consumer-Side Deduplication
&lt;/h2&gt;

&lt;p&gt;Every event should have a stable identifier.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"event_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"b89b5f43-0d7c-45d7-8a9b-9dbcf9fcb1f8"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"event_type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"order.created"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"aggregate_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"order-123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"payload"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"order_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"order-123"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The consumer stores processed event IDs.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;processed_events&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;consumer_name&lt;/span&gt; &lt;span class="kt"&gt;TEXT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;event_id&lt;/span&gt; &lt;span class="kt"&gt;UUID&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;processed_at&lt;/span&gt; &lt;span class="kt"&gt;TIMESTAMPTZ&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="kc"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;consumer_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The consumer should insert the deduplication record and apply its business change in the same transaction.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nc"&gt;Consumer&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;Handle&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BeginTx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"begin transaction: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Rollback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ExecContext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sb"&gt;`
        INSERT INTO processed_events (
            consumer_name,
            event_id
        )
        VALUES ($1, $2)
        ON CONFLICT DO NOTHING
        `&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"inventory-service"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;ID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"insert processed event: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;affected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;RowsAffected&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"read affected rows: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;affected&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// Already processed.&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Commit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inventory&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ApplyEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"apply inventory event: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Commit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"commit consumer transaction: %w"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The broker message should only be acknowledged after the database transaction commits.&lt;/p&gt;

&lt;p&gt;If the consumer crashes before committing, the transaction rolls back and the message can be delivered again.&lt;/p&gt;

&lt;p&gt;If the transaction commits but the broker acknowledgement is lost, the message may be delivered again. The &lt;code&gt;processed_events&lt;/code&gt; constraint makes the second delivery harmless.&lt;/p&gt;

&lt;p&gt;There is still an important boundary:&lt;/p&gt;

&lt;p&gt;If the consumer's business operation calls an external API, that external side effect needs its own idempotency or outbox strategy.&lt;/p&gt;

&lt;p&gt;Deduplication only protects effects committed inside the same transaction.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Event Ordering Is a Separate Problem
&lt;/h2&gt;

&lt;p&gt;The outbox query uses:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Does this guarantee that every event reaches consumers in business order?&lt;/p&gt;

&lt;p&gt;Not necessarily.&lt;/p&gt;

&lt;p&gt;Multiple workers can claim different events and publish them at different speeds.&lt;/p&gt;

&lt;p&gt;Consider:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Order 123:

1. order.created
2. order.paid
3. order.shipped
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Worker A claims &lt;code&gt;order.created&lt;/code&gt; and becomes slow.&lt;/p&gt;

&lt;p&gt;Worker B claims &lt;code&gt;order.paid&lt;/code&gt; and publishes it first.&lt;/p&gt;

&lt;p&gt;The broker might receive the events in the wrong order.&lt;/p&gt;

&lt;p&gt;If ordering matters, define the ordering boundary explicitly.&lt;/p&gt;

&lt;p&gt;Possible strategies include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Partition events by aggregate ID.&lt;/li&gt;
&lt;li&gt;Use a stable aggregate key in Kafka.&lt;/li&gt;
&lt;li&gt;Maintain an aggregate version or sequence number.&lt;/li&gt;
&lt;li&gt;Ensure only one active publisher processes a given aggregate.&lt;/li&gt;
&lt;li&gt;Use a per-aggregate dispatch mechanism.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, add:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt;
&lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;aggregate_version&lt;/span&gt; &lt;span class="kt"&gt;BIGINT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then enforce uniqueness where appropriate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;idx_outbox_aggregate_version&lt;/span&gt;
&lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;aggregate_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;aggregate_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;aggregate_version&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A stable Kafka partition key helps preserve order within a partition, but it does not fix out-of-order publication before messages reach Kafka.&lt;/p&gt;

&lt;p&gt;If strict per-aggregate ordering is required, the dispatcher must also enforce it.&lt;/p&gt;

&lt;p&gt;This is an important trade-off: stronger ordering constraints can reduce parallelism.&lt;/p&gt;

&lt;h2&gt;
  
  
  12. Running the Worker Continuously
&lt;/h2&gt;

&lt;p&gt;The worker needs a lifecycle.&lt;/p&gt;

&lt;p&gt;A simple polling loop:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nc"&gt;Worker&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;ticker&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NewTicker&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;ticker&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Stop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ProcessBatch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;logger&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="s2"&gt;"outbox batch failed"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;ticker&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;C&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is intentionally simple.&lt;/p&gt;

&lt;p&gt;A production worker should also consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Immediate processing when work is found.&lt;/li&gt;
&lt;li&gt;Adaptive polling when the queue is empty.&lt;/li&gt;
&lt;li&gt;Bounded parallel publishing.&lt;/li&gt;
&lt;li&gt;Graceful shutdown.&lt;/li&gt;
&lt;li&gt;Lease renewal for long-running publication.&lt;/li&gt;
&lt;li&gt;Per-event timeouts.&lt;/li&gt;
&lt;li&gt;Database and broker connection health.&lt;/li&gt;
&lt;li&gt;Backpressure when the broker slows down.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One detail is worth emphasizing:&lt;/p&gt;

&lt;p&gt;The lease duration must be longer than the expected processing time, or the worker must renew its lease while processing.&lt;/p&gt;

&lt;p&gt;Otherwise, another worker may reclaim an event while the original worker is still publishing it.&lt;/p&gt;

&lt;p&gt;Even with lease renewal, consumer deduplication remains the final protection against duplicate delivery.&lt;/p&gt;

&lt;h2&gt;
  
  
  13. Testing the Outbox
&lt;/h2&gt;

&lt;p&gt;A few unit tests are useful, but the most important guarantees involve database transactions and concurrent workers.&lt;/p&gt;

&lt;p&gt;Use PostgreSQL integration tests for the following scenarios.&lt;/p&gt;

&lt;h3&gt;
  
  
  Test 1: Business transaction rollback
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Insert an order.&lt;/li&gt;
&lt;li&gt;Insert an outbox event.&lt;/li&gt;
&lt;li&gt;Force the transaction to fail.&lt;/li&gt;
&lt;li&gt;Verify that neither record exists.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Test 2: Concurrent workers
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Insert multiple pending events.&lt;/li&gt;
&lt;li&gt;Start several workers concurrently.&lt;/li&gt;
&lt;li&gt;Verify that no two active claims own the same row at the same time.&lt;/li&gt;
&lt;li&gt;Verify that all events eventually become eligible for publication.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Test 3: Publish succeeds, mark-published fails
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Make the publisher succeed.&lt;/li&gt;
&lt;li&gt;Simulate a database failure during &lt;code&gt;MarkPublished&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Let the lease expire.&lt;/li&gt;
&lt;li&gt;Verify that the event can be published again.&lt;/li&gt;
&lt;li&gt;Verify that the consumer processes the event only once.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Test 4: Worker crash and lease recovery
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Claim an event.&lt;/li&gt;
&lt;li&gt;Simulate a worker crash.&lt;/li&gt;
&lt;li&gt;Wait until the lease expires.&lt;/li&gt;
&lt;li&gt;Verify that another worker can claim it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Test 5: Stale worker
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Worker A claims an event.&lt;/li&gt;
&lt;li&gt;Its lease expires.&lt;/li&gt;
&lt;li&gt;Worker B reclaims it.&lt;/li&gt;
&lt;li&gt;Worker A attempts to update the row.&lt;/li&gt;
&lt;li&gt;Verify that the old claim token cannot update the event.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Test 6: Poison event
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Configure the publisher to fail permanently.&lt;/li&gt;
&lt;li&gt;Verify retry scheduling.&lt;/li&gt;
&lt;li&gt;Verify that the event reaches the configured dead-letter or alerting policy.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Test 7: Ordering
&lt;/h3&gt;

&lt;p&gt;If your application requires per-aggregate ordering:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Insert multiple events for the same aggregate.&lt;/li&gt;
&lt;li&gt;Process them with multiple workers.&lt;/li&gt;
&lt;li&gt;Verify that consumers observe the expected sequence.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Avoid testing only the happy path. Most outbox bugs appear between two successful operations.&lt;/p&gt;

&lt;h2&gt;
  
  
  14. Observability: Know When Events Are Falling Behind
&lt;/h2&gt;

&lt;p&gt;A transactional outbox can be functioning correctly while downstream systems are still experiencing serious delays.&lt;/p&gt;

&lt;p&gt;You need visibility into the queue.&lt;/p&gt;

&lt;p&gt;Useful metrics include:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;outbox_pending_events
outbox_oldest_pending_age_seconds
outbox_publish_attempts_total
outbox_publish_failures_total
outbox_publish_duration_seconds
outbox_events_published_total
outbox_lease_expirations_total
outbox_dead_letter_events_total
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Some useful operational signals:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;What it might indicate&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Pending event count increasing&lt;/td&gt;
&lt;td&gt;Publisher throughput is below event creation rate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Oldest event age increasing&lt;/td&gt;
&lt;td&gt;Events are not being delivered quickly enough&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retry count increasing&lt;/td&gt;
&lt;td&gt;Broker or network instability&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lease expirations increasing&lt;/td&gt;
&lt;td&gt;Workers are crashing or taking too long&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dead-letter count increasing&lt;/td&gt;
&lt;td&gt;Events require investigation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Avoid using event IDs, aggregate IDs, or customer IDs as metric labels. They create high-cardinality metrics.&lt;/p&gt;

&lt;p&gt;For logs and traces, event IDs and aggregate IDs are useful correlation fields.&lt;/p&gt;

&lt;p&gt;A particularly valuable metric is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NOW() - MIN(created_at)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;for unpublished events.&lt;/p&gt;

&lt;p&gt;The oldest pending event age often tells you more about the health of the pipeline than the total pending count.&lt;/p&gt;

&lt;p&gt;A queue containing 10,000 events might be healthy if they were created seconds ago. A queue containing 20 events might be unhealthy if the oldest has been waiting for an hour.&lt;/p&gt;

&lt;h2&gt;
  
  
  15. Scaling and Operational Trade-offs
&lt;/h2&gt;

&lt;p&gt;The transactional outbox is straightforward to introduce, but the operational model deserves attention.&lt;/p&gt;

&lt;h3&gt;
  
  
  Polling vs. CDC
&lt;/h3&gt;

&lt;p&gt;The implementation in this article uses polling.&lt;/p&gt;

&lt;p&gt;A worker periodically queries PostgreSQL for pending events.&lt;/p&gt;

&lt;p&gt;Advantages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Simple architecture.&lt;/li&gt;
&lt;li&gt;No additional CDC infrastructure.&lt;/li&gt;
&lt;li&gt;Easy to inspect and operate.&lt;/li&gt;
&lt;li&gt;Works with standard PostgreSQL deployments.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Disadvantages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Polling creates database queries even when the queue is empty.&lt;/li&gt;
&lt;li&gt;High event volume can increase database load.&lt;/li&gt;
&lt;li&gt;Batch sizing and polling intervals need tuning.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An alternative is Change Data Capture (CDC).&lt;/p&gt;

&lt;p&gt;Tools such as Debezium can capture database changes from the PostgreSQL write-ahead log and publish them to a broker.&lt;/p&gt;

&lt;p&gt;Advantages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Avoids repeated polling queries.&lt;/li&gt;
&lt;li&gt;Can scale well for high-volume event streams.&lt;/li&gt;
&lt;li&gt;Integrates naturally with Kafka-based architectures.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Disadvantages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Additional infrastructure and operational complexity.&lt;/li&gt;
&lt;li&gt;Schema and connector management.&lt;/li&gt;
&lt;li&gt;More complex replay and recovery behavior.&lt;/li&gt;
&lt;li&gt;Still requires idempotent consumers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The right choice depends on throughput, latency requirements, operational capacity, and existing infrastructure.&lt;/p&gt;

&lt;h3&gt;
  
  
  Retention and cleanup
&lt;/h3&gt;

&lt;p&gt;Published events should not accumulate forever.&lt;/p&gt;

&lt;p&gt;A cleanup job can delete old published events in batches:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="n"&gt;old_events&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
    &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;published_at&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="kt"&gt;INTERVAL&lt;/span&gt; &lt;span class="s1"&gt;'7 days'&lt;/span&gt;
    &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
    &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;DELETE&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;outbox_events&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;old_events&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The retention period should reflect debugging, replay, compliance, and recovery requirements.&lt;/p&gt;

&lt;p&gt;For larger systems, consider partitioning by creation time or moving published events to an archive.&lt;/p&gt;

&lt;p&gt;Never remove unpublished events just because they are old.&lt;/p&gt;

&lt;h3&gt;
  
  
  Backpressure
&lt;/h3&gt;

&lt;p&gt;If event creation is faster than publication, the outbox grows.&lt;/p&gt;

&lt;p&gt;A worker cannot solve an indefinitely increasing backlog by polling more frequently.&lt;/p&gt;

&lt;p&gt;Monitor the publication rate and pending age.&lt;/p&gt;

&lt;p&gt;Consider bounded publishing concurrency, broker capacity, database capacity, and alert thresholds.&lt;/p&gt;

&lt;p&gt;The outbox is durable storage, not infinite capacity.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Production Checklist
&lt;/h2&gt;

&lt;p&gt;Before deploying a transactional outbox, verify the following:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Business changes and outbox inserts share one database transaction.&lt;/li&gt;
&lt;li&gt;[ ] Every event has a stable unique ID.&lt;/li&gt;
&lt;li&gt;[ ] Event payloads have an explicit schema and versioning strategy.&lt;/li&gt;
&lt;li&gt;[ ] Workers claim rows using &lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;[ ] Database transactions are short and never span broker publication.&lt;/li&gt;
&lt;li&gt;[ ] Claims have bounded leases.&lt;/li&gt;
&lt;li&gt;[ ] Stale workers cannot update records owned by newer claims.&lt;/li&gt;
&lt;li&gt;[ ] Retry scheduling uses exponential backoff and jitter.&lt;/li&gt;
&lt;li&gt;[ ] Poison events have an explicit handling policy.&lt;/li&gt;
&lt;li&gt;[ ] Consumers deduplicate events using durable event IDs.&lt;/li&gt;
&lt;li&gt;[ ] Consumer deduplication and business writes share a transaction.&lt;/li&gt;
&lt;li&gt;[ ] Ordering requirements are explicitly defined.&lt;/li&gt;
&lt;li&gt;[ ] Graceful shutdown and lease recovery are tested.&lt;/li&gt;
&lt;li&gt;[ ] Pending event age and retry rates are monitored.&lt;/li&gt;
&lt;li&gt;[ ] Published event retention and cleanup are configured.&lt;/li&gt;
&lt;li&gt;[ ] HTTP operations that create events are idempotent.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Reliable event publishing is not just about calling Kafka after a database insert.&lt;/p&gt;

&lt;p&gt;It is about handling the failure windows between independent systems.&lt;/p&gt;

&lt;p&gt;The transactional outbox pattern gives your application a durable boundary: when a business transaction commits, the event that represents that change is committed with it.&lt;/p&gt;

&lt;p&gt;A separate worker handles publication, retries failures, and recovers abandoned work.&lt;/p&gt;

&lt;p&gt;But the pattern does not guarantee exactly-once delivery.&lt;/p&gt;

&lt;p&gt;A publisher can crash after the broker accepts an event. A lease can expire while a worker is still running. A message can be delivered more than once.&lt;/p&gt;

&lt;p&gt;That's why the complete design needs stable event IDs, consumer-side deduplication, explicit ordering rules, and operational visibility.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Don't try to make two independent systems commit together. Make the intent durable in one transaction, then deliver it reliably.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's the foundation of a dependable event-driven architecture in Go.&lt;/p&gt;




&lt;h3&gt;
  
  
  Further Reading
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://microservices.io/patterns/data/transactional-outbox.html" rel="noopener noreferrer"&gt;Microservices.io — Transactional Outbox&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://debezium.io/documentation/reference/stable/transformations/outbox-event-router.html" rel="noopener noreferrer"&gt;Debezium — Outbox Event Router&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.postgresql.org/docs/current/sql-select.html" rel="noopener noreferrer"&gt;PostgreSQL — SELECT and FOR UPDATE&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.postgresql.org/docs/current/sql-insert.html" rel="noopener noreferrer"&gt;PostgreSQL — INSERT and ON CONFLICT&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pkg.go.dev/database/sql" rel="noopener noreferrer"&gt;Go — database/sql&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://kafka.apache.org/documentation/#semantics" rel="noopener noreferrer"&gt;Apache Kafka — Message Delivery Semantics&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>go</category>
      <category>productivity</category>
      <category>programming</category>
      <category>backend</category>
    </item>
    <item>
      <title>Coordinated Omission in Go: Why Your Load Tests Underestimate Latency</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Sat, 07 Nov 2026 13:00:34 +0000</pubDate>
      <link>https://dev.to/serifcolakel/coordinated-omission-in-go-why-your-load-tests-underestimate-latency-39m8</link>
      <guid>https://dev.to/serifcolakel/coordinated-omission-in-go-why-your-load-tests-underestimate-latency-39m8</guid>
      <description>&lt;p&gt;Your load test reports a p99 latency of 120 ms.&lt;/p&gt;

&lt;p&gt;Everything looks healthy.&lt;/p&gt;

&lt;p&gt;You increase traffic, run the test again, and the results still look reasonable.&lt;/p&gt;

&lt;p&gt;But after deploying the same service to production, users start reporting requests that take several seconds.&lt;/p&gt;

&lt;p&gt;What went wrong?&lt;/p&gt;

&lt;p&gt;One possibility is that your load test never measured the worst part of the system's behavior.&lt;/p&gt;

&lt;p&gt;This problem is known as &lt;strong&gt;coordinated omission&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;It happens when a load generator unintentionally reduces its request rate while the system under test is slowing down. The slowest periods receive fewer measurements precisely because the generator is waiting for previous requests to finish.&lt;/p&gt;

&lt;p&gt;The result is a misleading latency distribution.&lt;/p&gt;

&lt;p&gt;This is particularly important when evaluating backend services where occasional pauses, dependency slowdowns, garbage collection, and traffic bursts can significantly affect tail latency.&lt;/p&gt;

&lt;p&gt;In this article, we'll explore:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What coordinated omission is and why it happens&lt;/li&gt;
&lt;li&gt;Closed-loop versus open-loop load testing&lt;/li&gt;
&lt;li&gt;Why average latency and percentiles can be misleading&lt;/li&gt;
&lt;li&gt;How measurement intervals affect latency distributions&lt;/li&gt;
&lt;li&gt;Building a simple open-loop load generator in Go&lt;/li&gt;
&lt;li&gt;Handling generator saturation and dropped requests&lt;/li&gt;
&lt;li&gt;Using HDR Histogram and latency correction&lt;/li&gt;
&lt;li&gt;Designing more reliable performance tests&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The objective is simple: &lt;strong&gt;measure the latency users would actually experience, not just the latency your load generator happens to observe.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  1. What Is Coordinated Omission?
&lt;/h2&gt;

&lt;p&gt;Imagine a service normally processes requests in 20 milliseconds.&lt;/p&gt;

&lt;p&gt;Your load generator sends one request, waits for the response, and then sends another.&lt;/p&gt;

&lt;p&gt;Everything looks fine.&lt;/p&gt;

&lt;p&gt;Now imagine the service experiences a temporary 2-second pause.&lt;/p&gt;

&lt;p&gt;During those two seconds, the load generator is also waiting for the previous request to finish.&lt;/p&gt;

&lt;p&gt;It doesn't send new requests.&lt;/p&gt;

&lt;p&gt;When the service recovers, the load generator resumes sending requests.&lt;/p&gt;

&lt;p&gt;Your test may record only one extremely slow request, even though a real production system could have received hundreds or thousands of requests during that same period.&lt;/p&gt;

&lt;p&gt;This is coordinated omission.&lt;/p&gt;

&lt;p&gt;The load generator and the system under test are effectively slowing down together.&lt;/p&gt;

&lt;p&gt;Consider this example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Normal operation:

Request 1 -&amp;gt; 20 ms
Request 2 -&amp;gt; 20 ms
Request 3 -&amp;gt; 20 ms
Request 4 -&amp;gt; 20 ms

Service slowdown:

Request 5 -&amp;gt; 2,000 ms

After recovery:

Request 6 -&amp;gt; 20 ms
Request 7 -&amp;gt; 20 ms
Request 8 -&amp;gt; 20 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A closed-loop load generator waits for Request 5 to finish before sending Request 6.&lt;/p&gt;

&lt;p&gt;But in production, incoming traffic doesn't necessarily stop just because one request is taking longer.&lt;/p&gt;

&lt;p&gt;During the 2-second pause, new requests may continue arriving and accumulate.&lt;/p&gt;

&lt;p&gt;The load test therefore misses many of the requests that would have experienced the slowdown.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The problem isn't necessarily that the measured request duration is wrong. The problem is that the test failed to generate and measure requests that should have existed.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Closed-Loop vs. Open-Loop Load Testing
&lt;/h2&gt;

&lt;p&gt;Understanding coordinated omission starts with understanding how load is generated.&lt;/p&gt;

&lt;p&gt;There are two common approaches.&lt;/p&gt;

&lt;h3&gt;
  
  
  Closed-loop load generation
&lt;/h3&gt;

&lt;p&gt;In a closed-loop model, the client waits for a response before sending its next request.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Client
  |
  v
Send Request
  |
  v
Wait for Response
  |
  v
Send Next Request
  |
  v
Wait for Response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;makeRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;duration&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Since&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;recordLatency&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;duration&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;recordError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is simple and useful for certain workloads.&lt;/p&gt;

&lt;p&gt;However, the request rate depends on response time.&lt;/p&gt;

&lt;p&gt;If each request takes 20 ms, one client can generate approximately 50 requests per second.&lt;/p&gt;

&lt;p&gt;If the service slows down to 200 ms, that same client generates only approximately 5 requests per second.&lt;/p&gt;

&lt;p&gt;The offered load decreases as the service becomes slower.&lt;/p&gt;

&lt;p&gt;That's the core problem.&lt;/p&gt;

&lt;p&gt;Closed-loop testing is not inherently incorrect. It models workloads where clients genuinely wait for one operation to finish before starting another.&lt;/p&gt;

&lt;p&gt;But it can be misleading when used to represent independently arriving production traffic.&lt;/p&gt;

&lt;h3&gt;
  
  
  Open-loop load generation
&lt;/h3&gt;

&lt;p&gt;In an open-loop model, requests are scheduled independently of previous responses.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Scheduled arrivals:

0 ms      -&amp;gt; Request 1
10 ms     -&amp;gt; Request 2
20 ms     -&amp;gt; Request 3
30 ms     -&amp;gt; Request 4
40 ms     -&amp;gt; Request 5
50 ms     -&amp;gt; Request 6
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The arrival schedule continues even when earlier requests haven't completed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Load Generator
      |
      +---- Request 1 -----------+
      |                          |
      +---- Request 2 ------+    |
      |                     |    |
      +---- Request 3 --+   |    |
      |                 |   |    |
      v                 v   v    v
         Service Under Load
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This better represents systems where incoming traffic is independent of response completion.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Public HTTP APIs&lt;/li&gt;
&lt;li&gt;Event ingestion services&lt;/li&gt;
&lt;li&gt;Message consumers&lt;/li&gt;
&lt;li&gt;Webhook receivers&lt;/li&gt;
&lt;li&gt;Market data ingestion&lt;/li&gt;
&lt;li&gt;Background processing pipelines&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important difference is that &lt;strong&gt;the arrival rate remains independent of the system's response time&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This makes open-loop testing particularly useful for identifying saturation behavior and tail-latency problems.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. A Simple Example of Measurement Bias
&lt;/h2&gt;

&lt;p&gt;Let's compare two load generators.&lt;/p&gt;

&lt;p&gt;Assume:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Normal response time: 20 ms&lt;/li&gt;
&lt;li&gt;Temporary service pause: 2 seconds&lt;/li&gt;
&lt;li&gt;Target arrival rate: 100 requests/second&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A closed-loop generator with one outstanding request behaves differently from an open-loop generator.&lt;/p&gt;

&lt;h3&gt;
  
  
  Closed-loop behavior
&lt;/h3&gt;

&lt;p&gt;The generator sends a request and waits.&lt;/p&gt;

&lt;p&gt;When the service pauses, the next request cannot be sent until the previous one completes.&lt;/p&gt;

&lt;p&gt;The generator may observe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Request 1: 20 ms
Request 2: 20 ms
Request 3: 20 ms
Request 4: 2,000 ms
Request 5: 20 ms
Request 6: 20 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Only a small number of requests represent the slowdown.&lt;/p&gt;

&lt;h3&gt;
  
  
  Open-loop behavior
&lt;/h3&gt;

&lt;p&gt;At 100 requests per second, new requests are scheduled every 10 milliseconds.&lt;/p&gt;

&lt;p&gt;During a 2-second pause, approximately 200 requests would arrive.&lt;/p&gt;

&lt;p&gt;Those requests may experience substantial waiting time, depending on the system's queueing and processing behavior.&lt;/p&gt;

&lt;p&gt;A simplified view:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Scheduled Request    Completion
--------------------------------
0 ms                 2,000 ms
10 ms                2,010 ms
20 ms                2,020 ms
30 ms                2,030 ms
...
1,990 ms             3,990 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The actual completion pattern depends on service capacity and scheduling, but the key point is that many requests are affected.&lt;/p&gt;

&lt;p&gt;A closed-loop test may report one slow request.&lt;/p&gt;

&lt;p&gt;An open-loop test can reveal a much larger group of requests experiencing elevated latency.&lt;/p&gt;

&lt;p&gt;This is why the two approaches can produce dramatically different p99 measurements.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Why Percentiles Can Be Misleading
&lt;/h2&gt;

&lt;p&gt;Performance reports often focus on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Average latency&lt;/li&gt;
&lt;li&gt;p50&lt;/li&gt;
&lt;li&gt;p95&lt;/li&gt;
&lt;li&gt;p99&lt;/li&gt;
&lt;li&gt;p99.9&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These metrics are useful, but only if the underlying sample represents the workload being evaluated.&lt;/p&gt;

&lt;p&gt;Consider this hypothetical result:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Closed-loop test&lt;/th&gt;
&lt;th&gt;Open-loop test&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Average latency&lt;/td&gt;
&lt;td&gt;25 ms&lt;/td&gt;
&lt;td&gt;350 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p50&lt;/td&gt;
&lt;td&gt;20 ms&lt;/td&gt;
&lt;td&gt;300 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p95&lt;/td&gt;
&lt;td&gt;20 ms&lt;/td&gt;
&lt;td&gt;1,500 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99&lt;/td&gt;
&lt;td&gt;2,000 ms&lt;/td&gt;
&lt;td&gt;1,950 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recorded requests&lt;/td&gt;
&lt;td&gt;1,000&lt;/td&gt;
&lt;td&gt;10,000&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The numbers are illustrative, not a universal outcome.&lt;/p&gt;

&lt;p&gt;Notice something interesting: p99 might look relatively similar, even though the average and p95 are dramatically different.&lt;/p&gt;

&lt;p&gt;Why?&lt;/p&gt;

&lt;p&gt;Because the open-loop test captures more requests affected by the slowdown.&lt;/p&gt;

&lt;p&gt;The important lesson is that a percentile is not meaningful independently of its sampling process.&lt;/p&gt;

&lt;p&gt;A p99 calculated from a biased sample is still a biased p99.&lt;/p&gt;

&lt;h3&gt;
  
  
  The importance of the measurement window
&lt;/h3&gt;

&lt;p&gt;Suppose your service experiences a 2-second pause once every minute.&lt;/p&gt;

&lt;p&gt;If your load generator stops sending requests during that pause, your measurement window may contain very few samples from the affected period.&lt;/p&gt;

&lt;p&gt;Even a large total request count doesn't guarantee representative sampling.&lt;/p&gt;

&lt;p&gt;Always ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How were requests scheduled?&lt;/li&gt;
&lt;li&gt;Did the generator wait for previous responses?&lt;/li&gt;
&lt;li&gt;Were requests skipped when the system slowed down?&lt;/li&gt;
&lt;li&gt;Was latency measured from scheduled arrival or actual transmission?&lt;/li&gt;
&lt;li&gt;Did the load generator itself become saturated?&lt;/li&gt;
&lt;li&gt;Were errors and dropped requests included in the report?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These questions are just as important as the latency numbers themselves.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Scheduled Time vs. Actual Start Time
&lt;/h2&gt;

&lt;p&gt;One of the most important details in an open-loop load test is the starting point of latency measurement.&lt;/p&gt;

&lt;p&gt;Consider a request scheduled for 100 ms.&lt;/p&gt;

&lt;p&gt;But because the load generator is busy, it actually starts sending the request at 180 ms.&lt;/p&gt;

&lt;p&gt;There are now two possible measurements.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Actual request duration:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Response time - Actual start time
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This measures the time spent after the request begins transmission.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Scheduled end-to-end latency:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Response time - Scheduled arrival time
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This includes the delay between the intended arrival and the actual request start.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Scheduled arrival: 100 ms
Actual start:      180 ms
Response:          250 ms

Actual request duration:       70 ms
Scheduled end-to-end latency: 150 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If we record only 70 ms, we hide 80 ms of delay caused by the load generator.&lt;/p&gt;

&lt;p&gt;If the goal is to evaluate the system from the perspective of an independently arriving request, the scheduled arrival time is the more meaningful reference point.&lt;/p&gt;

&lt;p&gt;However, there is an important distinction:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If the generator is intentionally queueing requests before transmission, scheduled-to-response latency captures that queueing delay.&lt;/li&gt;
&lt;li&gt;If the generator cannot schedule or execute requests because it is saturated, the test may no longer represent the intended workload.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You should record generator delay separately instead of hiding it inside one latency number.&lt;/p&gt;

&lt;p&gt;A useful set of measurements includes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Scheduled arrival
Actual request start
Response completion
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;From these timestamps, we can calculate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Generator scheduling delay = Actual start - Scheduled arrival

Request duration = Response completion - Actual start

Scheduled end-to-end latency = Response completion - Scheduled arrival
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This makes it easier to distinguish system performance from load-generator limitations.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Building an Open-Loop Load Generator in Go
&lt;/h2&gt;

&lt;p&gt;Let's implement a small open-loop load generator.&lt;/p&gt;

&lt;p&gt;The objective is to schedule requests at a fixed interval without waiting for previous requests to complete.&lt;/p&gt;

&lt;p&gt;We'll also enforce a maximum number of concurrent requests.&lt;/p&gt;

&lt;p&gt;Most importantly, when the generator cannot launch a scheduled request, we'll count it as dropped instead of silently skipping it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;main&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"context"&lt;/span&gt;
    &lt;span class="s"&gt;"errors"&lt;/span&gt;
    &lt;span class="s"&gt;"sync"&lt;/span&gt;
    &lt;span class="s"&gt;"sync/atomic"&lt;/span&gt;
    &lt;span class="s"&gt;"time"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;ErrConcurrencyLimit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"concurrency limit reached"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;RequestFunc&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Stats&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Scheduled&lt;/span&gt; &lt;span class="n"&gt;atomic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Int64&lt;/span&gt;
    &lt;span class="n"&gt;Started&lt;/span&gt;   &lt;span class="n"&gt;atomic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Int64&lt;/span&gt;
    &lt;span class="n"&gt;Completed&lt;/span&gt; &lt;span class="n"&gt;atomic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Int64&lt;/span&gt;
    &lt;span class="n"&gt;Dropped&lt;/span&gt;   &lt;span class="n"&gt;atomic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Int64&lt;/span&gt;
    &lt;span class="n"&gt;Failed&lt;/span&gt;    &lt;span class="n"&gt;atomic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Int64&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Result&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;ScheduledAt&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Time&lt;/span&gt;
    &lt;span class="n"&gt;StartedAt&lt;/span&gt;   &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Time&lt;/span&gt;
    &lt;span class="n"&gt;FinishedAt&lt;/span&gt;  &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Time&lt;/span&gt;
    &lt;span class="n"&gt;Err&lt;/span&gt;         &lt;span class="kt"&gt;error&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;LoadGenerator&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;
    &lt;span class="n"&gt;maxInFlight&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;

    &lt;span class="n"&gt;request&lt;/span&gt; &lt;span class="n"&gt;RequestFunc&lt;/span&gt;

    &lt;span class="n"&gt;stats&lt;/span&gt; &lt;span class="n"&gt;Stats&lt;/span&gt;
    &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="n"&gt;Result&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;NewLoadGenerator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;maxInFlight&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;request&lt;/span&gt; &lt;span class="n"&gt;RequestFunc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;LoadGenerator&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"interval must be greater than zero"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;maxInFlight&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"maxInFlight must be greater than zero"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"request function cannot be nil"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;LoadGenerator&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;interval&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;    &lt;span class="n"&gt;interval&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;maxInFlight&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;maxInFlight&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;     &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;     &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="n"&gt;Result&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;maxInFlight&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;LoadGenerator&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;duration&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;duration&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"duration must be greater than zero"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;startedAt&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;deadline&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;startedAt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;duration&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;semaphore&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;maxInFlight&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;wg&lt;/span&gt; &lt;span class="n"&gt;sync&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WaitGroup&lt;/span&gt;

    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Wait&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="nb"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}()&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;scheduledAt&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;startedAt&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;scheduledAt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Before&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;deadline&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="n"&gt;scheduledAt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;scheduledAt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interval&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;wait&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Until&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scheduledAt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;wait&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;timer&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewTimer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;C&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

            &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;C&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Scheduled&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;semaphore&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{}{}&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="k"&gt;go&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scheduled&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Time&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
                &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;semaphore&lt;/span&gt; &lt;span class="p"&gt;}()&lt;/span&gt;

                &lt;span class="n"&gt;actualStart&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
                &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Started&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

                &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

                &lt;span class="n"&gt;finished&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Failed&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;

                &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Completed&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

                &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt; &lt;span class="n"&gt;Result&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="n"&gt;ScheduledAt&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;scheduled&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;StartedAt&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;   &lt;span class="n"&gt;actualStart&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;FinishedAt&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;  &lt;span class="n"&gt;finished&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;         &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
            &lt;span class="p"&gt;}(&lt;/span&gt;&lt;span class="n"&gt;scheduledAt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stats&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Dropped&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This example demonstrates several important ideas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Requests are scheduled independently of previous responses.&lt;/li&gt;
&lt;li&gt;The generator uses a fixed arrival interval.&lt;/li&gt;
&lt;li&gt;Maximum in-flight concurrency is bounded.&lt;/li&gt;
&lt;li&gt;Requests that cannot be started are counted as dropped.&lt;/li&gt;
&lt;li&gt;Scheduled, actual start, and completion timestamps are preserved.&lt;/li&gt;
&lt;li&gt;Request execution is measured separately from scheduling delay.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The implementation is intentionally a small demonstration rather than a complete benchmarking framework.&lt;/p&gt;

&lt;h3&gt;
  
  
  Understanding the request interval
&lt;/h3&gt;

&lt;p&gt;Suppose we want to generate 100 requests per second.&lt;/p&gt;

&lt;p&gt;The interval is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;10&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 second / 100 requests = 10 milliseconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We can initialize the generator:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;generator&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;NewLoadGenerator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="m"&gt;10&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="m"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;makeRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nb"&gt;panic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WithTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Background&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="m"&gt;30&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;generator&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;30&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c"&gt;// Handle cancellation or test failure.&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With this configuration:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Target arrival rate: 100 requests/second&lt;/li&gt;
&lt;li&gt;Maximum in-flight requests: 200&lt;/li&gt;
&lt;li&gt;Test duration: 30 seconds&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The maximum concurrency is a safety boundary for the generator.&lt;/p&gt;

&lt;p&gt;It is not a guarantee that the target arrival rate will be achieved.&lt;/p&gt;

&lt;p&gt;If requests take too long, the concurrency limit may be reached and scheduled requests will be dropped.&lt;/p&gt;

&lt;p&gt;That's an important result, not something to hide.&lt;/p&gt;

&lt;h3&gt;
  
  
  A limitation of this example
&lt;/h3&gt;

&lt;p&gt;The example uses one scheduling goroutine and launches a goroutine for each accepted request.&lt;/p&gt;

&lt;p&gt;It is suitable for demonstrating open-loop scheduling, but a production-grade load generator needs additional protections:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A dedicated result collector&lt;/li&gt;
&lt;li&gt;A non-blocking or carefully sized result pipeline&lt;/li&gt;
&lt;li&gt;Explicit generator CPU and memory monitoring&lt;/li&gt;
&lt;li&gt;Better cancellation and shutdown behavior&lt;/li&gt;
&lt;li&gt;Histograms for latency measurements&lt;/li&gt;
&lt;li&gt;Accurate reporting of missed schedules&lt;/li&gt;
&lt;li&gt;Protection against excessive goroutine creation&lt;/li&gt;
&lt;li&gt;Separate reporting of generator delay and service latency&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In particular, if the result channel fills and no consumer is reading from it, request goroutines can block while reporting results.&lt;/p&gt;

&lt;p&gt;A real implementation should run a dedicated collector concurrently with request generation.&lt;/p&gt;

&lt;p&gt;The test harness itself must never become the hidden bottleneck.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Coordinated Omission Correction
&lt;/h2&gt;

&lt;p&gt;Open-loop load generation is one way to reduce coordinated omission.&lt;/p&gt;

&lt;p&gt;Another approach is to correct latency measurements for the expected arrival interval.&lt;/p&gt;

&lt;p&gt;This technique is commonly associated with HDR Histogram.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Observed request latency: 1,000 ms
Expected arrival interval: 10 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the generator was waiting for a previous response, many expected requests may not have been recorded.&lt;/p&gt;

&lt;p&gt;A coordinated-omission-aware histogram can account for those missing observations by recording additional latency values at intervals related to the expected request cadence.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Observed latency: 1,000 ms
Expected interval: 10 ms

Additional observations represent:
990 ms
980 ms
970 ms
...
10 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact recording behavior depends on the histogram API and correction method.&lt;/p&gt;

&lt;p&gt;The important idea is that a long pause should not be represented by only one sample if the test model implies that many requests would have arrived during that period.&lt;/p&gt;

&lt;h3&gt;
  
  
  HDR Histogram in Go
&lt;/h3&gt;

&lt;p&gt;HDR Histogram is useful for recording latency distributions with a wide dynamic range.&lt;/p&gt;

&lt;p&gt;A Go implementation is available through:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/HdrHistogram/hdrhistogram-go" rel="noopener noreferrer"&gt;HDR Histogram Go&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A basic example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;main&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"fmt"&lt;/span&gt;
    &lt;span class="s"&gt;"time"&lt;/span&gt;

    &lt;span class="n"&gt;hdrhistogram&lt;/span&gt; &lt;span class="s"&gt;"github.com/HdrHistogram/hdrhistogram-go"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;histogram&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;hdrhistogram&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="m"&gt;60_000_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;latencies&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="m"&gt;10&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="m"&gt;20&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="m"&gt;30&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="m"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;latency&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;range&lt;/span&gt; &lt;span class="n"&gt;latencies&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;histogram&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RecordValue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;latency&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Microseconds&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="s"&gt;"p50: %d µs&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;histogram&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ValueAtQuantile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;50&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="s"&gt;"p95: %d µs&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;histogram&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ValueAtQuantile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;95&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="s"&gt;"p99: %d µs&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;histogram&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ValueAtQuantile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;99&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This example records observed values.&lt;/p&gt;

&lt;p&gt;It does not automatically correct coordinated omission.&lt;/p&gt;

&lt;p&gt;Correction requires the expected arrival interval and an appropriate recording strategy.&lt;/p&gt;

&lt;p&gt;For example, when using a histogram API that supports it, a corrected recording operation can account for the expected interval:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;histogram&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RecordCorrectedValue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;latency&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Microseconds&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="n"&gt;expectedInterval&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Microseconds&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Always check the library version and API you're using before integrating this into a test harness.&lt;/p&gt;

&lt;h3&gt;
  
  
  Correction is not a replacement for a good load model
&lt;/h3&gt;

&lt;p&gt;There is an important limitation.&lt;/p&gt;

&lt;p&gt;Coordinated omission correction estimates observations that were not recorded based on an assumed arrival interval.&lt;/p&gt;

&lt;p&gt;It cannot reconstruct every detail of a real production traffic pattern.&lt;/p&gt;

&lt;p&gt;If production traffic is bursty, user-driven, or has complex dependencies, a constant-rate correction may not accurately represent it.&lt;/p&gt;

&lt;p&gt;Use correction as a measurement technique, not as a substitute for realistic workload generation.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. The Load Generator Can Become the Bottleneck
&lt;/h2&gt;

&lt;p&gt;There's another problem worth discussing.&lt;/p&gt;

&lt;p&gt;An open-loop generator is only useful if it can actually generate the intended workload.&lt;/p&gt;

&lt;p&gt;Imagine the target is 50,000 requests per second.&lt;/p&gt;

&lt;p&gt;But the load generator runs out of CPU, network bandwidth, file descriptors, or available connections.&lt;/p&gt;

&lt;p&gt;Your test may report low throughput and high latency.&lt;/p&gt;

&lt;p&gt;Is the service saturated?&lt;/p&gt;

&lt;p&gt;Or is the generator saturated?&lt;/p&gt;

&lt;p&gt;Without separate measurements, it's difficult to know.&lt;/p&gt;

&lt;p&gt;A load generator should monitor:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Why it matters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Scheduled requests&lt;/td&gt;
&lt;td&gt;Intended workload&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Started requests&lt;/td&gt;
&lt;td&gt;Requests actually transmitted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Completed requests&lt;/td&gt;
&lt;td&gt;Finished requests&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dropped requests&lt;/td&gt;
&lt;td&gt;Requests not started&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;In-flight requests&lt;/td&gt;
&lt;td&gt;Current generator concurrency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Generator scheduling delay&lt;/td&gt;
&lt;td&gt;Delay before request execution&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Generator CPU usage&lt;/td&gt;
&lt;td&gt;Local resource pressure&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Generator memory usage&lt;/td&gt;
&lt;td&gt;Potential resource exhaustion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network utilization&lt;/td&gt;
&lt;td&gt;Possible bandwidth bottleneck&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For distributed load testing, multiple generator instances may be required.&lt;/p&gt;

&lt;p&gt;But adding more generators introduces another consideration: their clocks and scheduling behavior should be sufficiently consistent for the measurements you're trying to make.&lt;/p&gt;

&lt;p&gt;Don't assume that a test generating 50,000 requests per second is actually doing so just because that was the configured target.&lt;/p&gt;

&lt;p&gt;Measure the achieved arrival rate.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Designing a Reliable Load Test
&lt;/h2&gt;

&lt;p&gt;A good load test should answer a specific question.&lt;/p&gt;

&lt;p&gt;Instead of simply asking:&lt;/p&gt;

&lt;p&gt;"How many requests per second can this service handle?"&lt;/p&gt;

&lt;p&gt;Ask:&lt;/p&gt;

&lt;p&gt;"What happens to latency and error rates when the service receives 2,000 independently arriving requests per second?"&lt;/p&gt;

&lt;p&gt;A practical testing strategy can include several phases.&lt;/p&gt;

&lt;h3&gt;
  
  
  Phase 1: Baseline
&lt;/h3&gt;

&lt;p&gt;Run the service under low traffic.&lt;/p&gt;

&lt;p&gt;Measure:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Average latency&lt;/li&gt;
&lt;li&gt;p50, p95, p99&lt;/li&gt;
&lt;li&gt;Throughput&lt;/li&gt;
&lt;li&gt;Error rate&lt;/li&gt;
&lt;li&gt;Resource utilization&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This establishes a baseline.&lt;/p&gt;

&lt;h3&gt;
  
  
  Phase 2: Gradual ramp-up
&lt;/h3&gt;

&lt;p&gt;Increase the arrival rate gradually.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;500 req/s
1,000 req/s
1,500 req/s
2,000 req/s
2,500 req/s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Observe where latency begins increasing disproportionately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Phase 3: Sustained load
&lt;/h3&gt;

&lt;p&gt;Hold the target arrival rate for a meaningful period.&lt;/p&gt;

&lt;p&gt;This helps identify:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Queue accumulation&lt;/li&gt;
&lt;li&gt;Memory growth&lt;/li&gt;
&lt;li&gt;Connection pool saturation&lt;/li&gt;
&lt;li&gt;Garbage collection effects&lt;/li&gt;
&lt;li&gt;Resource exhaustion&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Phase 4: Burst traffic
&lt;/h3&gt;

&lt;p&gt;Send short bursts above the normal arrival rate.&lt;/p&gt;

&lt;p&gt;Observe how quickly the system recovers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Phase 5: Dependency slowdown
&lt;/h3&gt;

&lt;p&gt;Introduce latency into a downstream dependency.&lt;/p&gt;

&lt;p&gt;Measure whether the application maintains predictable behavior.&lt;/p&gt;

&lt;h3&gt;
  
  
  Phase 6: Recovery
&lt;/h3&gt;

&lt;p&gt;Return the system to normal traffic.&lt;/p&gt;

&lt;p&gt;Check whether queue depth, latency, and resource usage return to baseline.&lt;/p&gt;

&lt;p&gt;The recovery phase is important.&lt;/p&gt;

&lt;p&gt;A service that survives a traffic spike but takes several minutes to recover may still have a serious resilience problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. A Practical Checklist
&lt;/h2&gt;

&lt;p&gt;Before trusting a performance test, ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Does the load model represent the real production traffic pattern?&lt;/li&gt;
&lt;li&gt;Is the arrival rate independent of response completion where appropriate?&lt;/li&gt;
&lt;li&gt;Are requests scheduled at known intervals?&lt;/li&gt;
&lt;li&gt;Is latency measured from scheduled arrival time when required?&lt;/li&gt;
&lt;li&gt;Are generator scheduling delays recorded separately?&lt;/li&gt;
&lt;li&gt;Are dropped or missed requests visible?&lt;/li&gt;
&lt;li&gt;Is the achieved arrival rate close to the configured target?&lt;/li&gt;
&lt;li&gt;Is the load generator itself monitored?&lt;/li&gt;
&lt;li&gt;Are p95, p99, and p99.9 calculated from representative samples?&lt;/li&gt;
&lt;li&gt;Is coordinated omission correction appropriate for the workload?&lt;/li&gt;
&lt;li&gt;Have we tested both steady-state and burst traffic?&lt;/li&gt;
&lt;li&gt;Do we understand how the system behaves during recovery?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you cannot answer these questions, your latency report may be less reliable than it looks.&lt;/p&gt;




&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Performance testing isn't just about sending requests and measuring how long they take.&lt;/p&gt;

&lt;p&gt;It's about generating the right workload and ensuring that the measurement process doesn't hide the behavior we're trying to understand.&lt;/p&gt;

&lt;p&gt;Coordinated omission is a good example of how a seemingly reasonable test can produce misleading results.&lt;/p&gt;

&lt;p&gt;A closed-loop load generator may reduce its request rate when the service slows down, making the system appear healthier than it would under independently arriving production traffic.&lt;/p&gt;

&lt;p&gt;Open-loop scheduling, accurate timestamps, generator monitoring, and appropriate histogram techniques help address this problem.&lt;/p&gt;

&lt;p&gt;For Go engineers, the implementation details matter:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Schedule arrivals independently of previous responses.&lt;/li&gt;
&lt;li&gt;Measure scheduled time and actual start time separately.&lt;/li&gt;
&lt;li&gt;Bound concurrency without silently hiding dropped work.&lt;/li&gt;
&lt;li&gt;Monitor the load generator as carefully as the service.&lt;/li&gt;
&lt;li&gt;Use latency histograms that reflect the workload being tested.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;A performance test should reveal the system's limitations, not adapt to them.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If your load generator slows down whenever your service slows down, you may be measuring the generator's patience rather than your system's performance.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.youtube.com/watch?v=lJ8ydIuPFeU" rel="noopener noreferrer"&gt;Gil Tene — How NOT to Measure Latency&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/HdrHistogram/HdrHistogram" rel="noopener noreferrer"&gt;HDR Histogram — Coordinated Omission&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/HdrHistogram/hdrhistogram-go" rel="noopener noreferrer"&gt;HDR Histogram Go&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/giltene/wrk2" rel="noopener noreferrer"&gt;wrk2 — A Constant Throughput, Correct Latency Recording Fork of wrk&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pkg.go.dev/time" rel="noopener noreferrer"&gt;Go Documentation — time Package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pkg.go.dev/context" rel="noopener noreferrer"&gt;Go Documentation — Context Package&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>go</category>
      <category>productivity</category>
      <category>backend</category>
      <category>programming</category>
    </item>
    <item>
      <title>Durable Execution for AI Agents: State, Checkpointing, Retries and Human-in-the-Loop</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Tue, 03 Nov 2026 19:08:28 +0000</pubDate>
      <link>https://dev.to/serifcolakel/durable-execution-for-ai-agents-state-checkpointing-retries-and-human-in-the-loop-99f</link>
      <guid>https://dev.to/serifcolakel/durable-execution-for-ai-agents-state-checkpointing-retries-and-human-in-the-loop-99f</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwhd3tzlb57cfx8cnm38w.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwhd3tzlb57cfx8cnm38w.png" alt=" " width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;How to build AI agents that survive failures, resume interrupted tasks, handle approvals, and execute reliably in production.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Why Do AI Agents Need Durable Execution?
&lt;/h2&gt;

&lt;p&gt;Most AI applications start with a relatively straightforward execution model.&lt;/p&gt;

&lt;p&gt;A user sends a message, the application invokes an LLM, the model may call a tool, and the application returns a response.&lt;/p&gt;

&lt;p&gt;For simple interactions, this is usually sufficient.&lt;/p&gt;

&lt;p&gt;However, agentic applications introduce a different level of complexity.&lt;/p&gt;

&lt;p&gt;An agent may execute several model calls, invoke multiple tools, delegate tasks to other agents, wait for external systems, and require human approval before continuing.&lt;/p&gt;

&lt;p&gt;Consider an agent responsible for reviewing and processing a business request:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The user submits a request.&lt;/li&gt;
&lt;li&gt;The agent analyzes the request.&lt;/li&gt;
&lt;li&gt;It retrieves information from internal systems.&lt;/li&gt;
&lt;li&gt;It delegates an analysis task to a specialist agent.&lt;/li&gt;
&lt;li&gt;It generates a proposed action.&lt;/li&gt;
&lt;li&gt;It requests approval from an authorized user.&lt;/li&gt;
&lt;li&gt;After approval, it executes the operation.&lt;/li&gt;
&lt;li&gt;It records the result and completes the task.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This workflow may take seconds, minutes, or even several hours.&lt;/p&gt;

&lt;p&gt;Now imagine the application crashes after step five.&lt;/p&gt;

&lt;p&gt;What happens to the work already completed?&lt;/p&gt;

&lt;p&gt;Does the agent start over? Are previous model calls executed again? Will the user need to submit the request again? Could an external operation be executed twice?&lt;/p&gt;

&lt;p&gt;These are not unusual edge cases. They are fundamental engineering problems in long-running AI systems.&lt;/p&gt;

&lt;p&gt;A conventional in-memory agent generally keeps its execution state inside the current process. When that process terminates unexpectedly, the state may be lost.&lt;/p&gt;

&lt;p&gt;Saving conversation messages to a database does not solve this problem by itself.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Durable Execution is the ability to preserve workflow progress and resume execution from a known point after failures, interruptions, or external events.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It allows us to build agents that are not dependent on one uninterrupted application process.&lt;/p&gt;

&lt;p&gt;The objective is not merely to make an agent remember what happened. It is to make the entire execution reliable.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. State Persistence vs. Durable Execution
&lt;/h2&gt;

&lt;p&gt;These concepts are related, but they are not interchangeable.&lt;/p&gt;

&lt;h3&gt;
  
  
  State Persistence
&lt;/h3&gt;

&lt;p&gt;State persistence means storing information so that it can be retrieved later.&lt;/p&gt;

&lt;p&gt;For an AI application, this might include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Conversation history&lt;/li&gt;
&lt;li&gt;User preferences&lt;/li&gt;
&lt;li&gt;Current task status&lt;/li&gt;
&lt;li&gt;Agent configuration&lt;/li&gt;
&lt;li&gt;Intermediate results&lt;/li&gt;
&lt;li&gt;Retrieved documents&lt;/li&gt;
&lt;li&gt;Tool execution results&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, we can store conversation messages in PostgreSQL and load them when a user returns to the application.&lt;/p&gt;

&lt;p&gt;This provides conversation continuity.&lt;/p&gt;

&lt;p&gt;However, storing messages does not automatically tell us where an interrupted workflow should resume.&lt;/p&gt;

&lt;h3&gt;
  
  
  Durable Execution
&lt;/h3&gt;

&lt;p&gt;Durable execution goes further.&lt;/p&gt;

&lt;p&gt;It records the execution progress and the results of completed steps so that the workflow can continue without repeating all previous work.&lt;/p&gt;

&lt;p&gt;Consider this example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Agent Workflow
      |
      v
Retrieve Context
      |
      v
LLM Analysis
      |
      v
Execute Tool
      |
      v
Generate Report
      |
      X Application Crashes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without durable execution, the application may need to restart the entire workflow.&lt;/p&gt;

&lt;p&gt;With checkpointing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Agent Workflow
      |
      v
Retrieve Context       ✓ Checkpoint
      |
      v
LLM Analysis           ✓ Checkpoint
      |
      v
Execute Tool           ✓ Checkpoint
      |
      v
Generate Report        X Failed
      |
      v
Application Restarts
      |
      v
Resume from last
completed checkpoint
      |
      v
Generate Report
      |
      v
Complete
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The completed results can be reused while the failed operation is retried.&lt;/p&gt;

&lt;p&gt;This distinction becomes particularly important when model calls are expensive or when tools perform operations against external systems.&lt;/p&gt;

&lt;h3&gt;
  
  
  What about Agent Memory?
&lt;/h3&gt;

&lt;p&gt;Memory and durable execution are also different concepts.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concept&lt;/th&gt;
&lt;th&gt;Responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Conversation history&lt;/td&gt;
&lt;td&gt;What the user and agent discussed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Short-term memory&lt;/td&gt;
&lt;td&gt;Information needed during the current interaction&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Long-term memory&lt;/td&gt;
&lt;td&gt;Information retained across interactions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Checkpointing&lt;/td&gt;
&lt;td&gt;Where execution stopped and what has completed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Durable execution&lt;/td&gt;
&lt;td&gt;How the workflow resumes and continues reliably&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For example, an agent may remember a user's preferences through long-term memory while a durable workflow maintains the execution state of an ongoing approval process.&lt;/p&gt;

&lt;p&gt;One does not replace the other.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://docs.langchain.com/oss/javascript/langgraph/persistence" rel="noopener noreferrer"&gt;LangGraph — Persistence&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Core Concepts Behind Durable Execution
&lt;/h2&gt;

&lt;p&gt;Before choosing a framework, it helps to understand the underlying engineering primitives.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.1. Checkpointing
&lt;/h3&gt;

&lt;p&gt;A checkpoint is a persisted representation of workflow state at a specific point in execution.&lt;/p&gt;

&lt;p&gt;Depending on the runtime, it may include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Completed step identifiers&lt;/li&gt;
&lt;li&gt;Step outputs&lt;/li&gt;
&lt;li&gt;Workflow state&lt;/li&gt;
&lt;li&gt;Execution metadata&lt;/li&gt;
&lt;li&gt;Pending operations&lt;/li&gt;
&lt;li&gt;Current task status&lt;/li&gt;
&lt;li&gt;References to external payloads&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When a workflow resumes, the runtime uses these checkpoints to determine which operations have already completed.&lt;/p&gt;

&lt;p&gt;Checkpointing is particularly useful when a workflow contains expensive or long-running operations.&lt;/p&gt;

&lt;p&gt;However, we should avoid storing unnecessarily large objects in every checkpoint.&lt;/p&gt;

&lt;p&gt;For example, instead of persisting a 50 MB document inside workflow state, we can store it in object storage and keep only its identifier and metadata in the checkpoint.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.2. Replay
&lt;/h3&gt;

&lt;p&gt;Replay is the process of reconstructing workflow execution using its previously recorded history.&lt;/p&gt;

&lt;p&gt;Some durable execution engines replay workflow code to rebuild the current execution state.&lt;/p&gt;

&lt;p&gt;This introduces an important engineering requirement: deterministic execution.&lt;/p&gt;

&lt;p&gt;If the same workflow is replayed with the same history, it should make the same workflow decisions.&lt;/p&gt;

&lt;p&gt;Consider:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;workflow&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;timestamp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

  &lt;span class="c1"&gt;// Continue execution...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the workflow is replayed later, &lt;code&gt;Date.now()&lt;/code&gt; returns a different value.&lt;/p&gt;

&lt;p&gt;This could change its execution path.&lt;/p&gt;

&lt;p&gt;Likewise, direct network calls, random values, and database queries should not be performed inside deterministic workflow definitions when the runtime expects replay-safe code.&lt;/p&gt;

&lt;p&gt;These operations should be isolated into activities or durable steps.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;workflow&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;timestamp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;getCurrentTimeStep&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

  &lt;span class="c1"&gt;// Continue using the recorded result...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The durable runtime records the step result and reuses it during replay.&lt;/p&gt;

&lt;p&gt;This is a foundational concept in systems such as Temporal.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.3. Durable Steps
&lt;/h3&gt;

&lt;p&gt;A durable workflow is usually divided into independent execution units.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Workflow
   |
   +-- Step 1: Retrieve documents
   |
   +-- Step 2: Generate analysis
   |
   +-- Step 3: Validate output
   |
   +-- Step 4: Request approval
   |
   +-- Step 5: Execute operation
   |
   +-- Step 6: Save final result
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each completed step produces a checkpoint.&lt;/p&gt;

&lt;p&gt;If step five fails, the workflow should not need to repeat document retrieval, model analysis, and validation.&lt;/p&gt;

&lt;p&gt;This also allows us to apply different retry policies to different operations.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.4. Durable Waits
&lt;/h3&gt;

&lt;p&gt;A workflow may need to wait for something that is outside the application's control.&lt;/p&gt;

&lt;p&gt;Examples include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Human approval&lt;/li&gt;
&lt;li&gt;An external webhook&lt;/li&gt;
&lt;li&gt;A scheduled execution time&lt;/li&gt;
&lt;li&gt;A payment confirmation&lt;/li&gt;
&lt;li&gt;A response from another agent&lt;/li&gt;
&lt;li&gt;Completion of a long-running background operation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A regular asynchronous function can wait for an event, but keeping the process alive indefinitely is not a scalable approach.&lt;/p&gt;

&lt;p&gt;Durable execution systems can suspend the workflow, persist its state, and resume it when the expected event arrives.&lt;/p&gt;

&lt;p&gt;The application does not need to keep a worker occupied during the waiting period.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.5. Retries and Recovery
&lt;/h3&gt;

&lt;p&gt;Failures should be handled according to their type.&lt;/p&gt;

&lt;p&gt;A network timeout, provider rate limit, validation error, and authorization failure should not necessarily follow the same retry policy.&lt;/p&gt;

&lt;p&gt;A retry policy should define:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Maximum attempts&lt;/li&gt;
&lt;li&gt;Initial delay&lt;/li&gt;
&lt;li&gt;Backoff strategy&lt;/li&gt;
&lt;li&gt;Maximum delay&lt;/li&gt;
&lt;li&gt;Retryable error categories&lt;/li&gt;
&lt;li&gt;Overall execution deadline&lt;/li&gt;
&lt;li&gt;Cancellation behavior&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A durable execution engine provides the infrastructure for retries, but the application still needs to define what should be retried and why.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. The Most Important Concept: Idempotency
&lt;/h2&gt;

&lt;p&gt;This is one of the most important aspects of building reliable AI agents.&lt;/p&gt;

&lt;p&gt;Durable execution can resume workflows and retry failed steps. But retries can introduce duplicate side effects.&lt;/p&gt;

&lt;p&gt;Consider an agent that generates a report and publishes it to an external system.&lt;/p&gt;

&lt;p&gt;The operation succeeds, but the application crashes before the workflow records the result.&lt;/p&gt;

&lt;p&gt;When execution resumes, the workflow may attempt to publish the report again.&lt;/p&gt;

&lt;p&gt;Without additional protection, we could create two reports.&lt;/p&gt;

&lt;p&gt;This is why idempotency matters.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;An operation is idempotent when executing it multiple times with the same logical input produces the same intended outcome as executing it once.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For example, consider a report publishing API:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;publishReport&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;reportId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nl"&gt;operationId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://api.example.com/reports&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;POST&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Content-Type&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Idempotency-Key&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;operationId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
        &lt;span class="na"&gt;reportId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;reportId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;content&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}),&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="s2"&gt;`Report publishing failed: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important detail is &lt;code&gt;operationId&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It must be generated and persisted by a trusted part of the application before the external operation is attempted. It should not be an arbitrary identifier generated by the LLM.&lt;/p&gt;

&lt;p&gt;The receiving service must also implement idempotency-key handling. Merely sending the header does not guarantee duplicate prevention.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why isn't exactly-once execution automatic?
&lt;/h3&gt;

&lt;p&gt;This is a common misunderstanding.&lt;/p&gt;

&lt;p&gt;A workflow engine can guarantee that a successfully checkpointed step is not unnecessarily executed again during replay.&lt;/p&gt;

&lt;p&gt;However, an external side effect and the workflow's checkpoint cannot always participate in the same atomic transaction.&lt;/p&gt;

&lt;p&gt;There is a failure window:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;An external API accepts the request.&lt;/li&gt;
&lt;li&gt;The external operation succeeds.&lt;/li&gt;
&lt;li&gt;The application crashes.&lt;/li&gt;
&lt;li&gt;The workflow has not recorded the successful result.&lt;/li&gt;
&lt;li&gt;The workflow retries.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This means we should not assume that every external operation executes exactly once.&lt;/p&gt;

&lt;p&gt;A more accurate approach is to design for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;At-least-once execution attempts.&lt;/li&gt;
&lt;li&gt;Idempotent external operations.&lt;/li&gt;
&lt;li&gt;Durable operation identifiers.&lt;/li&gt;
&lt;li&gt;Deduplication at the receiving service.&lt;/li&gt;
&lt;li&gt;Transactional outbox patterns where appropriate.&lt;/li&gt;
&lt;li&gt;Reconciliation for operations whose result is uncertain.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For critical business operations, this distinction is non-negotiable.&lt;/p&gt;

&lt;p&gt;A durable workflow is not a substitute for transactional business design.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Human-in-the-Loop: When an Agent Needs Approval
&lt;/h2&gt;

&lt;p&gt;Not every operation should be executed automatically.&lt;/p&gt;

&lt;p&gt;In enterprise applications, an agent may need authorization before performing an operation that has financial, operational, or security implications.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Publishing a report&lt;/li&gt;
&lt;li&gt;Updating customer information&lt;/li&gt;
&lt;li&gt;Executing a database modification&lt;/li&gt;
&lt;li&gt;Approving a business transaction&lt;/li&gt;
&lt;li&gt;Changing production configuration&lt;/li&gt;
&lt;li&gt;Triggering an external workflow&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A traditional implementation may look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Agent
  |
  v
Generate Proposed Action
  |
  v
Save Pending Approval
  |
  v
Notify User
  |
  v
Wait for Response
  |
  v
User Approves
  |
  v
Continue Agent
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The problem is that the application needs to preserve enough state to resume the original execution.&lt;/p&gt;

&lt;p&gt;A durable workflow makes this much easier.&lt;/p&gt;

&lt;p&gt;It can:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Execute the agent until an approval boundary.&lt;/li&gt;
&lt;li&gt;Persist the current state.&lt;/li&gt;
&lt;li&gt;Suspend execution.&lt;/li&gt;
&lt;li&gt;Notify the appropriate user.&lt;/li&gt;
&lt;li&gt;Wait for an approval event.&lt;/li&gt;
&lt;li&gt;Resume from the saved state.&lt;/li&gt;
&lt;li&gt;Continue or abort based on the decision.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The approval process may take seconds, hours, or days.&lt;/p&gt;

&lt;p&gt;Importantly, the application should verify the identity and authorization of the person responding. The decision should be bound to the original operation, its parameters, and its expiration policy.&lt;/p&gt;

&lt;p&gt;An approval should not be a generic boolean that can be reused for another action.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Implementing Durable Agents with Vercel AI SDK
&lt;/h2&gt;

&lt;p&gt;For TypeScript-based applications, the Vercel AI SDK ecosystem provides an interesting integration through &lt;code&gt;WorkflowAgent&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The standard &lt;code&gt;ToolLoopAgent&lt;/code&gt; runs its execution loop in memory.&lt;/p&gt;

&lt;p&gt;This is suitable for short interactions, but its progress can be lost when the underlying process terminates.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;WorkflowAgent&lt;/code&gt;, provided through &lt;code&gt;@ai-sdk/workflow&lt;/code&gt;, runs the agent inside the Vercel Workflow runtime.&lt;/p&gt;

&lt;p&gt;It provides:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Persistent execution progress&lt;/li&gt;
&lt;li&gt;Independent tool retries&lt;/li&gt;
&lt;li&gt;Recovery after process interruptions&lt;/li&gt;
&lt;li&gt;Human approval that can survive long waits&lt;/li&gt;
&lt;li&gt;Workflow-level execution visibility&lt;/li&gt;
&lt;li&gt;Resumable chat streaming through &lt;code&gt;WorkflowChatTransport&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An important distinction: durability comes from the Workflow runtime, not simply from changing the agent class.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Install the packages
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm add ai @ai-sdk/workflow workflow zod
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 2: Define durable tool operations
&lt;/h3&gt;

&lt;p&gt;A tool operation can be marked as a durable step using the &lt;code&gt;'use step'&lt;/code&gt; directive.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;searchDocumentsStep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;use step&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;`https://api.example.com/search?q=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nf"&gt;encodeURIComponent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="s2"&gt;`Document search failed: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The runtime can persist the result of this step and retry it independently when a retryable failure occurs.&lt;/p&gt;

&lt;p&gt;This is particularly useful when the operation interacts with an unreliable external service.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Create a WorkflowAgent
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;WorkflowAgent&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;ModelCallStreamPart&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@ai-sdk/workflow&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;convertToModelMessages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;UIMessage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ai&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;getWritable&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;workflow&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;zod&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;searchDocumentsStep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;use step&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;`https://api.example.com/search?q=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nf"&gt;encodeURIComponent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Document search failed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;publishReportStep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;reportId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;use step&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://api.example.com/reports/publish&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;POST&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Content-Type&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="na"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Report publishing failed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;runAssistant&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nx"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;UIMessage&lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;use workflow&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;agent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;WorkflowAgent&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;anthropic/claude-sonnet-4-6&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="na"&gt;instructions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`
      You are an enterprise research assistant.

      Search internal documentation when needed.
      Prepare reports based on verified information.
      Never publish a report without approval.
    `&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="na"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;searchDocuments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
        &lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Search internal documentation.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;inputSchema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;object&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
          &lt;span class="na"&gt;query&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;string&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
        &lt;span class="p"&gt;}),&lt;/span&gt;
        &lt;span class="na"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;searchDocumentsStep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}),&lt;/span&gt;

      &lt;span class="na"&gt;publishReport&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
        &lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Publish a prepared report.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;inputSchema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;object&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
          &lt;span class="na"&gt;reportId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;string&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
        &lt;span class="p"&gt;}),&lt;/span&gt;
        &lt;span class="na"&gt;needsApproval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;publishReportStep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}),&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stream&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;convertToModelMessages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="na"&gt;writable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;getWritable&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;ModelCallStreamPart&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This example demonstrates the core pattern:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The workflow function is marked with &lt;code&gt;'use workflow'&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Tool implementations that need durable retries are marked with &lt;code&gt;'use step'&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The report publishing tool requires human approval.&lt;/li&gt;
&lt;li&gt;The workflow can suspend and resume while waiting for the approval decision.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a real deployment, the publishing operation should also implement server-controlled idempotency and authorization. The simple example focuses on the workflow integration rather than the complete business security layer.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Understand the difference between WorkflowAgent and ToolLoopAgent
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Feature&lt;/th&gt;
&lt;th&gt;ToolLoopAgent&lt;/th&gt;
&lt;th&gt;WorkflowAgent&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Execution&lt;/td&gt;
&lt;td&gt;In-memory&lt;/td&gt;
&lt;td&gt;Durable workflow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Process restart recovery&lt;/td&gt;
&lt;td&gt;No automatic recovery&lt;/td&gt;
&lt;td&gt;Supported by runtime&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool retries&lt;/td&gt;
&lt;td&gt;Application-managed&lt;/td&gt;
&lt;td&gt;Per-step retry support&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Long approval waits&lt;/td&gt;
&lt;td&gt;Requires custom handling&lt;/td&gt;
&lt;td&gt;Durable suspension&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool execution visibility&lt;/td&gt;
&lt;td&gt;AI SDK telemetry&lt;/td&gt;
&lt;td&gt;Workflow step visibility&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Streaming&lt;/td&gt;
&lt;td&gt;Direct stream consumption&lt;/td&gt;
&lt;td&gt;Workflow writable&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The recommended approach is not to replace every &lt;code&gt;ToolLoopAgent&lt;/code&gt; with &lt;code&gt;WorkflowAgent&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;For short, synchronous agent interactions, an in-memory agent is often simpler.&lt;/p&gt;

&lt;p&gt;Use durable execution when the workflow needs to outlive its original request, survive process interruptions, wait for external events, or retry individual operations independently.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Make Chat Streaming Resumable
&lt;/h3&gt;

&lt;p&gt;One challenge in long-running AI applications is that the backend workflow and the frontend connection have different lifecycles.&lt;/p&gt;

&lt;p&gt;A user might:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Refresh the browser.&lt;/li&gt;
&lt;li&gt;Lose network connectivity.&lt;/li&gt;
&lt;li&gt;Navigate away from a page.&lt;/li&gt;
&lt;li&gt;Reconnect after a function timeout.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A durable workflow may still be running even though the original HTTP connection has disappeared.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;WorkflowChatTransport&lt;/code&gt; provides a mechanism for resuming the chat stream using the workflow run ID and previously received stream chunks.&lt;/p&gt;

&lt;p&gt;A simplified client configuration:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;use client&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;useChat&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@ai-sdk/react&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;WorkflowChatTransport&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@ai-sdk/workflow&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;useMemo&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;react&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;Chat&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;transport&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;useMemo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
      &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;WorkflowChatTransport&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
        &lt;span class="na"&gt;api&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/api/chat&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;maxConsecutiveErrors&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;initialStartIndex&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}),&lt;/span&gt;
    &lt;span class="p"&gt;[],&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;sendMessage&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;useChat&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="nx"&gt;transport&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="c1"&gt;// Render conversation and handle user input.&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The server also needs the corresponding workflow stream endpoints.&lt;/p&gt;

&lt;p&gt;This is an important architectural detail: a durable backend does not automatically make the browser connection durable. The frontend needs a way to reconnect and retrieve the appropriate part of the workflow stream.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://vercel.com/kb/guide/what-is-workflowagent" rel="noopener noreferrer"&gt;Vercel — WorkflowAgent&lt;/a&gt; · &lt;a href="https://vercel.com/kb/guide/durableagent-to-workflowagent" rel="noopener noreferrer"&gt;AI SDK — WorkflowAgent Migration&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  7. LangGraph: Checkpointing and Interrupts
&lt;/h2&gt;

&lt;p&gt;LangGraph approaches durable agent execution through graph state and persistence.&lt;/p&gt;

&lt;p&gt;Its persistence architecture provides two complementary mechanisms:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Checkpointers&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Persist graph state for a specific thread.&lt;/p&gt;

&lt;p&gt;They support:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Conversation continuity&lt;/li&gt;
&lt;li&gt;Recovery from interruptions&lt;/li&gt;
&lt;li&gt;Human-in-the-loop workflows&lt;/li&gt;
&lt;li&gt;Time travel&lt;/li&gt;
&lt;li&gt;Fault tolerance&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Stores&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Persist application-defined information across different threads.&lt;/p&gt;

&lt;p&gt;They are useful for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;User preferences&lt;/li&gt;
&lt;li&gt;Long-term memory&lt;/li&gt;
&lt;li&gt;Shared knowledge&lt;/li&gt;
&lt;li&gt;Cross-conversation information&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A simplified configuration:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;MemorySaver&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;StateGraph&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;Annotation&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@langchain/langgraph&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;State&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;Annotation&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Root&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Annotation&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;reducer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;current&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;update&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;current&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;update&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="na"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt;
  &lt;span class="p"&gt;}),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;builder&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;StateGraph&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;State&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// Add nodes and edges...&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;checkpointer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;MemorySaver&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;graph&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;builder&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="nx"&gt;checkpointer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;graph&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;invoke&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Analyze the requested document.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;configurable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;thread_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;analysis-123&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This example uses an in-memory checkpointer for illustration.&lt;/p&gt;

&lt;p&gt;It is not suitable for surviving process restarts. For production, use a persistent implementation such as PostgreSQL or another supported durable checkpointer.&lt;/p&gt;

&lt;h3&gt;
  
  
  Human-in-the-loop with interrupts
&lt;/h3&gt;

&lt;p&gt;LangGraph provides the &lt;code&gt;interrupt()&lt;/code&gt; function to pause a graph and wait for external input.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;interrupt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@langchain/langgraph&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;approvalNode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;proposedAction&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;decision&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;interrupt&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;proposedAction&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Approval is required.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;decision&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;approved&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;rejected&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;approved&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The graph persists its state through its checkpointer.&lt;/p&gt;

&lt;p&gt;Later, the application can resume the same thread using &lt;code&gt;Command&lt;/code&gt; with the user's response.&lt;/p&gt;

&lt;p&gt;This is useful for workflows where execution must pause for approval or additional input.&lt;/p&gt;

&lt;p&gt;One subtle but important implementation detail: when resuming an interrupted graph, the interrupted node may execute again from its beginning. Operations performed before &lt;code&gt;interrupt()&lt;/code&gt; therefore need to be designed carefully to avoid duplicate side effects.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://docs.langchain.com/oss/javascript/langgraph/persistence" rel="noopener noreferrer"&gt;LangGraph Persistence&lt;/a&gt; · &lt;a href="https://docs.langchain.com/oss/javascript/langgraph/interrupts" rel="noopener noreferrer"&gt;LangGraph Interrupts&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Temporal: Durable Execution as an Infrastructure Primitive
&lt;/h2&gt;

&lt;p&gt;Temporal takes a different approach.&lt;/p&gt;

&lt;p&gt;Rather than focusing specifically on agent state or conversation management, Temporal provides a general-purpose durable execution platform.&lt;/p&gt;

&lt;p&gt;It is designed for long-running, stateful workflows that need to survive failures.&lt;/p&gt;

&lt;p&gt;Two concepts are particularly important:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Workflow&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Defines the orchestration logic and execution history.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Activity&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Performs external or non-deterministic operations such as API requests, database access, LLM calls, and file operations.&lt;/p&gt;

&lt;p&gt;A simplified AI workflow could look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Temporal Workflow
       |
       v
Retrieve Context Activity
       |
       v
LLM Analysis Activity
       |
       v
Validation Activity
       |
       v
Wait for Approval Signal
       |
       v
Execute Business Operation
       |
       v
Complete
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Temporal records workflow history and uses deterministic replay to reconstruct execution state.&lt;/p&gt;

&lt;p&gt;External interactions belong in Activities so their results can be recorded and reused during replay.&lt;/p&gt;

&lt;p&gt;This separation is fundamental.&lt;/p&gt;

&lt;p&gt;An LLM invocation should not be performed directly in a deterministic workflow definition. It should be executed as an Activity.&lt;/p&gt;

&lt;p&gt;The same applies to database calls and external HTTP requests.&lt;/p&gt;

&lt;h3&gt;
  
  
  When does Temporal make sense?
&lt;/h3&gt;

&lt;p&gt;Temporal is particularly relevant when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Workflows span multiple services.&lt;/li&gt;
&lt;li&gt;Tasks may run for hours or days.&lt;/li&gt;
&lt;li&gt;Strong recovery requirements exist.&lt;/li&gt;
&lt;li&gt;Human approval is part of the process.&lt;/li&gt;
&lt;li&gt;Many independent workers execute tasks.&lt;/li&gt;
&lt;li&gt;Workflow execution is critical to business operations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It also introduces additional infrastructure and operational responsibilities.&lt;/p&gt;

&lt;p&gt;Teams need to understand workflow history, Activity retries, Worker deployment, workflow versioning, and task queue management.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://docs.temporal.io/ai" rel="noopener noreferrer"&gt;Temporal — Durable AI&lt;/a&gt; · &lt;a href="https://docs.temporal.io/develop/typescript" rel="noopener noreferrer"&gt;Temporal TypeScript SDK&lt;/a&gt; · &lt;a href="https://docs.temporal.io/workflow-definition" rel="noopener noreferrer"&gt;Temporal Workflow Determinism&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Inngest and DBOS: Other Approaches Worth Evaluating
&lt;/h2&gt;

&lt;p&gt;Temporal and LangGraph are not the only options.&lt;/p&gt;

&lt;p&gt;For TypeScript-based applications, two other projects are worth considering.&lt;/p&gt;

&lt;h3&gt;
  
  
  Inngest
&lt;/h3&gt;

&lt;p&gt;Inngest provides durable functions and agent workflows.&lt;/p&gt;

&lt;p&gt;Its execution model is built around steps.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;inngest&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;./client&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;researchWorkflow&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;inngest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createFunction&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;research-workflow&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;step&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;context&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;step&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;retrieve-context&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;retrieveContext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;query&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;

    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;analysis&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;step&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;generate-analysis&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;generateAnalysis&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;context&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;

    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;report&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;step&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;validate-report&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;validateReport&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;analysis&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;report&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The runtime checkpoints completed steps and resumes from the failed step when execution recovers.&lt;/p&gt;

&lt;p&gt;Inngest also provides &lt;code&gt;step.waitForEvent()&lt;/code&gt; for human approvals and other external events.&lt;/p&gt;

&lt;p&gt;Its durable agent approach goes further by treating each model call and tool call as a separately checkpointed step.&lt;/p&gt;

&lt;p&gt;This is useful when we want to write ordinary TypeScript functions instead of defining an explicit graph.&lt;/p&gt;

&lt;h3&gt;
  
  
  DBOS
&lt;/h3&gt;

&lt;p&gt;DBOS provides durable execution for TypeScript and other languages, using a database-backed workflow model.&lt;/p&gt;

&lt;p&gt;Its TypeScript SDK offers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Workflow registration&lt;/li&gt;
&lt;li&gt;Durable steps&lt;/li&gt;
&lt;li&gt;Configurable retries&lt;/li&gt;
&lt;li&gt;Workflow IDs&lt;/li&gt;
&lt;li&gt;Idempotency and deduplication mechanisms&lt;/li&gt;
&lt;li&gt;Background execution&lt;/li&gt;
&lt;li&gt;Recovery after process restarts&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A simplified example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;DBOS&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@dbos-inc/dbos-sdk&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;retrieveDocuments&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// External API or database operation&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;document-1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;document-2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;analyzeDocuments&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// LLM interaction&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Analysis completed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;researchWorkflow&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;documents&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;DBOS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;runStep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;retrieveDocuments&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;retrieve-documents&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;retriesAllowed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;maxAttempts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;analysis&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;DBOS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;runStep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;analyzeDocuments&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;analyze-documents&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;retriesAllowed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;maxAttempts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;documents&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;analysis&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;workflow&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;DBOS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;registerWorkflow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nx"&gt;researchWorkflow&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This example shows the basic step model. A production application must initialize DBOS, register the workflow, and configure its database and worker lifecycle.&lt;/p&gt;

&lt;p&gt;DBOS also requires deterministic workflow code and appropriate serialization of workflow inputs and outputs.&lt;/p&gt;

&lt;h3&gt;
  
  
  How do these approaches compare?
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Technology&lt;/th&gt;
&lt;th&gt;Main strength&lt;/th&gt;
&lt;th&gt;Best suited for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;LangGraph&lt;/td&gt;
&lt;td&gt;Graph state and checkpointing&lt;/td&gt;
&lt;td&gt;Stateful agent workflows&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Temporal&lt;/td&gt;
&lt;td&gt;General-purpose durable execution&lt;/td&gt;
&lt;td&gt;Complex distributed business processes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inngest&lt;/td&gt;
&lt;td&gt;Durable TypeScript functions and agents&lt;/td&gt;
&lt;td&gt;Event-driven agent workflows&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DBOS&lt;/td&gt;
&lt;td&gt;Database-backed durable execution&lt;/td&gt;
&lt;td&gt;TypeScript applications needing durable steps&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Vercel Workflow + WorkflowAgent&lt;/td&gt;
&lt;td&gt;Native AI SDK integration&lt;/td&gt;
&lt;td&gt;Durable agents in Vercel-based applications&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There is no universally correct choice.&lt;/p&gt;

&lt;p&gt;The right decision depends on the execution model, deployment environment, workflow duration, infrastructure constraints, and the level of operational control required.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://www.inngest.com/docs/durable-execution/durable-agents" rel="noopener noreferrer"&gt;Inngest Durable Agents&lt;/a&gt; · &lt;a href="https://docs.dbos.dev/typescript/tutorials/workflow-tutorial" rel="noopener noreferrer"&gt;DBOS TypeScript Workflows&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Designing a Production-Ready Agent Workflow
&lt;/h2&gt;

&lt;p&gt;Choosing a durable execution framework is only one part of the solution.&lt;/p&gt;

&lt;p&gt;We also need to design the workflow itself.&lt;/p&gt;

&lt;p&gt;Let's consider a document analysis and approval workflow.&lt;/p&gt;

&lt;p&gt;The system should:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Receive a document.&lt;/li&gt;
&lt;li&gt;Extract its content.&lt;/li&gt;
&lt;li&gt;Analyze the information.&lt;/li&gt;
&lt;li&gt;Generate a proposed report.&lt;/li&gt;
&lt;li&gt;Wait for human approval.&lt;/li&gt;
&lt;li&gt;Publish the approved report.&lt;/li&gt;
&lt;li&gt;Store the final result.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A possible architecture:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;             User
              |
              v
        AI Application
              |
              v
       Workflow Runtime
              |
      +-------+--------+
      |                |
      v                v
  Extract Step    Analysis Step
      |                |
      v                v
   Storage          LLM / Tools
      |                |
      +--------+-------+
               |
               v
        Validation Step
               |
               v
        Approval Required
               |
        +------+------+
        |             |
      Reject        Approve
        |             |
        v             v
      Stop      Publish Step
                      |
                      v
                 Save Result
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Several design decisions matter here.&lt;/p&gt;

&lt;h3&gt;
  
  
  Persist the right information
&lt;/h3&gt;

&lt;p&gt;Do not put every document, model response, or binary object into the workflow's event history.&lt;/p&gt;

&lt;p&gt;Use references to external storage for large payloads.&lt;/p&gt;

&lt;p&gt;Persist only the data required to recover the next step.&lt;/p&gt;

&lt;h3&gt;
  
  
  Separate planning from execution
&lt;/h3&gt;

&lt;p&gt;The LLM may decide which operation should happen next, but the runtime should control which operations are actually executed.&lt;/p&gt;

&lt;p&gt;The agent should not own unrestricted workflow persistence or authorization decisions.&lt;/p&gt;

&lt;h3&gt;
  
  
  Define explicit failure boundaries
&lt;/h3&gt;

&lt;p&gt;A document extraction failure should not necessarily require the analysis to restart.&lt;/p&gt;

&lt;p&gt;An approval timeout should not cause the document to be processed again.&lt;/p&gt;

&lt;p&gt;A publishing failure should not require repeating the original LLM request.&lt;/p&gt;

&lt;p&gt;Each operation should have a clear recovery strategy.&lt;/p&gt;

&lt;h3&gt;
  
  
  Keep business state authoritative
&lt;/h3&gt;

&lt;p&gt;A workflow checkpoint records execution progress.&lt;/p&gt;

&lt;p&gt;It should not replace the authoritative transaction state of the business system.&lt;/p&gt;

&lt;p&gt;For example, an approval workflow may persist its state in the orchestration engine, but the actual transaction should be committed and recorded in the relevant business system.&lt;/p&gt;

&lt;h3&gt;
  
  
  Make workflows version-aware
&lt;/h3&gt;

&lt;p&gt;Long-running workflows can remain active across application deployments.&lt;/p&gt;

&lt;p&gt;A workflow that started using one version of the code may resume after a newer version has been deployed.&lt;/p&gt;

&lt;p&gt;Changing step ordering, workflow logic, or serialized state formats can break recovery if compatibility is not considered.&lt;/p&gt;

&lt;p&gt;Workflow versioning, migration strategies, and backward-compatible payloads should be part of the deployment design.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Common Mistakes in Durable Agent Implementations
&lt;/h2&gt;

&lt;p&gt;There are a few mistakes I would specifically avoid.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Treating conversation history as workflow state&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Saving messages does not mean the agent can resume from the exact point where it stopped.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Retrying everything&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A failed tool should not force all previous model calls and tool executions to run again.&lt;/p&gt;

&lt;p&gt;Retry boundaries should be explicit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Assuming exactly-once side effects&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A durable workflow cannot automatically guarantee exactly-once behavior across arbitrary external services.&lt;/p&gt;

&lt;p&gt;Use idempotency keys, deduplication, and reconciliation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Storing large payloads inside checkpoints&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Store references to documents and large artifacts instead of repeatedly serializing them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Ignoring workflow versioning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A long-running workflow may outlive multiple application deployments.&lt;/p&gt;

&lt;p&gt;Plan how in-flight executions will behave when code changes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Keeping workers alive while waiting for approval&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A durable wait should suspend execution and release the worker rather than holding a process open.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Using durable execution for every interaction&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A simple question-answering endpoint does not necessarily need a workflow engine.&lt;/p&gt;

&lt;p&gt;The added infrastructure should solve an actual reliability problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Confusing agent reasoning with business authorization&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;An agent deciding to perform an action does not mean that action is authorized.&lt;/p&gt;

&lt;p&gt;Sensitive operations must be validated and approved by trusted application logic.&lt;/p&gt;

&lt;h2&gt;
  
  
  12. What Can an AI Chapter Lead Contribute?
&lt;/h2&gt;

&lt;p&gt;I believe durable execution is particularly important when multiple teams are building AI-powered features within the same organization.&lt;/p&gt;

&lt;p&gt;Without shared patterns, each team may implement its own persistence, retry logic, background execution, and approval mechanisms.&lt;/p&gt;

&lt;p&gt;This creates duplicated engineering effort and inconsistent reliability.&lt;/p&gt;

&lt;p&gt;There are several practical areas where an AI Chapter Lead can contribute.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Establish a Common Durable Execution Pattern
&lt;/h3&gt;

&lt;p&gt;Define when an agent should run in memory and when it should run inside a durable workflow.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Use case&lt;/th&gt;
&lt;th&gt;Recommended approach&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Simple chat response&lt;/td&gt;
&lt;td&gt;In-memory agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Short tool interaction&lt;/td&gt;
&lt;td&gt;Standard agent with controlled retries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Multi-step background task&lt;/td&gt;
&lt;td&gt;Durable workflow&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Human approval&lt;/td&gt;
&lt;td&gt;Durable wait and resume&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Long-running research&lt;/td&gt;
&lt;td&gt;Durable agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cross-service business process&lt;/td&gt;
&lt;td&gt;Workflow engine&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Independent subagent tasks&lt;/td&gt;
&lt;td&gt;Durable child workflows or delegated executions&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This prevents unnecessary infrastructure while ensuring that critical workflows have appropriate recovery mechanisms.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Build a Shared Workflow Abstraction
&lt;/h3&gt;

&lt;p&gt;We can create a reusable internal package that provides:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Workflow lifecycle management&lt;/li&gt;
&lt;li&gt;Checkpoint persistence&lt;/li&gt;
&lt;li&gt;Standard retry policies&lt;/li&gt;
&lt;li&gt;Idempotency helpers&lt;/li&gt;
&lt;li&gt;Cancellation support&lt;/li&gt;
&lt;li&gt;Human approval mechanisms&lt;/li&gt;
&lt;li&gt;Execution state management&lt;/li&gt;
&lt;li&gt;Observability integration&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Teams should not need to implement these capabilities independently.&lt;/p&gt;

&lt;p&gt;However, shared abstractions should not hide framework-specific behavior that affects reliability.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Define Standard Retry and Recovery Policies
&lt;/h3&gt;

&lt;p&gt;Not every failure should trigger another attempt.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;Strategy&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Temporary network failure&lt;/td&gt;
&lt;td&gt;Retry with backoff&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Provider rate limit&lt;/td&gt;
&lt;td&gt;Respect retry guidance and budget&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Invalid model output&lt;/td&gt;
&lt;td&gt;Validate and optionally regenerate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Authorization failure&lt;/td&gt;
&lt;td&gt;Stop and escalate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Invalid tool arguments&lt;/td&gt;
&lt;td&gt;Correct or request clarification&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Business rule violation&lt;/td&gt;
&lt;td&gt;Stop or request human review&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;External operation timeout&lt;/td&gt;
&lt;td&gt;Reconcile before retrying&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;We can define common retry budgets, deadlines, maximum attempts, and error classification rules.&lt;/p&gt;

&lt;p&gt;This reduces inconsistent failure handling across applications.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Establish Idempotency Standards
&lt;/h3&gt;

&lt;p&gt;Every operation that creates an external side effect should have an explicit idempotency strategy.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Database writes&lt;/li&gt;
&lt;li&gt;Report publishing&lt;/li&gt;
&lt;li&gt;Email sending&lt;/li&gt;
&lt;li&gt;External API requests&lt;/li&gt;
&lt;li&gt;Business transaction execution&lt;/li&gt;
&lt;li&gt;Agent-to-agent task submission&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is especially important when workflows involve retries or uncertain execution outcomes.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Standardize Human Approval
&lt;/h3&gt;

&lt;p&gt;We can establish a shared approval model that defines:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Who can approve an action.&lt;/li&gt;
&lt;li&gt;Which operations require approval.&lt;/li&gt;
&lt;li&gt;How approval requests are stored.&lt;/li&gt;
&lt;li&gt;How decisions are associated with workflow executions.&lt;/li&gt;
&lt;li&gt;What happens when approval expires.&lt;/li&gt;
&lt;li&gt;How rejected actions are handled.&lt;/li&gt;
&lt;li&gt;How approval decisions are audited.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This makes human-in-the-loop workflows consistent across teams.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Build a Reference Implementation
&lt;/h3&gt;

&lt;p&gt;A practical internal example should demonstrate:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Durable agent execution.&lt;/li&gt;
&lt;li&gt;Step-level checkpointing.&lt;/li&gt;
&lt;li&gt;Independent retries.&lt;/li&gt;
&lt;li&gt;Human approval.&lt;/li&gt;
&lt;li&gt;Resume after application restart.&lt;/li&gt;
&lt;li&gt;Idempotent external operations.&lt;/li&gt;
&lt;li&gt;Distributed tracing.&lt;/li&gt;
&lt;li&gt;Workflow versioning.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is more useful than a standalone demo that only shows a successful agent response.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Measure Reliability
&lt;/h3&gt;

&lt;p&gt;We should define measurable outcomes.&lt;/p&gt;

&lt;p&gt;Useful metrics include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Workflow completion rate&lt;/li&gt;
&lt;li&gt;Recovery success rate&lt;/li&gt;
&lt;li&gt;Number of repeated model calls&lt;/li&gt;
&lt;li&gt;Duplicate side-effect attempts&lt;/li&gt;
&lt;li&gt;Approval wait duration&lt;/li&gt;
&lt;li&gt;Retry count per workflow&lt;/li&gt;
&lt;li&gt;Average workflow duration&lt;/li&gt;
&lt;li&gt;Cost per successful workflow&lt;/li&gt;
&lt;li&gt;Human escalation rate&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The objective should be to understand whether durable execution actually improves reliability without creating unnecessary complexity.&lt;/p&gt;

&lt;h2&gt;
  
  
  13. When Should We Adopt Durable Execution?
&lt;/h2&gt;

&lt;p&gt;A durable execution framework is particularly valuable when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A task involves multiple dependent operations.&lt;/li&gt;
&lt;li&gt;Execution can exceed HTTP request timeouts.&lt;/li&gt;
&lt;li&gt;A workflow must survive process restarts.&lt;/li&gt;
&lt;li&gt;Expensive model calls should not be repeated unnecessarily.&lt;/li&gt;
&lt;li&gt;A human may need to approve an operation later.&lt;/li&gt;
&lt;li&gt;The agent interacts with external systems that may fail independently.&lt;/li&gt;
&lt;li&gt;Multiple workers or services participate in one business process.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a short chat response, a regular AI SDK agent may be sufficient.&lt;/p&gt;

&lt;p&gt;For a multi-hour research workflow or a business operation that cannot be safely restarted from the beginning, durable execution becomes much more important.&lt;/p&gt;

&lt;p&gt;The correct architecture is the simplest one that provides the required failure guarantees.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;AI agents are becoming more capable, but greater capability introduces additional operational complexity.&lt;/p&gt;

&lt;p&gt;An agent that can call tools, delegate tasks, and interact with external systems must also be able to recover from failures, preserve execution progress, and handle interruptions safely.&lt;/p&gt;

&lt;p&gt;Durable execution provides the foundation for this.&lt;/p&gt;

&lt;p&gt;Checkpointing preserves progress. Retries recover from transient failures. Idempotency prevents duplicate side effects. Durable waits enable human approval without keeping workers occupied. Workflow versioning allows long-running executions to survive application changes.&lt;/p&gt;

&lt;p&gt;For AI engineering teams, the goal should not be to make every agent use a workflow engine.&lt;/p&gt;

&lt;p&gt;It should be to identify which operations require durability and apply the right execution model to them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;My recommendation:&lt;/strong&gt; Start with one multi-step workflow that includes an external API call and a human approval step. Implement it with a durable execution framework, simulate process failures, and verify that completed work is not repeated and external side effects are not duplicated.&lt;/p&gt;

&lt;p&gt;A production-ready agent is not one that only works when everything goes right.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It is one that knows how to recover when things go wrong.&lt;/strong&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  References and Further Reading
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Vercel AI SDK and Workflow
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/kb/guide/what-is-workflowagent" rel="noopener noreferrer"&gt;Vercel — What is WorkflowAgent?&lt;/a&gt; — Durable AI agents, retries, approvals, and resumable execution.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/kb/guide/ai-gateway-and-ai-sdk" rel="noopener noreferrer"&gt;Vercel — Build AI Agents with AI SDK and AI Gateway&lt;/a&gt; — AI SDK and Workflow integration.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/kb/workflow-sdk" rel="noopener noreferrer"&gt;Vercel — Workflow SDK&lt;/a&gt; — Durable functions, steps, and execution model.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/kb/guide/durableagent-to-workflowagent" rel="noopener noreferrer"&gt;Vercel — Migrate from DurableAgent to WorkflowAgent&lt;/a&gt; — AI SDK v7 integration changes.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  LangGraph
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://docs.langchain.com/oss/javascript/langgraph/persistence" rel="noopener noreferrer"&gt;LangGraph — Persistence&lt;/a&gt; — Checkpointers, stores, state, and memory.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.langchain.com/oss/javascript/langgraph/interrupts" rel="noopener noreferrer"&gt;LangGraph — Interrupts&lt;/a&gt; — Human-in-the-loop and resumable execution.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.langchain.com/oss/javascript/langgraph/fault-tolerance" rel="noopener noreferrer"&gt;LangGraph — Fault Tolerance&lt;/a&gt; — Recovery and execution reliability.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Temporal
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://docs.temporal.io/ai" rel="noopener noreferrer"&gt;Temporal — Durable AI&lt;/a&gt; — Durable execution for AI applications.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.temporal.io/develop/typescript" rel="noopener noreferrer"&gt;Temporal — TypeScript SDK&lt;/a&gt; — Workflows, Activities, Workers, and message passing.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.temporal.io/workflow-definition" rel="noopener noreferrer"&gt;Temporal — Workflow Determinism&lt;/a&gt; — Replay-safe workflow implementation.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.temporal.io/ai/cookbook" rel="noopener noreferrer"&gt;Temporal — AI Cookbook&lt;/a&gt; — AI integrations, agent loops, tool calling, and human approval.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Inngest
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://www.inngest.com/docs/durable-execution" rel="noopener noreferrer"&gt;Inngest — Durable Execution&lt;/a&gt; — Steps, retries, checkpoints, and event handling.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.inngest.com/docs/durable-execution/durable-agents" rel="noopener noreferrer"&gt;Inngest — Durable Agents&lt;/a&gt; — Agent loops, independent retries, and subagent delegation.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.inngest.com/docs/durable-execution/durable-agents/human-in-the-loop" rel="noopener noreferrer"&gt;Inngest — Human-in-the-Loop&lt;/a&gt; — Durable approval and resume patterns.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  DBOS
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://docs.dbos.dev/typescript/tutorials/workflow-tutorial" rel="noopener noreferrer"&gt;DBOS — TypeScript Workflows&lt;/a&gt; — Durable workflows and automatic recovery.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.dbos.dev/typescript/reference/workflows-steps" rel="noopener noreferrer"&gt;DBOS — Workflows and Steps&lt;/a&gt; — Retry configuration and step execution.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.dbos.dev/architecture" rel="noopener noreferrer"&gt;DBOS — Architecture&lt;/a&gt; — Checkpointing, recovery, and deterministic execution.&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>javascript</category>
      <category>programming</category>
      <category>react</category>
    </item>
    <item>
      <title>Queueing Theory in Go: Understanding Latency, Throughput, and Saturation</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Tue, 03 Nov 2026 18:22:18 +0000</pubDate>
      <link>https://dev.to/serifcolakel/queueing-theory-in-go-understanding-latency-throughput-and-saturation-49mg</link>
      <guid>https://dev.to/serifcolakel/queueing-theory-in-go-understanding-latency-throughput-and-saturation-49mg</guid>
      <description>&lt;p&gt;Your service handles 500 requests per second.&lt;/p&gt;

&lt;p&gt;The CPU is not fully utilized. The database is responding normally. The average processing time looks reasonable.&lt;/p&gt;

&lt;p&gt;Yet your p99 latency keeps increasing.&lt;/p&gt;

&lt;p&gt;What is happening?&lt;/p&gt;

&lt;p&gt;In many cases, the problem isn't how long requests take to execute. It's how long they spend waiting before execution even begins.&lt;/p&gt;

&lt;p&gt;This is where &lt;strong&gt;queueing theory&lt;/strong&gt; becomes useful.&lt;/p&gt;

&lt;p&gt;Queueing theory gives us a way to reason about arrival rates, service capacity, concurrency, waiting time, and system saturation. It helps explain why a service can look healthy under moderate traffic and suddenly experience a dramatic increase in latency as it approaches its capacity.&lt;/p&gt;

&lt;p&gt;In this article, we'll explore the fundamentals of queueing theory and apply them to practical Go backend systems.&lt;/p&gt;

&lt;p&gt;We'll cover:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Arrival rate, service rate, and utilization&lt;/li&gt;
&lt;li&gt;Little's Law and its practical applications&lt;/li&gt;
&lt;li&gt;Why latency increases near saturation&lt;/li&gt;
&lt;li&gt;The relationship between concurrency and throughput&lt;/li&gt;
&lt;li&gt;Building bounded queues and worker pools in Go&lt;/li&gt;
&lt;li&gt;Queue sizing, admission control, and overload behavior&lt;/li&gt;
&lt;li&gt;Metrics and load-testing strategies&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The goal isn't to turn every backend engineer into a mathematician. It's to make better engineering decisions using a few surprisingly powerful relationships.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Every Backend Service Is a Queueing System
&lt;/h2&gt;

&lt;p&gt;Consider a typical HTTP service:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Client Requests
      |
      v
   HTTP Server
      |
      v
  Request Queue
      |
      v
  Worker Pool
      |
      v
 Database / API
      |
      v
   Response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every request follows a similar lifecycle:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;It arrives at the system.&lt;/li&gt;
&lt;li&gt;It waits for an available resource.&lt;/li&gt;
&lt;li&gt;It gets processed.&lt;/li&gt;
&lt;li&gt;It leaves the system.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The important distinction is that &lt;strong&gt;processing time and waiting time are different things&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Imagine two requests:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Request&lt;/th&gt;
&lt;th&gt;Waiting time&lt;/th&gt;
&lt;th&gt;Processing time&lt;/th&gt;
&lt;th&gt;Total&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A&lt;/td&gt;
&lt;td&gt;5 ms&lt;/td&gt;
&lt;td&gt;45 ms&lt;/td&gt;
&lt;td&gt;50 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B&lt;/td&gt;
&lt;td&gt;250 ms&lt;/td&gt;
&lt;td&gt;45 ms&lt;/td&gt;
&lt;td&gt;295 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Both requests require exactly the same processing time.&lt;/p&gt;

&lt;p&gt;But Request B takes almost six times longer because it spends most of its lifetime waiting.&lt;/p&gt;

&lt;p&gt;This is why optimizing database queries or application code alone doesn't always improve end-to-end latency.&lt;/p&gt;

&lt;p&gt;Sometimes the queue is the bottleneck.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. The Three Variables That Explain Queueing Behavior
&lt;/h2&gt;

&lt;p&gt;Most basic queueing models start with three variables.&lt;/p&gt;

&lt;h3&gt;
  
  
  Arrival Rate (λ)
&lt;/h3&gt;

&lt;p&gt;The arrival rate represents how frequently new requests enter the system.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;λ = 500 requests/second&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It describes incoming demand, not necessarily completed work.&lt;/p&gt;

&lt;h3&gt;
  
  
  Service Rate (μ)
&lt;/h3&gt;

&lt;p&gt;The service rate describes how quickly a worker can process requests.&lt;/p&gt;

&lt;p&gt;If a worker takes an average of 20 milliseconds per request:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;μ = 1 / 0.020 = 50 requests/second&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One worker can theoretically process 50 requests per second under that average service time.&lt;/p&gt;

&lt;p&gt;With 10 equivalent workers, the theoretical aggregate service capacity becomes:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;c × μ = 10 × 50 = 500 requests/second&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Here, &lt;code&gt;c&lt;/code&gt; represents the number of workers.&lt;/p&gt;

&lt;p&gt;This assumes the workers can operate independently and that another shared resource doesn't become the bottleneck.&lt;/p&gt;

&lt;h3&gt;
  
  
  Utilization (ρ)
&lt;/h3&gt;

&lt;p&gt;Utilization describes how much of the available service capacity is being consumed.&lt;/p&gt;

&lt;p&gt;For a simple system with &lt;code&gt;c&lt;/code&gt; equivalent workers:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ρ = λ / (c × μ)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Suppose:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Arrival rate: 400 requests/second&lt;/li&gt;
&lt;li&gt;Service rate per worker: 50 requests/second&lt;/li&gt;
&lt;li&gt;Number of workers: 10&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Then:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ρ = 400 / (10 × 50) = 0.8 = 80%&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The system is operating at 80% utilization.&lt;/p&gt;

&lt;p&gt;That sounds reasonable.&lt;/p&gt;

&lt;p&gt;But here's the important part: &lt;strong&gt;80% utilization does not mean 20% of the system is freely available at all times.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Traffic is variable. Requests don't arrive at perfectly regular intervals, and processing times aren't identical.&lt;/p&gt;

&lt;p&gt;A system can experience queueing delays even when its average utilization is below 100%.&lt;/p&gt;

&lt;p&gt;And as utilization approaches capacity, those delays can grow rapidly.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Little's Law: The Most Useful Queueing Relationship
&lt;/h2&gt;

&lt;p&gt;One of the most practical results in queueing theory is Little's Law:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;L = λ × W&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;L&lt;/code&gt; = average number of requests in the entire system&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;λ&lt;/code&gt; = average arrival rate&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;W&lt;/code&gt; = average time spent in the entire system&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In backend engineering, this gives us a direct relationship between throughput, latency, and concurrency.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Throughput: 200 requests/second
Average latency: 150 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Convert the latency to seconds:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;W = 0.150 seconds&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Then:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;L = 200 × 0.150 = 30&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;On average, there are approximately 30 requests in the system.&lt;/p&gt;

&lt;p&gt;This is incredibly useful because it allows us to estimate concurrency from production measurements.&lt;/p&gt;

&lt;p&gt;Let's look at another example.&lt;/p&gt;

&lt;p&gt;A service processes 1,000 requests per second, with an average time in the system of 80 milliseconds.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;L = 1,000 × 0.080 = 80&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The service has approximately 80 requests in flight on average.&lt;/p&gt;

&lt;p&gt;This doesn't mean we should automatically configure 80 workers.&lt;/p&gt;

&lt;p&gt;It means the observed workload has an average concurrency of approximately 80. Worker count, resource limits, queue capacity, and service time still need to be considered separately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Queue length versus requests in the system
&lt;/h3&gt;

&lt;p&gt;There's an important distinction here.&lt;/p&gt;

&lt;p&gt;Little's Law can describe either the entire system or just the waiting queue, provided the measurements use matching definitions.&lt;/p&gt;

&lt;p&gt;For the entire system:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;L = λ × W&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For the waiting queue:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Lq = λ × Wq&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Lq&lt;/code&gt; = average number of requests waiting in the queue&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;λ&lt;/code&gt; = average arrival rate into the queue&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Wq&lt;/code&gt; = average time spent waiting in the queue&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The distinction matters when instrumenting a Go worker pool.&lt;/p&gt;

&lt;p&gt;If we measure only queue depth, we shouldn't compare it directly with total request latency. We need to compare it with queue waiting time.&lt;/p&gt;

&lt;h3&gt;
  
  
  When should you use Little's Law?
&lt;/h3&gt;

&lt;p&gt;It's useful for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Estimating average in-flight requests&lt;/li&gt;
&lt;li&gt;Understanding queue growth&lt;/li&gt;
&lt;li&gt;Planning concurrency limits&lt;/li&gt;
&lt;li&gt;Checking whether observed metrics are consistent&lt;/li&gt;
&lt;li&gt;Estimating the waiting time implied by a queue&lt;/li&gt;
&lt;li&gt;Identifying where work accumulates&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;However, Little's Law describes long-run averages in a stable system. It doesn't predict p99 latency or tell us how a system behaves during a sudden traffic spike.&lt;/p&gt;

&lt;p&gt;For that, we need to understand utilization and variability.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Why Latency Explodes Near Saturation
&lt;/h2&gt;

&lt;p&gt;Let's consider a simplified single-server queue.&lt;/p&gt;

&lt;p&gt;Assume:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Requests arrive according to a Poisson process.&lt;/li&gt;
&lt;li&gt;Service times are exponentially distributed.&lt;/li&gt;
&lt;li&gt;There is one server.&lt;/li&gt;
&lt;li&gt;The queue has unlimited capacity.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is commonly called an &lt;strong&gt;M/M/1 queue&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;For this particular model, the expected time in the system is:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;W = 1 / (μ - λ)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;And the expected waiting time in the queue is:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Wq = ρ / (μ - λ)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;These formulas are model-specific. Real production systems often have multiple workers, correlated traffic, variable service times, and bounded queues.&lt;/p&gt;

&lt;p&gt;But the model illustrates an important behavior.&lt;/p&gt;

&lt;p&gt;Assume the server can process 100 requests per second.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Arrival rate&lt;/th&gt;
&lt;th&gt;Utilization&lt;/th&gt;
&lt;th&gt;Average time in system&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;50 req/s&lt;/td&gt;
&lt;td&gt;50%&lt;/td&gt;
&lt;td&gt;20 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;70 req/s&lt;/td&gt;
&lt;td&gt;70%&lt;/td&gt;
&lt;td&gt;33.3 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;80 req/s&lt;/td&gt;
&lt;td&gt;80%&lt;/td&gt;
&lt;td&gt;50 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;90 req/s&lt;/td&gt;
&lt;td&gt;90%&lt;/td&gt;
&lt;td&gt;100 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;95 req/s&lt;/td&gt;
&lt;td&gt;95%&lt;/td&gt;
&lt;td&gt;200 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;99 req/s&lt;/td&gt;
&lt;td&gt;99%&lt;/td&gt;
&lt;td&gt;1,000 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;Illustrative M/M/1 model, with a service rate of 100 requests per second.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The service rate never changes.&lt;/p&gt;

&lt;p&gt;Yet the average response time increases from 20 milliseconds to one full second as utilization approaches 100%.&lt;/p&gt;

&lt;p&gt;This is the queueing effect that makes operating a service continuously at maximum utilization dangerous.&lt;/p&gt;

&lt;p&gt;The system doesn't necessarily become slower at processing individual requests. It becomes slower at getting requests to the point where processing can begin.&lt;/p&gt;

&lt;p&gt;This also explains why a small traffic increase can cause a disproportionately large latency increase.&lt;/p&gt;

&lt;p&gt;In real systems, variability can make the situation worse. Bursty arrivals, slow dependencies, garbage collection pauses, lock contention, and uneven request costs all contribute to queueing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;High utilization leaves very little room to absorb variability.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Concurrency Is Not the Same as Throughput
&lt;/h2&gt;

&lt;p&gt;A common approach to improving performance is increasing concurrency.&lt;/p&gt;

&lt;p&gt;If 10 workers aren't enough, why not use 100?&lt;/p&gt;

&lt;p&gt;Because more concurrency doesn't necessarily produce more throughput.&lt;/p&gt;

&lt;p&gt;Imagine a service that queries a database.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Go Service
    |
    v
100 Concurrent Requests
    |
    v
Database Connection Pool
    |
    v
20 Available Connections
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Only 20 requests can use a database connection simultaneously.&lt;/p&gt;

&lt;p&gt;The remaining requests may wait for a connection.&lt;/p&gt;

&lt;p&gt;Increasing application concurrency from 100 to 500 doesn't increase database capacity.&lt;/p&gt;

&lt;p&gt;It may simply move the queue from the application into the database connection pool.&lt;/p&gt;

&lt;p&gt;This is an important principle:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Increasing concurrency can move the bottleneck without removing it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A useful way to reason about concurrency is Little's Law:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;L = λ × W&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If throughput remains constant while the number of concurrent requests increases, the average time spent in the system must also increase.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Throughput: 100 requests/second
Concurrency: 100
Average time: 1 second
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now suppose concurrency increases to 300, but throughput remains 100 requests per second.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Concurrency: 300
Average time: 3 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The system is carrying more work, but it isn't completing more work.&lt;/p&gt;

&lt;p&gt;This is why measuring concurrency without measuring throughput and latency can be misleading.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Implementing a Bounded Worker Pool in Go
&lt;/h2&gt;

&lt;p&gt;Let's move from theory to implementation.&lt;/p&gt;

&lt;p&gt;A worker pool is a common way to control how much work executes concurrently.&lt;/p&gt;

&lt;p&gt;But a worker pool alone isn't enough.&lt;/p&gt;

&lt;p&gt;If the incoming queue is unbounded, the service can continue accepting work long after its processing capacity is exhausted.&lt;/p&gt;

&lt;p&gt;A bounded queue provides an explicit limit.&lt;/p&gt;

&lt;p&gt;Here's a simplified implementation using Go channels:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;main&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"context"&lt;/span&gt;
    &lt;span class="s"&gt;"errors"&lt;/span&gt;
    &lt;span class="s"&gt;"sync"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ErrQueueFull&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"worker queue is full"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ErrPoolClosed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"worker pool is closed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Job&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Pool&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="n"&gt;Job&lt;/span&gt;

    &lt;span class="n"&gt;ctx&lt;/span&gt;    &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;
    &lt;span class="n"&gt;cancel&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CancelFunc&lt;/span&gt;

    &lt;span class="n"&gt;mu&lt;/span&gt;     &lt;span class="n"&gt;sync&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Mutex&lt;/span&gt;
    &lt;span class="n"&gt;closed&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;

    &lt;span class="n"&gt;wg&lt;/span&gt; &lt;span class="n"&gt;sync&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WaitGroup&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;NewPool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;parent&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;workers&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;queueSize&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Pool&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;workers&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nb"&gt;panic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"workers must be greater than zero"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;queueSize&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nb"&gt;panic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"queue size cannot be negative"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WithCancel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;   &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="n"&gt;Job&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;queueSize&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;    &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;workers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;workers&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;go&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;worker&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Submit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="n"&gt;Job&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"job cannot be nil"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;closed&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ErrPoolClosed&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
    &lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ErrQueueFull&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;worker&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;closed&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;closed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;true&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Wait&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The implementation demonstrates a few important ideas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The number of workers is fixed.&lt;/li&gt;
&lt;li&gt;The queue has a bounded capacity.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Submit&lt;/code&gt; rejects work immediately when the queue is full.&lt;/li&gt;
&lt;li&gt;Worker execution receives a cancellation context.&lt;/li&gt;
&lt;li&gt;Submission and shutdown are coordinated using a mutex.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;Close&lt;/code&gt; cancels workers and waits for them to exit.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is an intentionally small example, not a complete production job-processing framework.&lt;/p&gt;

&lt;p&gt;In particular, it uses immediate cancellation rather than graceful draining. It also doesn't provide per-job results, panic recovery, retry handling, or error reporting.&lt;/p&gt;

&lt;p&gt;Those behaviors should be designed explicitly for the workload.&lt;/p&gt;

&lt;p&gt;Also remember that context cancellation is cooperative. A job that ignores its context can prevent &lt;code&gt;Close&lt;/code&gt; from returning.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why bounded queues matter
&lt;/h3&gt;

&lt;p&gt;Consider a service with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Workers: 20
Queue capacity: 500
Average processing time: 100 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each worker can theoretically process:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;μ = 1 / 0.1 = 10 jobs/second&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Aggregate service capacity:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;c × μ = 20 × 10 = 200 jobs/second&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If all workers are busy and the queue contains 500 jobs, the approximate queue-draining time is:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Wq ≈ 500 / 200 = 2.5 seconds&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is a rough estimate assuming the workers continue processing at the same aggregate rate and no new jobs arrive.&lt;/p&gt;

&lt;p&gt;If your API has a 500-millisecond response deadline, accepting work into this queue may be pointless.&lt;/p&gt;

&lt;p&gt;The request might time out before a worker even starts processing it.&lt;/p&gt;

&lt;p&gt;A queue is not automatically useful just because it can hold more work.&lt;/p&gt;

&lt;p&gt;It should hold only the amount of work your system can reasonably process before that work becomes irrelevant.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Queue Capacity Is a Latency Decision
&lt;/h2&gt;

&lt;p&gt;Developers often select queue capacity using arbitrary values:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="n"&gt;Job&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But why 1,000?&lt;/p&gt;

&lt;p&gt;Why not 100 or 10,000?&lt;/p&gt;

&lt;p&gt;A better approach is to start with a latency budget.&lt;/p&gt;

&lt;p&gt;Suppose:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Maximum acceptable queue waiting time: 500 ms
Aggregate processing capacity: 200 jobs/second
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A rough initial queue capacity can be estimated as:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Queue Capacity ≈ Aggregate Service Capacity × Target Queue Waiting Time&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Given:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Aggregate service capacity: 200 jobs/second&lt;/li&gt;
&lt;li&gt;Target queue waiting time: 0.5 seconds&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The calculation becomes:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Q ≈ 200 × 0.5 = 100 jobs&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This suggests that a queue capacity around 100 jobs corresponds to approximately 500 milliseconds of waiting at the assumed completion rate, if the queue is full and no new jobs arrive.&lt;/p&gt;

&lt;p&gt;This is not a worst-case guarantee.&lt;/p&gt;

&lt;p&gt;Actual waiting time depends on worker availability, service-time distribution, traffic bursts, and the number of jobs already executing.&lt;/p&gt;

&lt;p&gt;Still, it gives us a much more defensible starting point than choosing a large number without understanding its consequences.&lt;/p&gt;

&lt;h3&gt;
  
  
  Queue age is often more useful than queue depth
&lt;/h3&gt;

&lt;p&gt;Imagine two systems:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;System A:
Queue depth: 100
Oldest job: 10 ms

System B:
Queue depth: 20
Oldest job: 2 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;System B has fewer queued jobs, but its users may be experiencing a much worse situation.&lt;/p&gt;

&lt;p&gt;Queue depth tells us how much work is waiting.&lt;/p&gt;

&lt;p&gt;Queue age tells us how long that work has already been waiting.&lt;/p&gt;

&lt;p&gt;For latency-sensitive systems, both metrics are important.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Admission Control: Decide Whether Work Should Enter
&lt;/h2&gt;

&lt;p&gt;A bounded queue provides a capacity limit.&lt;/p&gt;

&lt;p&gt;Admission control adds a policy for deciding whether incoming work should be accepted.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;HandleRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;queue&lt;/span&gt; &lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="n"&gt;Job&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="n"&gt;Job&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;queue&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;

    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ErrQueueFull&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a simplified non-blocking admission decision.&lt;/p&gt;

&lt;p&gt;When the queue is full, the request is rejected instead of waiting indefinitely.&lt;/p&gt;

&lt;p&gt;In a real HTTP handler, this could translate into an overload response such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="k"&gt;HTTP&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="m"&gt;1.1&lt;/span&gt; &lt;span class="m"&gt;503&lt;/span&gt; &lt;span class="ne"&gt;Service Unavailable&lt;/span&gt;
&lt;span class="na"&gt;Retry-After&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact status and retry policy depend on the API contract.&lt;/p&gt;

&lt;p&gt;There are several possible admission strategies:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reject immediately&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Useful when waiting is unlikely to produce a result within the request deadline.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Wait for a bounded period&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Useful when short bursts are expected and a small amount of waiting is acceptable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Prioritize work&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Useful when critical operations should be processed ahead of background or low-priority tasks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Apply load shedding&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Useful when the system is overloaded and preserving the health of the remaining workload is more important than accepting every request.&lt;/p&gt;

&lt;p&gt;The important part is to make overload behavior intentional.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. The Connection to Backpressure, Load Shedding, and Adaptive Concurrency
&lt;/h2&gt;

&lt;p&gt;Queueing theory provides the foundation for several resilience patterns we've discussed in this series.&lt;/p&gt;

&lt;p&gt;They solve different parts of the same problem.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Pattern&lt;/th&gt;
&lt;th&gt;What it controls&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Backpressure&lt;/td&gt;
&lt;td&gt;How quickly work enters the system&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Load shedding&lt;/td&gt;
&lt;td&gt;Which work should be rejected&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bulkheads&lt;/td&gt;
&lt;td&gt;How resources are isolated&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retry budgets&lt;/td&gt;
&lt;td&gt;How much additional traffic retries generate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Adaptive concurrency&lt;/td&gt;
&lt;td&gt;How many requests execute simultaneously&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hedged requests&lt;/td&gt;
&lt;td&gt;How to reduce tail latency through controlled parallelism&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These mechanisms become much more effective when we understand the queueing behavior behind them.&lt;/p&gt;

&lt;p&gt;For example, an adaptive concurrency controller can reduce the number of in-flight requests when latency indicates that a dependency is approaching saturation.&lt;/p&gt;

&lt;p&gt;A bounded queue prevents the service from accumulating unlimited waiting work.&lt;/p&gt;

&lt;p&gt;Load shedding rejects requests that cannot be processed within a useful time window.&lt;/p&gt;

&lt;p&gt;Together, these mechanisms help keep the system stable under changing traffic conditions.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. What Should You Measure in Production?
&lt;/h2&gt;

&lt;p&gt;Queueing theory is useful only when its variables can be observed.&lt;/p&gt;

&lt;p&gt;At a minimum, I'd monitor the following metrics.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Why it matters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Arrival rate&lt;/td&gt;
&lt;td&gt;Incoming demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Completion rate&lt;/td&gt;
&lt;td&gt;Actual processing capacity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Queue depth&lt;/td&gt;
&lt;td&gt;Amount of waiting work&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Queue age&lt;/td&gt;
&lt;td&gt;How long work has been waiting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;In-flight requests&lt;/td&gt;
&lt;td&gt;Current concurrency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Queue wait duration&lt;/td&gt;
&lt;td&gt;Time spent waiting before execution&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service duration&lt;/td&gt;
&lt;td&gt;Actual processing time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rejected requests&lt;/td&gt;
&lt;td&gt;Admission-control behavior&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p95 / p99 latency&lt;/td&gt;
&lt;td&gt;User-facing latency distribution&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A particularly useful metric is the difference between total request duration and processing duration.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Total request duration: 350 ms
Queue wait:              250 ms
Processing duration:     100 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If processing duration is stable but total latency increases, the queue is a strong candidate for investigation.&lt;/p&gt;

&lt;p&gt;For Go services, you can expose these measurements through Prometheus.&lt;/p&gt;

&lt;p&gt;Example metric names:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;http_requests_total
http_request_duration_seconds

worker_pool_queue_depth
worker_pool_queue_wait_seconds
worker_pool_active_workers
worker_pool_jobs_rejected_total
worker_pool_job_duration_seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use histograms for durations and counters for events. Queue depth and active workers are gauges.&lt;/p&gt;

&lt;p&gt;Avoid putting unbounded values such as user IDs, request IDs, or raw URLs into metric labels. That creates unnecessary cardinality.&lt;/p&gt;

&lt;h3&gt;
  
  
  A practical diagnostic example
&lt;/h3&gt;

&lt;p&gt;Suppose a production service starts showing the following behavior:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Arrival rate:          800 req/s
Completion rate:       790 req/s
Active workers:        100
Queue depth:           Increasing
Queue wait p95:        450 ms
Processing p95:        60 ms
HTTP latency p95:      520 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first instinct might be to optimize the handler.&lt;/p&gt;

&lt;p&gt;But the measurements tell a different story.&lt;/p&gt;

&lt;p&gt;Processing time is relatively low compared with total latency.&lt;/p&gt;

&lt;p&gt;The queue is accumulating work, and the service is completing slightly fewer requests than it receives.&lt;/p&gt;

&lt;p&gt;Adding more workers may help only if the downstream resources have spare capacity.&lt;/p&gt;

&lt;p&gt;Before increasing concurrency, investigate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Database connection pool saturation&lt;/li&gt;
&lt;li&gt;Downstream API latency&lt;/li&gt;
&lt;li&gt;CPU and memory pressure&lt;/li&gt;
&lt;li&gt;Lock contention&lt;/li&gt;
&lt;li&gt;Uneven job execution times&lt;/li&gt;
&lt;li&gt;Whether requests are arriving in bursts&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The goal is to identify the actual bottleneck instead of simply increasing the number of workers.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Testing Queueing Behavior
&lt;/h2&gt;

&lt;p&gt;A worker pool that works correctly with 10 requests may behave very differently with 10,000.&lt;/p&gt;

&lt;p&gt;Load tests should examine more than maximum throughput.&lt;/p&gt;

&lt;p&gt;I'd test at least four scenarios.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Normal load&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Run below expected capacity and establish baseline latency, throughput, and resource usage.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Near-saturation load&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Gradually increase arrival rate toward service capacity. Observe how queue wait and p99 latency change.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Burst traffic&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Send a large number of requests in a short interval. Verify that the queue remains bounded and overload behavior is predictable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Dependency slowdown&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Artificially increase downstream service time. Observe whether concurrency, queue depth, and rejection rates remain controlled.&lt;/p&gt;

&lt;p&gt;For a Go worker pool, also test:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Queue-full behavior&lt;/li&gt;
&lt;li&gt;Cancellation before execution&lt;/li&gt;
&lt;li&gt;Shutdown while workers are active&lt;/li&gt;
&lt;li&gt;Jobs that ignore context cancellation&lt;/li&gt;
&lt;li&gt;Worker panics, if panic recovery is part of the design&lt;/li&gt;
&lt;li&gt;Concurrent submission during shutdown&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A particularly important test is whether your service continues accepting work after its useful processing capacity has been exhausted.&lt;/p&gt;

&lt;p&gt;A successful load test isn't necessarily one where every request is accepted.&lt;/p&gt;

&lt;p&gt;Sometimes the better result is a system that rejects excess work quickly while keeping accepted requests within their latency objectives.&lt;/p&gt;

&lt;h2&gt;
  
  
  12. A Practical Checklist
&lt;/h2&gt;

&lt;p&gt;Before deploying a queue-based Go service, ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Is the queue bounded?&lt;/li&gt;
&lt;li&gt;Is the queue capacity based on a latency budget?&lt;/li&gt;
&lt;li&gt;Do we measure queue wait time separately from processing time?&lt;/li&gt;
&lt;li&gt;Can requests be rejected when the system is saturated?&lt;/li&gt;
&lt;li&gt;Do jobs respect context cancellation?&lt;/li&gt;
&lt;li&gt;Can shutdown stop workers without leaking goroutines?&lt;/li&gt;
&lt;li&gt;Are downstream connection pools and concurrency limits understood?&lt;/li&gt;
&lt;li&gt;Do we monitor queue depth and oldest-job age?&lt;/li&gt;
&lt;li&gt;Have we tested burst traffic and dependency slowdown?&lt;/li&gt;
&lt;li&gt;Are we optimizing useful throughput rather than simply maximizing accepted requests?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These questions help turn queueing theory into concrete engineering decisions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Queueing theory explains something that's easy to overlook when building backend services:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A request doesn't need to be expensive to become slow. It only needs to wait.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Little's Law helps us reason about concurrency and latency. Utilization explains why waiting time increases rapidly near capacity. Bounded queues and admission control give us ways to prevent uncontrolled accumulation of work.&lt;/p&gt;

&lt;p&gt;In Go, goroutines and channels make it relatively easy to build concurrent systems. But concurrency itself isn't a capacity-planning strategy.&lt;/p&gt;

&lt;p&gt;The important questions are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How much work enters the system?&lt;/li&gt;
&lt;li&gt;How quickly can the system complete it?&lt;/li&gt;
&lt;li&gt;How much work can wait?&lt;/li&gt;
&lt;li&gt;How long can that work remain useful?&lt;/li&gt;
&lt;li&gt;What happens when demand exceeds capacity?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A reliable service doesn't need to accept every request.&lt;/p&gt;

&lt;p&gt;It needs to process accepted work predictably, protect its dependencies, and fail fast when it can no longer meet its latency objectives.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The goal isn't to keep every worker busy. It's to keep the system healthy and useful under load.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://pubsonline.informs.org/doi/10.1287/inte.2011.0613" rel="noopener noreferrer"&gt;Little's Law as Viewed on its 50th Anniversary — INFORMS&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://web.mit.edu/1.041/spring2023/lectures/L8-queuing-models-2023sp.pdf" rel="noopener noreferrer"&gt;MIT — Queueing Models and Little's Law&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://doi.org/10.1145/3232559" rel="noopener noreferrer"&gt;Amdahl's Law for Tail Latency — Communications of the ACM&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pkg.go.dev/context" rel="noopener noreferrer"&gt;Go Documentation — Context&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://go.dev/ref/mem" rel="noopener noreferrer"&gt;Go Documentation — Go Memory Model&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>go</category>
      <category>backend</category>
      <category>programming</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Observability for AI Agents: Tracing, Evaluation and Cost Control in Production</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Mon, 02 Nov 2026 18:10:12 +0000</pubDate>
      <link>https://dev.to/serifcolakel/observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production-1ln</link>
      <guid>https://dev.to/serifcolakel/observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production-1ln</guid>
      <description>&lt;p&gt;&lt;em&gt;How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Why Is Observability Different for AI Agents?
&lt;/h2&gt;

&lt;p&gt;Traditional backend observability is built around relatively predictable execution paths.&lt;/p&gt;

&lt;p&gt;A request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred.&lt;/p&gt;

&lt;p&gt;AI agents introduce a different level of complexity.&lt;/p&gt;

&lt;p&gt;An agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input.&lt;/p&gt;

&lt;p&gt;Consider a relatively simple task:&lt;/p&gt;

&lt;p&gt;A user asks an AI assistant to analyze a merge request and identify potential problems.&lt;/p&gt;

&lt;p&gt;Behind the scenes, the application might:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Receive the user's request.&lt;/li&gt;
&lt;li&gt;Invoke a coordinator agent.&lt;/li&gt;
&lt;li&gt;Call an LLM to determine the next action.&lt;/li&gt;
&lt;li&gt;Retrieve the merge request from GitLab.&lt;/li&gt;
&lt;li&gt;Read the changed files.&lt;/li&gt;
&lt;li&gt;Delegate security analysis to another agent.&lt;/li&gt;
&lt;li&gt;Execute additional tool calls.&lt;/li&gt;
&lt;li&gt;Evaluate the collected results.&lt;/li&gt;
&lt;li&gt;Generate a final report.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The user sees one response. The application has actually executed a multi-step workflow involving several components.&lt;/p&gt;

&lt;p&gt;Now imagine the response takes 45 seconds instead of the expected 8 seconds.&lt;/p&gt;

&lt;p&gt;Where should we investigate?&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Did the model take too long to respond?&lt;/li&gt;
&lt;li&gt;Did a tool call time out?&lt;/li&gt;
&lt;li&gt;Was the same operation executed repeatedly?&lt;/li&gt;
&lt;li&gt;Did the agent generate unnecessary steps?&lt;/li&gt;
&lt;li&gt;Did a subagent introduce additional latency?&lt;/li&gt;
&lt;li&gt;Was too much context sent to the model?&lt;/li&gt;
&lt;li&gt;Did a retry increase the overall execution time and cost?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Traditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AI observability is about understanding the entire execution path, not just measuring the final API response.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This is where OpenTelemetry becomes particularly useful.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. What Is OpenTelemetry?
&lt;/h2&gt;

&lt;p&gt;** is an open-source observability framework for collecting, processing, and exporting telemetry data.&lt;/p&gt;

&lt;p&gt;It provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms.&lt;/p&gt;

&lt;p&gt;Instead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported.&lt;/p&gt;

&lt;p&gt;OpenTelemetry is built around three primary signals:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;What it tells us&lt;/th&gt;
&lt;th&gt;AI agent example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Traces&lt;/td&gt;
&lt;td&gt;How an operation was executed&lt;/td&gt;
&lt;td&gt;Model call → tool → subagent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Metrics&lt;/td&gt;
&lt;td&gt;How the system performs over time&lt;/td&gt;
&lt;td&gt;Token usage, latency, error rate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Logs&lt;/td&gt;
&lt;td&gt;What happened at a specific point&lt;/td&gt;
&lt;td&gt;Tool timeout or agent exception&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These signals complement each other.&lt;/p&gt;

&lt;p&gt;A metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation.&lt;/p&gt;

&lt;p&gt;OpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector.&lt;/p&gt;

&lt;p&gt;An important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://opentelemetry.io/docs/" rel="noopener noreferrer"&gt;OpenTelemetry Documentation&lt;/a&gt; · &lt;a href="https://opentelemetry.io/docs/languages/js/" rel="noopener noreferrer"&gt;OpenTelemetry JavaScript&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Understanding Distributed Tracing
&lt;/h2&gt;

&lt;p&gt;Before discussing AI-specific instrumentation, we need to understand traces and spans.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;trace&lt;/strong&gt; represents the complete execution path of an operation.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;span&lt;/strong&gt; represents one individual unit of work within that execution.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Trace: POST /api/agent
|
+-- HTTP Request
    |
    +-- Coordinator Agent
        |
        +-- LLM Generation
        |
        +-- Tool: GitLab - Get Merge Request
        |
        +-- Tool: GitLab - Get Changed Files
        |
        +-- Security Agent
        |   |
        |   +-- LLM Generation
        |   |
        |   +-- Tool: Security Analysis
        |
        +-- Testing Agent
        |   |
        |   +-- LLM Generation
        |   |
        |   +-- Tool: Test Coverage
        |
        +-- Final Response Generation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each span has information such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Trace ID&lt;/li&gt;
&lt;li&gt;Span ID&lt;/li&gt;
&lt;li&gt;Parent span ID&lt;/li&gt;
&lt;li&gt;Start and end timestamps&lt;/li&gt;
&lt;li&gt;Duration&lt;/li&gt;
&lt;li&gt;Attributes&lt;/li&gt;
&lt;li&gt;Events&lt;/li&gt;
&lt;li&gt;Status and exception information&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This creates a hierarchical representation of the complete execution.&lt;/p&gt;

&lt;p&gt;The key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them.&lt;/p&gt;

&lt;p&gt;For AI agents, this distinction is important because one user request can generate many independent operations.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why is context propagation important?
&lt;/h3&gt;

&lt;p&gt;In distributed systems, a trace often crosses multiple services.&lt;/p&gt;

&lt;p&gt;Imagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service.&lt;/p&gt;

&lt;p&gt;Without context propagation, every service may generate its own independent trace.&lt;/p&gt;

&lt;p&gt;We would see several unrelated operations instead of one complete workflow.&lt;/p&gt;

&lt;p&gt;OpenTelemetry uses W3C Trace Context as its standard propagation format. The &lt;code&gt;traceparent&lt;/code&gt; header carries trace and parent-span information between instrumented services.&lt;/p&gt;

&lt;p&gt;The same principle applies to agent-based architectures.&lt;/p&gt;

&lt;p&gt;When an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution.&lt;/p&gt;

&lt;p&gt;Automatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction.&lt;/p&gt;

&lt;p&gt;The important point is that &lt;strong&gt;protocol-level interoperability and observability context propagation are separate concerns&lt;/strong&gt;. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://opentelemetry.io/docs/concepts/signals/traces/" rel="noopener noreferrer"&gt;OpenTelemetry Traces&lt;/a&gt; · &lt;a href="https://opentelemetry.io/docs/concepts/context-propagation/" rel="noopener noreferrer"&gt;Context Propagation&lt;/a&gt; · &lt;a href="https://www.w3.org/TR/trace-context/" rel="noopener noreferrer"&gt;W3C Trace Context&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  4. OpenTelemetry and GenAI Semantic Conventions
&lt;/h2&gt;

&lt;p&gt;One of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers.&lt;/p&gt;

&lt;p&gt;For example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes.&lt;/p&gt;

&lt;p&gt;This makes it difficult to compare different implementations using the same monitoring platform.&lt;/p&gt;

&lt;p&gt;OpenTelemetry addresses this through &lt;strong&gt;Semantic Conventions&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Semantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently.&lt;/p&gt;

&lt;p&gt;The GenAI semantic conventions extend this approach to generative AI applications.&lt;/p&gt;

&lt;p&gt;Some important attributes include:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Attribute&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.provider.name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Identifies the model provider&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.request.model&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Identifies the requested model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.response.model&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Identifies the actual response model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.operation.name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Describes the operation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.usage.input_tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Input token usage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.usage.output_tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Output token usage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.conversation.id&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Associates operations with a conversation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.agent.name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Identifies the agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gen_ai.tool.name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Identifies the executed tool&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For example, an LLM invocation could be represented by:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"gen_ai.operation.name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"chat"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"gen_ai.provider.name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"openai"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"gen_ai.request.model"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"example-model"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"gen_ai.usage.input_tokens"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"gen_ai.usage.output_tokens"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;350&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a simplified span attribute example.&lt;/p&gt;

&lt;p&gt;The benefit is that observability platforms do not need to understand a different custom schema for every application.&lt;/p&gt;

&lt;h3&gt;
  
  
  Agent-specific conventions
&lt;/h3&gt;

&lt;p&gt;The GenAI semantic conventions also define operations for agents and workflows.&lt;/p&gt;

&lt;p&gt;Some examples include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;invoke_agent&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;invoke_workflow&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;execute_tool&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;plan&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;retrieval&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is particularly valuable for multi-agent architectures.&lt;/p&gt;

&lt;p&gt;We can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;invoke_workflow merge-request-review
|
+-- invoke_agent coordinator
|   |
|   +-- chat model
|   +-- execute_tool fetch_merge_request
|
+-- invoke_agent security-agent
|   |
|   +-- chat model
|   +-- execute_tool analyze_vulnerabilities
|
+-- invoke_agent testing-agent
|   |
|   +-- chat model
|   +-- execute_tool analyze_test_coverage
|
+-- Final Aggregation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This makes the execution model much easier to understand.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;An important maturity consideration:&lt;/strong&gt; As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract.&lt;/p&gt;

&lt;p&gt;The current convention documents include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md" rel="noopener noreferrer"&gt;Semantic Conventions for GenAI Spans&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md" rel="noopener noreferrer"&gt;Semantic Conventions for Agent Spans&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md" rel="noopener noreferrer"&gt;Semantic Conventions for GenAI Metrics&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  5. Integrating OpenTelemetry with Vercel AI SDK
&lt;/h2&gt;

&lt;p&gt;Since many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration.&lt;/p&gt;

&lt;p&gt;The AI SDK already supports OpenTelemetry-based instrumentation.&lt;/p&gt;

&lt;p&gt;It can generate spans for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Text generation&lt;/li&gt;
&lt;li&gt;Streaming generation&lt;/li&gt;
&lt;li&gt;Individual model calls&lt;/li&gt;
&lt;li&gt;Agent execution&lt;/li&gt;
&lt;li&gt;Tool execution&lt;/li&gt;
&lt;li&gt;Embedding operations&lt;/li&gt;
&lt;li&gt;Reranking&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Install the packages
&lt;/h3&gt;

&lt;p&gt;For an AI SDK v7 application:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm add @ai-sdk/otel @vercel/otel
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The OpenTelemetry integration is provided through &lt;code&gt;@ai-sdk/otel&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This is an important version-specific detail.&lt;/p&gt;

&lt;p&gt;In AI SDK v7, OpenTelemetry integration moved out of the core &lt;code&gt;ai&lt;/code&gt; package. Older examples using the v6 &lt;code&gt;experimental_telemetry&lt;/code&gt; configuration should not be copied directly into a v7 application.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Register telemetry in Next.js
&lt;/h3&gt;

&lt;p&gt;Create an &lt;code&gt;instrumentation.ts&lt;/code&gt; file in the project root.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;registerOTel&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@vercel/otel&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;registerTelemetry&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ai&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;OpenTelemetry&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@ai-sdk/otel&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;register&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nf"&gt;registerOTel&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;serviceName&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ai-agent-platform&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="nf"&gt;registerTelemetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;OpenTelemetry&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
      &lt;span class="na"&gt;recordInputs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;recordOutputs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}),&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There are two responsibilities here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;registerOTel&lt;/code&gt; initializes the OpenTelemetry infrastructure for the Next.js application.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;registerTelemetry&lt;/code&gt; registers the AI SDK-specific integration that creates GenAI spans.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I have disabled input and output capture in this example intentionally.&lt;/p&gt;

&lt;p&gt;AI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision.&lt;/p&gt;

&lt;p&gt;We will return to this in the security section.&lt;/p&gt;

&lt;p&gt;For a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Instrument AI SDK calls
&lt;/h3&gt;

&lt;p&gt;Once telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration.&lt;/p&gt;

&lt;p&gt;We can also provide a &lt;code&gt;functionId&lt;/code&gt; to identify the business operation or agent.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;generateText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ai&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateText&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;myModel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="na"&gt;instructions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`
    You are a financial analysis assistant.
    Analyze the supplied information carefully.
  `&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="na"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Analyze the latest portfolio performance.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

  &lt;span class="na"&gt;telemetry&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;functionId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;portfolio-analysis-agent&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;functionId&lt;/code&gt; helps distinguish operations in traces.&lt;/p&gt;

&lt;p&gt;For example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;invoke_agent portfolio-analysis-agent
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 4: Understand the generated spans
&lt;/h3&gt;

&lt;p&gt;The AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions.&lt;/p&gt;

&lt;p&gt;For &lt;code&gt;generateText&lt;/code&gt; and &lt;code&gt;streamText&lt;/code&gt;, the integration records three main types of spans:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Agent invocation span&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The root span represents the complete generation operation, including its execution steps and tool calls.&lt;/p&gt;

&lt;p&gt;Example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;invoke_agent portfolio-analysis-agent
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;2. Step and model-call spans&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Each model interaction is represented by step and provider-call spans.&lt;/p&gt;

&lt;p&gt;These allow us to understand how many model calls occurred during an operation and how long each one took.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Tool execution spans&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Each client-side tool execution receives its own span.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;invoke_agent portfolio-analysis-agent
|
+-- step 0
|   |
|   +-- chat example-model
|   +-- execute_tool get_portfolio
|
+-- step 1
|   |
|   +-- chat example-model
|   +-- execute_tool get_market_data
|
+-- step 2
    |
    +-- chat example-model
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The resulting trace provides a much more complete representation of an agent workflow than a single model request log.&lt;/p&gt;

&lt;p&gt;The integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Add additional application context
&lt;/h3&gt;

&lt;p&gt;Sometimes the standard GenAI attributes are not enough.&lt;/p&gt;

&lt;p&gt;For example, we may want to know which business workflow initiated an operation or which application component owns it.&lt;/p&gt;

&lt;p&gt;The AI SDK provides &lt;code&gt;enrichSpan&lt;/code&gt; for adding custom attributes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;registerTelemetry&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ai&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;OpenTelemetry&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@ai-sdk/otel&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nf"&gt;registerTelemetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;OpenTelemetry&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;recordInputs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;recordOutputs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="na"&gt;enrichSpan&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;spanType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;operationId&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;({&lt;/span&gt;
      &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;app.component&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;portfolio-assistant&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;app.operation_id&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;operationId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;app.span_type&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;spanType&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}),&lt;/span&gt;
  &lt;span class="p"&gt;}),&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact context available to &lt;code&gt;enrichSpan&lt;/code&gt; depends on the span and the SDK's integration API.&lt;/p&gt;

&lt;p&gt;The objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://ai-sdk.dev/docs/ai-sdk-core/telemetry" rel="noopener noreferrer"&gt;AI SDK Telemetry&lt;/a&gt; · &lt;a href="https://nextjs.org/docs/app/guides/open-telemetry" rel="noopener noreferrer"&gt;Next.js OpenTelemetry&lt;/a&gt; · &lt;a href="https://github.com/vercel/ai/blob/main/skills/migrate-ai-sdk-v6-to-v7/SKILL.md" rel="noopener noreferrer"&gt;AI SDK v7 Migration Guide&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Instrumenting Your Own Agent Runtime
&lt;/h2&gt;

&lt;p&gt;Built-in instrumentation is useful, but it does not automatically understand every application-level decision.&lt;/p&gt;

&lt;p&gt;For example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval.&lt;/p&gt;

&lt;p&gt;These are application-level operations.&lt;/p&gt;

&lt;p&gt;We should instrument them explicitly.&lt;/p&gt;

&lt;p&gt;OpenTelemetry provides a tracing API for creating custom spans.&lt;/p&gt;

&lt;p&gt;Consider a custom agent orchestration function:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;SpanStatusCode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@opentelemetry/api&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;tracer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;trace&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getTracer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;agent-orchestrator&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;runReviewWorkflow&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;tracer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startActiveSpan&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;workflow.merge-request-review&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setAttribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;workflow.name&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;merge-request-review&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;);&lt;/span&gt;

      &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setAttribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;workflow.version&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;1.0.0&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;);&lt;/span&gt;

      &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;executeReview&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

        &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setStatus&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
          &lt;span class="na"&gt;code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;SpanStatusCode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;OK&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;});&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;cause&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
          &lt;span class="nx"&gt;cause&lt;/span&gt; &lt;span class="k"&gt;instanceof&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;
            &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;cause&lt;/span&gt;
            &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;cause&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;

        &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;recordException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

        &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setStatus&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
          &lt;span class="na"&gt;code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;SpanStatusCode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ERROR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;});&lt;/span&gt;

        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;finally&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nx"&gt;span&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Any supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated.&lt;/p&gt;

&lt;p&gt;We now have two levels of observability:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Framework-level instrumentation describes model and tool execution.&lt;/li&gt;
&lt;li&gt;Application-level instrumentation describes business workflows and orchestration decisions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This distinction is important.&lt;/p&gt;

&lt;p&gt;We should not expect a model instrumentation library to understand the complete business meaning of an application request.&lt;/p&gt;

&lt;h3&gt;
  
  
  Instrumenting remote agent calls
&lt;/h3&gt;

&lt;p&gt;For A2A-based systems, distributed tracing becomes even more important.&lt;/p&gt;

&lt;p&gt;Imagine a coordinator running in one service and a security agent running in another service.&lt;/p&gt;

&lt;p&gt;The coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Trace: merge-request-review
|
+-- Coordinator Agent
|   |
|   +-- Model Call
|   |
|   +-- A2A: Delegate to Security Agent
|       |
|       +-- Security Agent
|           |
|           +-- Model Call
|           |
|           +-- Tool Execution
|
+-- Final Aggregation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The HTTP request must carry the appropriate trace context, typically through W3C &lt;code&gt;traceparent&lt;/code&gt; and &lt;code&gt;tracestate&lt;/code&gt; headers.&lt;/p&gt;

&lt;p&gt;For custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context.&lt;/p&gt;

&lt;p&gt;This is how we avoid fragmented traces across independently deployed agents.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. What Should We Measure?
&lt;/h2&gt;

&lt;p&gt;Collecting telemetry is not enough.&lt;/p&gt;

&lt;p&gt;We need to define which metrics are useful for understanding the health and quality of an agent platform.&lt;/p&gt;

&lt;p&gt;I would divide AI observability metrics into five categories.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.1. Latency Metrics
&lt;/h3&gt;

&lt;p&gt;These describe how long operations take.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Why it matters&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;End-to-end workflow duration&lt;/td&gt;
&lt;td&gt;Measures total user-facing execution time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model request duration&lt;/td&gt;
&lt;td&gt;Identifies slow inference calls&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Time to first token&lt;/td&gt;
&lt;td&gt;Measures perceived response latency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool execution duration&lt;/td&gt;
&lt;td&gt;Identifies slow external operations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent invocation duration&lt;/td&gt;
&lt;td&gt;Measures the execution time of one agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Queue wait duration&lt;/td&gt;
&lt;td&gt;Identifies scheduling bottlenecks&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For streaming applications, time to first token is particularly useful.&lt;/p&gt;

&lt;p&gt;A workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.2. Reliability Metrics
&lt;/h3&gt;

&lt;p&gt;These help us understand whether workflows complete successfully.&lt;/p&gt;

&lt;p&gt;Examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agent execution success rate&lt;/li&gt;
&lt;li&gt;Tool execution failure rate&lt;/li&gt;
&lt;li&gt;Model request failure rate&lt;/li&gt;
&lt;li&gt;Timeout rate&lt;/li&gt;
&lt;li&gt;Retry rate&lt;/li&gt;
&lt;li&gt;Task cancellation rate&lt;/li&gt;
&lt;li&gt;Human intervention rate&lt;/li&gt;
&lt;li&gt;Incomplete workflow rate&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A high retry rate can be an early indicator of a problematic integration or an unstable external dependency.&lt;/p&gt;

&lt;p&gt;We should also distinguish between a successful model response and a successful business workflow.&lt;/p&gt;

&lt;p&gt;An LLM can return a perfectly valid response while the overall task fails to achieve its objective.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.3. Token and Model Usage
&lt;/h3&gt;

&lt;p&gt;Token usage should be tracked at multiple levels.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Input tokens&lt;/td&gt;
&lt;td&gt;Tokens sent to the model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output tokens&lt;/td&gt;
&lt;td&gt;Tokens generated by the model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cached input tokens&lt;/td&gt;
&lt;td&gt;Tokens served from provider cache&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reasoning tokens&lt;/td&gt;
&lt;td&gt;Model-reported reasoning token usage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tokens per workflow&lt;/td&gt;
&lt;td&gt;Total usage across an entire task&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tokens per agent&lt;/td&gt;
&lt;td&gt;Usage attributed to an individual agent&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.4. Agent and Tool Behavior
&lt;/h3&gt;

&lt;p&gt;These metrics describe how efficiently agents perform their work.&lt;/p&gt;

&lt;p&gt;Examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Number of tool calls per task&lt;/li&gt;
&lt;li&gt;Number of model calls per agent invocation&lt;/li&gt;
&lt;li&gt;Number of agent handoffs&lt;/li&gt;
&lt;li&gt;Repeated tool-call frequency&lt;/li&gt;
&lt;li&gt;Tool-selection success&lt;/li&gt;
&lt;li&gt;Average steps per workflow&lt;/li&gt;
&lt;li&gt;Average number of clarification requests&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem.&lt;/p&gt;

&lt;p&gt;The trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.5. Quality Metrics
&lt;/h3&gt;

&lt;p&gt;This is where AI observability starts to overlap with AI evaluation.&lt;/p&gt;

&lt;p&gt;Examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Answer correctness&lt;/li&gt;
&lt;li&gt;Groundedness&lt;/li&gt;
&lt;li&gt;Retrieval relevance&lt;/li&gt;
&lt;li&gt;Tool-selection accuracy&lt;/li&gt;
&lt;li&gt;Task completion&lt;/li&gt;
&lt;li&gt;Policy adherence&lt;/li&gt;
&lt;li&gt;Human acceptance rate&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These metrics cannot always be derived from execution telemetry alone.&lt;/p&gt;

&lt;p&gt;We need an evaluation strategy to measure the actual quality of the result.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Cost Control: Understanding the Real Cost of an Agent
&lt;/h2&gt;

&lt;p&gt;One of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow.&lt;/p&gt;

&lt;p&gt;An agent-based application may consume resources across several layers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;LLM inference&lt;/li&gt;
&lt;li&gt;Embedding generation&lt;/li&gt;
&lt;li&gt;Reranking&lt;/li&gt;
&lt;li&gt;External API calls&lt;/li&gt;
&lt;li&gt;Database queries&lt;/li&gt;
&lt;li&gt;Vector database operations&lt;/li&gt;
&lt;li&gt;Container or sandbox execution&lt;/li&gt;
&lt;li&gt;Remote agent invocations&lt;/li&gt;
&lt;li&gt;Infrastructure and networking&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The total cost should reflect the entire execution path.&lt;/p&gt;

&lt;p&gt;A useful starting point is to define a model cost formula:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
C_{\text{model}} =&lt;br&gt;
\sum_{i=1}^{n}&lt;br&gt;
\left(&lt;br&gt;
T_{\text{input},i}P_{\text{input},i}&lt;br&gt;
+&lt;br&gt;
T_{\text{output},i}P_{\text{output},i}&lt;br&gt;
\right)&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;(T) represents token usage.&lt;/li&gt;
&lt;li&gt;(P) represents the applicable price per token.&lt;/li&gt;
&lt;li&gt;(i) represents an individual model operation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities.&lt;/p&gt;

&lt;p&gt;We should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets.&lt;/p&gt;

&lt;p&gt;The complete workflow cost can then be represented as:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
C_{\text{workflow}} =&lt;br&gt;
C_{\text{model}}&lt;br&gt;
+&lt;br&gt;
C_{\text{tools}}&lt;br&gt;
+&lt;br&gt;
C_{\text{compute}}&lt;br&gt;
+&lt;br&gt;
C_{\text{storage}}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;This may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available.&lt;/p&gt;
&lt;h3&gt;
  
  
  Why is cost per successful task more useful?
&lt;/h3&gt;

&lt;p&gt;Imagine two implementations:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Agent A&lt;/th&gt;
&lt;th&gt;Agent B&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Average cost per attempt&lt;/td&gt;
&lt;td&gt;$0.04&lt;/td&gt;
&lt;td&gt;$0.07&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Successful completion rate&lt;/td&gt;
&lt;td&gt;90%&lt;/td&gt;
&lt;td&gt;98%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost per successful task&lt;/td&gt;
&lt;td&gt;~$0.044&lt;/td&gt;
&lt;td&gt;~$0.071&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Agent A is cheaper per attempt, but Agent B completes more tasks successfully.&lt;/p&gt;

&lt;p&gt;Neither is automatically better.&lt;/p&gt;

&lt;p&gt;The correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts.&lt;/p&gt;

&lt;p&gt;A useful metric is:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
\text{Cost per successful task} =&lt;br&gt;
\frac{\text{Total execution cost}}&lt;br&gt;
{\text{Number of successful tasks}}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;The numerator should include failed attempts that consumed resources, rather than counting only successful runs.&lt;/p&gt;
&lt;h3&gt;
  
  
  Model selection and routing
&lt;/h3&gt;

&lt;p&gt;Observability data can also help us make better model-routing decisions.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Use a smaller model for classification or simple extraction.&lt;/li&gt;
&lt;li&gt;Use a more capable model for complex analysis.&lt;/li&gt;
&lt;li&gt;Avoid unnecessary model calls when a deterministic operation is sufficient.&lt;/li&gt;
&lt;li&gt;Reuse cached context when appropriate.&lt;/li&gt;
&lt;li&gt;Set explicit iteration and tool-call limits.&lt;/li&gt;
&lt;li&gt;Stop an agent when a task exceeds its budget.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request.&lt;/p&gt;
&lt;h3&gt;
  
  
  Token metrics in OpenTelemetry
&lt;/h3&gt;

&lt;p&gt;The current GenAI token metric conventions include proposed counters such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;gen_ai.client.inference.usage.input_tokens&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;gen_ai.client.inference.usage.output_tokens&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;gen_ai.client.inference.usage.cache_read.input_tokens&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;gen_ai.client.inference.usage.reasoning.output_tokens&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These conventions are still in Development.&lt;/p&gt;

&lt;p&gt;There is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics.&lt;/p&gt;

&lt;p&gt;A span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations.&lt;/p&gt;

&lt;p&gt;For production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices.&lt;/p&gt;

&lt;p&gt;This makes historical cost calculations more reliable when provider pricing changes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-token-metrics.md" rel="noopener noreferrer"&gt;GenAI Token Metrics&lt;/a&gt; · &lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md" rel="noopener noreferrer"&gt;GenAI Metrics&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  9. Observability Is Not Evaluation
&lt;/h2&gt;

&lt;p&gt;This distinction is worth emphasizing.&lt;/p&gt;

&lt;p&gt;Observability tells us what happened during execution.&lt;/p&gt;

&lt;p&gt;Evaluation tells us whether the result was good enough.&lt;/p&gt;

&lt;p&gt;A trace can show that an agent:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Called the correct model.&lt;/li&gt;
&lt;li&gt;Used three tools.&lt;/li&gt;
&lt;li&gt;Retrieved five documents.&lt;/li&gt;
&lt;li&gt;Completed within four seconds.&lt;/li&gt;
&lt;li&gt;Returned a response without an exception.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;But none of these facts proves that the answer was correct.&lt;/p&gt;

&lt;p&gt;Consider a RAG application.&lt;/p&gt;

&lt;p&gt;The retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information.&lt;/p&gt;

&lt;p&gt;From an operational perspective, the request succeeded.&lt;/p&gt;

&lt;p&gt;From a quality perspective, it failed.&lt;/p&gt;

&lt;p&gt;We need both observability and evaluation.&lt;/p&gt;
&lt;h3&gt;
  
  
  Offline Evaluation
&lt;/h3&gt;

&lt;p&gt;Offline evaluation measures an AI application against a predefined dataset.&lt;/p&gt;

&lt;p&gt;For example, we can create 100 representative questions and expected outcomes.&lt;/p&gt;

&lt;p&gt;Whenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again.&lt;/p&gt;

&lt;p&gt;This helps identify regressions before deploying changes.&lt;/p&gt;

&lt;p&gt;Useful evaluation dimensions include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Answer correctness&lt;/li&gt;
&lt;li&gt;Relevance&lt;/li&gt;
&lt;li&gt;Groundedness&lt;/li&gt;
&lt;li&gt;Tool selection&lt;/li&gt;
&lt;li&gt;Expected task completion&lt;/li&gt;
&lt;li&gt;Policy compliance&lt;/li&gt;
&lt;/ul&gt;
&lt;h3&gt;
  
  
  Online Evaluation
&lt;/h3&gt;

&lt;p&gt;Online evaluation measures actual production interactions.&lt;/p&gt;

&lt;p&gt;We can select a sample of production traces and evaluate them using:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Rule-based checks&lt;/li&gt;
&lt;li&gt;Code-based evaluators&lt;/li&gt;
&lt;li&gt;LLM-as-a-Judge&lt;/li&gt;
&lt;li&gt;Human feedback&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important part is to turn real production failures into future test cases.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A production trace shows that an agent selected the wrong tool.&lt;/li&gt;
&lt;li&gt;We identify the relevant input and expected behavior.&lt;/li&gt;
&lt;li&gt;We create a regression dataset entry.&lt;/li&gt;
&lt;li&gt;We update the prompt or agent implementation.&lt;/li&gt;
&lt;li&gt;We rerun offline evaluations.&lt;/li&gt;
&lt;li&gt;We deploy and monitor the next set of production traces.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This creates a continuous improvement loop.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Production Traces
       |
       v
Failure Analysis
       |
       v
Evaluation Dataset
       |
       v
Offline Evaluation
       |
       v
Prompt / Model / Agent Changes
       |
       v
Deployment
       |
       v
Online Evaluation
       |
       +------&amp;gt; New Regression Cases
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Tools worth evaluating
&lt;/h3&gt;

&lt;p&gt;Two examples of AI-focused observability and evaluation platforms are:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Langfuse&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Provides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements.&lt;/p&gt;

&lt;p&gt;Documentation: &lt;a href="https://langfuse.com/docs/evaluation/overview" rel="noopener noreferrer"&gt;https://langfuse.com/docs/evaluation/overview&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Arize Phoenix&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;An open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions.&lt;/p&gt;

&lt;p&gt;It provides tracing, evaluation, datasets, experiments, and debugging capabilities.&lt;/p&gt;

&lt;p&gt;Documentation: &lt;a href="https://phoenix.arize.com/" rel="noopener noreferrer"&gt;https://phoenix.arize.com/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;These tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://langfuse.com/docs/evaluation/core-concepts" rel="noopener noreferrer"&gt;Langfuse Evaluation Concepts&lt;/a&gt; · &lt;a href="https://github.com/Arize-ai/phoenix" rel="noopener noreferrer"&gt;Arize Phoenix&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  10. The Role of OpenTelemetry Collector
&lt;/h2&gt;

&lt;p&gt;In a production architecture, applications should not necessarily export telemetry directly to every observability backend.&lt;/p&gt;

&lt;p&gt;This is where the OpenTelemetry Collector becomes useful.&lt;/p&gt;

&lt;p&gt;The Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations.&lt;/p&gt;

&lt;p&gt;It provides a central place for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Batching telemetry&lt;/li&gt;
&lt;li&gt;Filtering unwanted data&lt;/li&gt;
&lt;li&gt;Applying processors&lt;/li&gt;
&lt;li&gt;Handling authentication&lt;/li&gt;
&lt;li&gt;Reducing telemetry volume&lt;/li&gt;
&lt;li&gt;Sampling traces&lt;/li&gt;
&lt;li&gt;Routing signals to different backends&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A simplified architecture looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 AI Applications
                       |
            +----------+----------+
            |          |          |
            v          v          v
         Next.js     .NET API   Python Agent
            |          |          |
            +----------+----------+
                       |
                      OTLP
                       |
                       v
             OpenTelemetry Collector
                       |
              +--------+--------+
              |        |        |
              v        v        v
            Traces   Metrics   Logs
              |        |        |
              v        v        v
            Jaeger   Prometheus  Log Backend
              |
              v
            Grafana
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same telemetry can also be routed to AI-specific observability platforms where supported.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why centralize collection?
&lt;/h3&gt;

&lt;p&gt;In enterprise environments, we may have several applications and teams using different frameworks and languages.&lt;/p&gt;

&lt;p&gt;A Collector provides a common ingestion point and makes it possible to apply centralized policies.&lt;/p&gt;

&lt;p&gt;For example, a Collector can filter sensitive attributes before telemetry reaches the final backend.&lt;/p&gt;

&lt;p&gt;It also allows us to change exporters or observability providers without changing the instrumentation in every application.&lt;/p&gt;

&lt;h3&gt;
  
  
  Agent and gateway deployment
&lt;/h3&gt;

&lt;p&gt;OpenTelemetry supports different Collector deployment patterns.&lt;/p&gt;

&lt;p&gt;A common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors.&lt;/p&gt;

&lt;p&gt;This provides a separation between local telemetry collection and centralized processing.&lt;/p&gt;

&lt;p&gt;However, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://opentelemetry.io/docs/collector/" rel="noopener noreferrer"&gt;OpenTelemetry Collector&lt;/a&gt; · &lt;a href="https://opentelemetry.io/docs/collector/deploy/gateway/" rel="noopener noreferrer"&gt;Gateway Deployment Pattern&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Sampling: How Do We Control Telemetry Volume?
&lt;/h2&gt;

&lt;p&gt;Agent-based workflows can generate a significant amount of telemetry.&lt;/p&gt;

&lt;p&gt;Consider an agent that performs 15 model calls and 20 tool calls during a single task.&lt;/p&gt;

&lt;p&gt;If the application handles thousands of tasks per hour, storing every span indefinitely may become expensive.&lt;/p&gt;

&lt;p&gt;Sampling helps control this volume.&lt;/p&gt;

&lt;p&gt;There are two common approaches.&lt;/p&gt;

&lt;h3&gt;
  
  
  Head-Based Sampling
&lt;/h3&gt;

&lt;p&gt;The sampling decision is made when the trace starts.&lt;/p&gt;

&lt;p&gt;For example, we may decide to sample 10% of incoming requests.&lt;/p&gt;

&lt;p&gt;This is relatively simple and reduces the amount of telemetry generated or exported.&lt;/p&gt;

&lt;p&gt;However, we do not yet know whether a trace will eventually fail or become unusually slow.&lt;/p&gt;

&lt;h3&gt;
  
  
  Tail-Based Sampling
&lt;/h3&gt;

&lt;p&gt;The sampling decision is made after considering the trace's collected spans.&lt;/p&gt;

&lt;p&gt;This makes it possible to retain:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;All failed traces.&lt;/li&gt;
&lt;li&gt;All traces exceeding a latency threshold.&lt;/li&gt;
&lt;li&gt;Traces containing specific tool failures.&lt;/li&gt;
&lt;li&gt;A percentage of successful traces.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed.&lt;/p&gt;

&lt;p&gt;However, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance.&lt;/p&gt;

&lt;p&gt;This generally requires trace-ID-aware routing when using multiple Collector instances.&lt;/p&gt;

&lt;p&gt;A practical approach is to combine sampling strategies:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Preserve error and high-latency traces.&lt;/li&gt;
&lt;li&gt;Retain a representative sample of successful traces.&lt;/li&gt;
&lt;li&gt;Maintain complete aggregate metrics where possible.&lt;/li&gt;
&lt;li&gt;Apply stricter retention policies to verbose telemetry.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://opentelemetry.io/docs/concepts/sampling/" rel="noopener noreferrer"&gt;OpenTelemetry Sampling&lt;/a&gt; · &lt;a href="https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor" rel="noopener noreferrer"&gt;Tail Sampling Processor&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  12. Security and Privacy Considerations
&lt;/h2&gt;

&lt;p&gt;This is one of the most important aspects of AI observability in enterprise environments.&lt;/p&gt;

&lt;p&gt;Traditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names.&lt;/p&gt;

&lt;p&gt;AI telemetry can contain much more sensitive information.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;User prompts&lt;/li&gt;
&lt;li&gt;System instructions&lt;/li&gt;
&lt;li&gt;Generated responses&lt;/li&gt;
&lt;li&gt;Tool arguments&lt;/li&gt;
&lt;li&gt;Tool results&lt;/li&gt;
&lt;li&gt;Retrieved documents&lt;/li&gt;
&lt;li&gt;Conversation context&lt;/li&gt;
&lt;li&gt;Customer or business identifiers&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Sending all this information to an observability backend can create additional privacy and data governance risks.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do not capture prompts and responses by default
&lt;/h3&gt;

&lt;p&gt;A particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise.&lt;/p&gt;

&lt;p&gt;We can disable this behavior:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nf"&gt;registerTelemetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;OpenTelemetry&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;recordInputs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;recordOutputs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}),&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For many production applications, this is a better starting point.&lt;/p&gt;

&lt;p&gt;We can still collect useful operational telemetry:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Model identifiers&lt;/li&gt;
&lt;li&gt;Token counts&lt;/li&gt;
&lt;li&gt;Duration&lt;/li&gt;
&lt;li&gt;Finish reasons&lt;/li&gt;
&lt;li&gt;Tool execution times&lt;/li&gt;
&lt;li&gt;Error information&lt;/li&gt;
&lt;li&gt;Workflow identifiers&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;without exporting the complete conversation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Apply data minimization
&lt;/h3&gt;

&lt;p&gt;Before exporting telemetry, ask whether each attribute is genuinely needed.&lt;/p&gt;

&lt;p&gt;Avoid sending:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Access tokens&lt;/li&gt;
&lt;li&gt;Authorization headers&lt;/li&gt;
&lt;li&gt;Passwords&lt;/li&gt;
&lt;li&gt;Raw customer records&lt;/li&gt;
&lt;li&gt;Full database query results&lt;/li&gt;
&lt;li&gt;Unfiltered tool arguments&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For sensitive workflows, use approved metadata rather than full payloads.&lt;/p&gt;

&lt;h3&gt;
  
  
  Separate operational telemetry from application data
&lt;/h3&gt;

&lt;p&gt;An observability backend should not automatically become another storage system for business data.&lt;/p&gt;

&lt;p&gt;The fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely.&lt;/p&gt;

&lt;h3&gt;
  
  
  Define access and retention policies
&lt;/h3&gt;

&lt;p&gt;Enterprise environments should define:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Who can access traces.&lt;/li&gt;
&lt;li&gt;Which teams can view prompt and response content.&lt;/li&gt;
&lt;li&gt;How long telemetry is retained.&lt;/li&gt;
&lt;li&gt;How data is deleted.&lt;/li&gt;
&lt;li&gt;Which environments are allowed to capture full content.&lt;/li&gt;
&lt;li&gt;How telemetry exports are authenticated and encrypted.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md" rel="noopener noreferrer"&gt;OpenTelemetry GenAI Content Capture&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  13. Limitations and Trade-offs
&lt;/h2&gt;

&lt;p&gt;OpenTelemetry provides a strong foundation, but it is not a complete AI observability solution by itself.&lt;/p&gt;

&lt;h3&gt;
  
  
  Instrumentation coverage varies
&lt;/h3&gt;

&lt;p&gt;Not every framework, provider, or agent runtime exposes the same information.&lt;/p&gt;

&lt;p&gt;Some integrations can automatically trace model calls and tool execution, while others require custom instrumentation.&lt;/p&gt;

&lt;p&gt;We should verify what is actually captured rather than assuming that installing an OpenTelemetry package provides complete visibility.&lt;/p&gt;

&lt;h3&gt;
  
  
  GenAI semantic conventions are evolving
&lt;/h3&gt;

&lt;p&gt;The GenAI conventions are still under development.&lt;/p&gt;

&lt;p&gt;Attribute names, operation models, and implementation support may evolve.&lt;/p&gt;

&lt;p&gt;Teams should pin versions, review changes, and avoid coupling application business logic to conventions that have not reached the required maturity.&lt;/p&gt;

&lt;h3&gt;
  
  
  Observability does not guarantee answer quality
&lt;/h3&gt;

&lt;p&gt;A trace can explain what happened but does not prove that the model's result is correct.&lt;/p&gt;

&lt;p&gt;Evaluation datasets, deterministic checks, and human feedback remain necessary.&lt;/p&gt;

&lt;h3&gt;
  
  
  Telemetry has its own cost
&lt;/h3&gt;

&lt;p&gt;Collecting every input, output, tool argument, and execution event can generate significant storage and processing overhead.&lt;/p&gt;

&lt;p&gt;We should measure the cost of observability itself.&lt;/p&gt;

&lt;h3&gt;
  
  
  High-cardinality dimensions can become expensive
&lt;/h3&gt;

&lt;p&gt;Adding unique user IDs, conversation IDs, trace IDs, or request IDs to metric labels can create an excessive number of time series.&lt;/p&gt;

&lt;p&gt;Use traces for request-level correlation and keep metric dimensions bounded.&lt;/p&gt;

&lt;h3&gt;
  
  
  Distributed tracing requires correct context propagation
&lt;/h3&gt;

&lt;p&gt;A2A communication, queues, background tasks, and remote tool execution may cross process boundaries.&lt;/p&gt;

&lt;p&gt;If context is lost, we can end up with disconnected traces.&lt;/p&gt;

&lt;h3&gt;
  
  
  AI evaluation requires careful design
&lt;/h3&gt;

&lt;p&gt;LLM-as-a-Judge can provide useful signals, but it can also be inconsistent or biased.&lt;/p&gt;

&lt;p&gt;A single evaluation score should not become the only definition of quality.&lt;/p&gt;

&lt;p&gt;Combine deterministic checks, representative datasets, human review, and calibrated evaluators.&lt;/p&gt;

&lt;h2&gt;
  
  
  14. What Can an AI Chapter Lead Contribute?
&lt;/h2&gt;

&lt;p&gt;I think this is where observability moves from an implementation detail to an engineering capability.&lt;/p&gt;

&lt;p&gt;When multiple teams develop AI-powered applications, each team may implement different logging, tracing, and evaluation mechanisms.&lt;/p&gt;

&lt;p&gt;Over time, this creates fragmented visibility.&lt;/p&gt;

&lt;p&gt;One team measures token usage, another measures response latency, and another stores tool-call logs without a common correlation model.&lt;/p&gt;

&lt;p&gt;As an AI Chapter Lead, I would focus on establishing a shared observability standard that can be reused across teams and applications.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Define a Common Telemetry Standard
&lt;/h3&gt;

&lt;p&gt;Establish a baseline for the telemetry every agent-based application should produce.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Category&lt;/th&gt;
&lt;th&gt;Required information&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Application&lt;/td&gt;
&lt;td&gt;Service name, environment, version&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Workflow&lt;/td&gt;
&lt;td&gt;Workflow name, workflow version&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent&lt;/td&gt;
&lt;td&gt;Agent name, execution status&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Model&lt;/td&gt;
&lt;td&gt;Provider, requested model, response model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Usage&lt;/td&gt;
&lt;td&gt;Input/output tokens, cache usage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tools&lt;/td&gt;
&lt;td&gt;Tool name, duration, execution status&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Execution&lt;/td&gt;
&lt;td&gt;Trace ID, span ID, error information&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Business outcome&lt;/td&gt;
&lt;td&gt;Completed, failed, canceled, escalated&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The objective is to make observability consistent without forcing every team to use the same agent framework.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Build a Shared Instrumentation Package
&lt;/h3&gt;

&lt;p&gt;We can create an internal TypeScript package that wraps the common OpenTelemetry configuration.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;@company/ai-observability
|
+-- OpenTelemetry initialization
+-- AI SDK integration
+-- Agent span utilities
+-- Workflow instrumentation
+-- Tool execution tracking
+-- Cost attribution
+-- Evaluation metadata
+-- Privacy and redaction policies
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This package can provide standard helpers for application-level tracing and reduce duplicated implementation.&lt;/p&gt;

&lt;p&gt;For teams using .NET, Python, or Go, equivalent instrumentation conventions can be established without requiring identical SDK implementations.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Standardize Agent and Workflow Metrics
&lt;/h3&gt;

&lt;p&gt;We should define a common set of metrics that helps teams understand both technical performance and business outcomes.&lt;/p&gt;

&lt;p&gt;Examples:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agent success rate&lt;/li&gt;
&lt;li&gt;Workflow completion rate&lt;/li&gt;
&lt;li&gt;Tool-call failure rate&lt;/li&gt;
&lt;li&gt;Model latency&lt;/li&gt;
&lt;li&gt;Cost per successful task&lt;/li&gt;
&lt;li&gt;Average token usage&lt;/li&gt;
&lt;li&gt;Human intervention rate&lt;/li&gt;
&lt;li&gt;Evaluation quality scores&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A shared dashboard can make it easier to compare systems and identify performance regressions.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Establish an Internal Evaluation Framework
&lt;/h3&gt;

&lt;p&gt;Observability without evaluation gives us only part of the picture.&lt;/p&gt;

&lt;p&gt;We can create reusable evaluation datasets and testing patterns for common agent behaviors.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tool selection accuracy&lt;/li&gt;
&lt;li&gt;Missing-source detection&lt;/li&gt;
&lt;li&gt;Correct handling of incomplete input&lt;/li&gt;
&lt;li&gt;Permission violations&lt;/li&gt;
&lt;li&gt;Unsupported claims&lt;/li&gt;
&lt;li&gt;Failure recovery&lt;/li&gt;
&lt;li&gt;Task completion quality&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is particularly valuable because agent behavior can change when prompts, tools, models, or orchestration logic are modified.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Establish Privacy and Data Governance
&lt;/h3&gt;

&lt;p&gt;Define what can and cannot be captured by default.&lt;/p&gt;

&lt;p&gt;The shared implementation should make safe defaults easy to use and prevent teams from accidentally exporting sensitive prompts, responses, or tool payloads.&lt;/p&gt;

&lt;p&gt;This should include clear ownership, access policies, retention, and audit requirements.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Run a Cross-Team Pilot
&lt;/h3&gt;

&lt;p&gt;A useful pilot would be a multi-agent workflow that involves several independently deployed services.&lt;/p&gt;

&lt;p&gt;For example, a merge request review involving coordinator, security, and testing agents.&lt;/p&gt;

&lt;p&gt;The pilot should demonstrate:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;One trace across the complete request.&lt;/li&gt;
&lt;li&gt;Parent-child relationships between agents.&lt;/li&gt;
&lt;li&gt;Model and tool execution spans.&lt;/li&gt;
&lt;li&gt;Token and cost attribution.&lt;/li&gt;
&lt;li&gt;Error and retry visibility.&lt;/li&gt;
&lt;li&gt;Evaluation results associated with the completed workflow.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This would provide a reference architecture that other teams can reuse.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Make Observability Part of the Development Lifecycle
&lt;/h3&gt;

&lt;p&gt;Observability should not begin after production deployment.&lt;/p&gt;

&lt;p&gt;We can introduce telemetry and evaluation requirements into the development process itself.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;New agents must have execution tracing.&lt;/li&gt;
&lt;li&gt;New tools must expose execution status and duration.&lt;/li&gt;
&lt;li&gt;Prompt changes must pass relevant regression evaluations.&lt;/li&gt;
&lt;li&gt;New workflows must define success and failure criteria.&lt;/li&gt;
&lt;li&gt;Production deployments must have basic operational dashboards.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This makes observability a standard engineering practice rather than a separate monitoring task.&lt;/p&gt;

&lt;h2&gt;
  
  
  15. A Practical Adoption Roadmap
&lt;/h2&gt;

&lt;p&gt;I would avoid introducing every observability component at once.&lt;/p&gt;

&lt;p&gt;A phased approach is more useful.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Phase&lt;/th&gt;
&lt;th&gt;Focus&lt;/th&gt;
&lt;th&gt;Expected result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;OpenTelemetry initialization&lt;/td&gt;
&lt;td&gt;Basic traces and metrics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;AI SDK instrumentation&lt;/td&gt;
&lt;td&gt;Model and tool visibility&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Custom agent spans&lt;/td&gt;
&lt;td&gt;Complete workflow tracing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Cost attribution&lt;/td&gt;
&lt;td&gt;Usage and cost per task&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Evaluation&lt;/td&gt;
&lt;td&gt;Offline and online quality checks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Security and governance&lt;/td&gt;
&lt;td&gt;Privacy and access controls&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;Shared dashboards&lt;/td&gt;
&lt;td&gt;Cross-team operational visibility&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The first milestone should be simple:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;For every important agent workflow, we should be able to answer what happened, how long it took, how much it cost, and whether it achieved the intended result.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Once this foundation is established, more advanced capabilities such as anomaly detection, adaptive sampling, automated evaluation, and cost-aware model routing become much easier to introduce.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;AI agents introduce a new level of complexity to application observability.&lt;/p&gt;

&lt;p&gt;A single user request can trigger multiple model calls, tool executions, retrieval operations, and remote agent interactions. Monitoring only the HTTP request or final model response is not enough to understand the complete execution.&lt;/p&gt;

&lt;p&gt;OpenTelemetry provides a vendor-neutral foundation for collecting traces, metrics, and logs across these components.&lt;/p&gt;

&lt;p&gt;Combined with GenAI semantic conventions, AI SDK instrumentation, distributed tracing, and dedicated evaluation systems, it gives engineering teams a much better understanding of how agent-based applications behave in production.&lt;/p&gt;

&lt;p&gt;But the most important distinction remains:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Observability tells us what happened. Evaluation tells us whether it was good. Cost attribution tells us whether it was worth it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For an AI Chapter Lead, the opportunity is to make these capabilities part of the engineering standard: shared instrumentation, consistent metrics, measurable quality, and secure telemetry practices.&lt;/p&gt;

&lt;p&gt;The goal should not be to collect more telemetry.&lt;/p&gt;

&lt;p&gt;It should be to make every production agent workflow understandable, measurable, and continuously improvable.&lt;/p&gt;




&lt;h2&gt;
  
  
  References and Further Reading
&lt;/h2&gt;

&lt;h3&gt;
  
  
  OpenTelemetry
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/docs/" rel="noopener noreferrer"&gt;OpenTelemetry Official Documentation&lt;/a&gt; — Core concepts, signals, instrumentation, and architecture.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/docs/languages/js/" rel="noopener noreferrer"&gt;OpenTelemetry JavaScript&lt;/a&gt; — TypeScript and Node.js instrumentation.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/docs/concepts/signals/traces/" rel="noopener noreferrer"&gt;OpenTelemetry Tracing&lt;/a&gt; — Traces, spans, and distributed execution.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/docs/concepts/context-propagation/" rel="noopener noreferrer"&gt;OpenTelemetry Context Propagation&lt;/a&gt; — Distributed trace context.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/docs/collector/" rel="noopener noreferrer"&gt;OpenTelemetry Collector&lt;/a&gt; — Receiving, processing, and exporting telemetry.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/docs/collector/deploy/gateway/" rel="noopener noreferrer"&gt;OpenTelemetry Gateway Deployment&lt;/a&gt; — Centralized telemetry collection.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://opentelemetry.io/docs/concepts/sampling/" rel="noopener noreferrer"&gt;OpenTelemetry Sampling&lt;/a&gt; — Head-based and tail-based sampling.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  GenAI Semantic Conventions
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md" rel="noopener noreferrer"&gt;GenAI Semantic Conventions — Spans&lt;/a&gt; — Model operations, retrieval, and tool execution.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md" rel="noopener noreferrer"&gt;GenAI Agent Spans&lt;/a&gt; — Agent invocation, workflows, and tool spans.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md" rel="noopener noreferrer"&gt;GenAI Metrics&lt;/a&gt; — Agent duration, tool calls, and workflow metrics.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-token-metrics.md" rel="noopener noreferrer"&gt;GenAI Token Metrics&lt;/a&gt; — Token consumption and usage measurements.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Vercel AI SDK
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://ai-sdk.dev/docs/ai-sdk-core/telemetry" rel="noopener noreferrer"&gt;AI SDK Telemetry&lt;/a&gt; — OpenTelemetry integration, span hierarchy, and custom attributes.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/vercel/ai/blob/main/skills/migrate-ai-sdk-v6-to-v7/SKILL.md" rel="noopener noreferrer"&gt;AI SDK v7 Migration Guide&lt;/a&gt; — Version-specific telemetry changes.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://nextjs.org/docs/app/guides/open-telemetry" rel="noopener noreferrer"&gt;Next.js OpenTelemetry&lt;/a&gt; — Application-level instrumentation.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  AI Evaluation and Observability Platforms
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://langfuse.com/docs/evaluation/overview" rel="noopener noreferrer"&gt;Langfuse Evaluation&lt;/a&gt; — Online and offline evaluation.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://langfuse.com/docs/evaluation/core-concepts" rel="noopener noreferrer"&gt;Langfuse Core Concepts&lt;/a&gt; — Datasets, experiments, and evaluators.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/Arize-ai/phoenix" rel="noopener noreferrer"&gt;Arize Phoenix&lt;/a&gt; — Open-source AI observability and evaluation.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/Arize-ai/openinference" rel="noopener noreferrer"&gt;OpenInference&lt;/a&gt; — OpenTelemetry-based instrumentation conventions for AI applications.&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>productivity</category>
      <category>programming</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Hedged Requests in Go: Reducing Tail Latency Without Overloading Dependencies</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Mon, 02 Nov 2026 17:28:15 +0000</pubDate>
      <link>https://dev.to/serifcolakel/hedged-requests-in-go-reducing-tail-latency-without-overloading-dependencies-3dl7</link>
      <guid>https://dev.to/serifcolakel/hedged-requests-in-go-reducing-tail-latency-without-overloading-dependencies-3dl7</guid>
      <description>&lt;p&gt;Most requests in a distributed system finish quickly.&lt;/p&gt;

&lt;p&gt;A few don't.&lt;/p&gt;

&lt;p&gt;Your API might have a median response time of 40ms, while its p99 latency reaches 900ms. Most users get a fast response, but a small percentage experience significantly longer waits.&lt;/p&gt;

&lt;p&gt;These slow requests are particularly problematic in systems that depend on multiple services.&lt;/p&gt;

&lt;p&gt;A single slow dependency can delay an entire operation, even when every other component is responding normally.&lt;/p&gt;

&lt;p&gt;One way to address this problem is &lt;strong&gt;request hedging&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Instead of waiting indefinitely for an unusually slow request, the client starts another copy after a configured delay. Whichever request succeeds first wins, and the remaining requests are canceled.&lt;/p&gt;

&lt;p&gt;This technique can significantly reduce tail latency, but it comes with an important trade-off: it generates additional traffic.&lt;/p&gt;

&lt;p&gt;If every slow request triggers another request without limits, the mechanism designed to improve latency can overload the dependency it's trying to protect.&lt;/p&gt;

&lt;p&gt;In this article, we'll explore:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;What tail latency is and why averages hide it&lt;/li&gt;
&lt;li&gt;How hedged requests work&lt;/li&gt;
&lt;li&gt;The difference between hedging and retries&lt;/li&gt;
&lt;li&gt;How to implement hedged requests in Go&lt;/li&gt;
&lt;li&gt;How to cancel losing requests safely&lt;/li&gt;
&lt;li&gt;How to integrate hedging with retry budgets and adaptive concurrency&lt;/li&gt;
&lt;li&gt;When hedging helps and when it makes things worse&lt;/li&gt;
&lt;li&gt;What to monitor and test in production&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  1. The Problem: Tail Latency
&lt;/h2&gt;

&lt;p&gt;Let's say an API receives 10,000 requests.&lt;/p&gt;

&lt;p&gt;Its response-time distribution looks like this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Percentile&lt;/th&gt;
&lt;th&gt;Response time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p50&lt;/td&gt;
&lt;td&gt;40ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p90&lt;/td&gt;
&lt;td&gt;75ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p95&lt;/td&gt;
&lt;td&gt;120ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99&lt;/td&gt;
&lt;td&gt;900ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99.9&lt;/td&gt;
&lt;td&gt;2.5s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Looking only at the average or median would suggest that the service is performing well.&lt;/p&gt;

&lt;p&gt;But 1% of 10,000 requests means 100 requests take at least 900ms.&lt;/p&gt;

&lt;p&gt;For latency-sensitive applications, these outliers matter.&lt;/p&gt;

&lt;p&gt;Consider an API that aggregates data from five independent services:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 API Request
                      |
          +-----------+-----------+
          |           |           |
       Service A  Service B   Service C
          |           |           |
       Service D  Service E       |
          |           |           |
          +-----------+-----------+
                      |
                 Final Response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the API needs all five responses, its total latency is constrained by the slowest dependency.&lt;/p&gt;

&lt;p&gt;Even if four services respond in 30ms, one service taking 800ms can delay the entire operation.&lt;/p&gt;

&lt;p&gt;This is one reason tail latency becomes increasingly important as distributed systems grow.&lt;/p&gt;

&lt;p&gt;Jeff Dean and Luiz André Barroso discuss this problem in their influential paper, &lt;em&gt;The Tail at Scale&lt;/em&gt;. They describe techniques for building responsive services despite unpredictable latency in individual components.&lt;/p&gt;

&lt;p&gt;Reference: &lt;a href="https://research.google/pubs/the-tail-at-scale/" rel="noopener noreferrer"&gt;https://research.google/pubs/the-tail-at-scale/&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. What Are Hedged Requests?
&lt;/h2&gt;

&lt;p&gt;A hedged request is a duplicate request sent while the original request is still in progress.&lt;/p&gt;

&lt;p&gt;The client does not immediately send multiple copies.&lt;/p&gt;

&lt;p&gt;Instead, it waits for a configurable period.&lt;/p&gt;

&lt;p&gt;If the original request hasn't completed, the client sends another request, ideally to a different replica.&lt;/p&gt;

&lt;p&gt;The first successful response wins.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Time
 |
 |  Primary request
 |  ────────────────────────────────&amp;gt;
 |
 |  Wait: 100ms
 |
 |  Hedge request
 |  ─────────────────────&amp;gt;
 |
 |  Hedge response: 160ms
 |  Primary response: 450ms
 |
 |  Return hedge response
 |  Cancel primary request
 |
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important detail is that the second request starts before the first one fails.&lt;/p&gt;

&lt;p&gt;This is fundamentally different from a traditional retry.&lt;/p&gt;

&lt;p&gt;A retry waits for an unsuccessful attempt before trying again.&lt;/p&gt;

&lt;p&gt;A hedge assumes that an unusually slow request might be an outlier and starts another attempt proactively.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does this work?
&lt;/h3&gt;

&lt;p&gt;Imagine two replicas:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Replica A is experiencing temporary contention.&lt;/li&gt;
&lt;li&gt;Replica B is healthy.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The primary request goes to A.&lt;/p&gt;

&lt;p&gt;After 100ms, it still hasn't completed, so the client sends a hedge to B.&lt;/p&gt;

&lt;p&gt;If B responds in 60ms, the client returns that response instead of waiting for A.&lt;/p&gt;

&lt;p&gt;The total latency becomes approximately 160ms instead of 450ms.&lt;/p&gt;

&lt;p&gt;The original request might still finish later, but its result is no longer needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Hedging vs. Retrying
&lt;/h2&gt;

&lt;p&gt;These mechanisms are related, but they solve slightly different problems.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mechanism&lt;/th&gt;
&lt;th&gt;When the next attempt starts&lt;/th&gt;
&lt;th&gt;Primary goal&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Retry&lt;/td&gt;
&lt;td&gt;After an attempt fails&lt;/td&gt;
&lt;td&gt;Recover from transient failures&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hedging&lt;/td&gt;
&lt;td&gt;While an attempt is still running&lt;/td&gt;
&lt;td&gt;Reduce tail latency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Timeout&lt;/td&gt;
&lt;td&gt;After a deadline expires&lt;/td&gt;
&lt;td&gt;Bound waiting time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Circuit breaker&lt;/td&gt;
&lt;td&gt;When a dependency is unhealthy&lt;/td&gt;
&lt;td&gt;Stop sending harmful traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A retry responds to failure.&lt;/p&gt;

&lt;p&gt;A hedge responds to excessive waiting.&lt;/p&gt;

&lt;p&gt;This distinction matters because hedging can increase downstream load even when the original request would eventually succeed.&lt;/p&gt;

&lt;p&gt;For that reason, hedging should generally be limited to safe, idempotent operations, especially reads.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Choosing the Hedging Delay
&lt;/h2&gt;

&lt;p&gt;The hedging delay is one of the most important configuration decisions.&lt;/p&gt;

&lt;p&gt;If the delay is too short, the client duplicates too many requests.&lt;/p&gt;

&lt;p&gt;If it is too long, the hedge provides little latency improvement.&lt;/p&gt;

&lt;p&gt;A reasonable starting point is to use a measured latency percentile for a particular operation.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Percentile&lt;/th&gt;
&lt;th&gt;Latency&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p50&lt;/td&gt;
&lt;td&gt;40ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p90&lt;/td&gt;
&lt;td&gt;70ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p95&lt;/td&gt;
&lt;td&gt;100ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99&lt;/td&gt;
&lt;td&gt;450ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A 100ms hedging delay means that requests exceeding the p95 threshold become candidates for hedging.&lt;/p&gt;

&lt;p&gt;Under stable conditions, approximately 5% of requests may reach that threshold.&lt;/p&gt;

&lt;p&gt;This does not mean exactly 5% of requests will generate a hedge: requests can complete at the threshold boundary, and actual behavior depends on the latency distribution and scheduling.&lt;/p&gt;

&lt;p&gt;The important principle is to trigger hedging in the tail, not in the middle of normal traffic.&lt;/p&gt;

&lt;p&gt;The original &lt;em&gt;Tail at Scale&lt;/em&gt; paper discusses using a high-percentile latency threshold to limit duplicate traffic while reducing extreme latency. Its examples demonstrate why carefully selected hedging delays can achieve substantial tail-latency improvements with relatively modest additional load.&lt;/p&gt;

&lt;p&gt;Source: &lt;a href="https://www.andrew.cmu.edu/user/gaurij/18-847F-Readings/tail_at_scale.pdf" rel="noopener noreferrer"&gt;https://www.andrew.cmu.edu/user/gaurij/18-847F-Readings/tail_at_scale.pdf&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Implementing Hedged Requests in Go
&lt;/h2&gt;

&lt;p&gt;Let's build a small hedging executor.&lt;/p&gt;

&lt;p&gt;Our implementation will support:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An immediate primary request&lt;/li&gt;
&lt;li&gt;A configurable hedging delay&lt;/li&gt;
&lt;li&gt;A maximum number of attempts&lt;/li&gt;
&lt;li&gt;Context cancellation&lt;/li&gt;
&lt;li&gt;First-successful-response semantics&lt;/li&gt;
&lt;li&gt;Retryable error classification&lt;/li&gt;
&lt;li&gt;An optional shared retry budget&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We'll use Go generics so the executor can return any response type.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Define the policy
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;hedge&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"context"&lt;/span&gt;
    &lt;span class="s"&gt;"errors"&lt;/span&gt;
    &lt;span class="s"&gt;"fmt"&lt;/span&gt;
    &lt;span class="s"&gt;"time"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ErrInvalidPolicy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"invalid hedging policy"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;ErrBudgetDenied&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"hedging budget exhausted"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Budget&lt;/span&gt; &lt;span class="k"&gt;interface&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;TryTake&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Policy&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Delay&lt;/span&gt;       &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;
    &lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;

    &lt;span class="n"&gt;Budget&lt;/span&gt;      &lt;span class="n"&gt;Budget&lt;/span&gt;
    &lt;span class="n"&gt;ShouldHedge&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt; &lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="n"&gt;T&lt;/span&gt;
    &lt;span class="n"&gt;err&lt;/span&gt;   &lt;span class="kt"&gt;error&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;Budget&lt;/code&gt; interface allows us to reuse the retry budget implementation from the previous article.&lt;/p&gt;

&lt;p&gt;The original request does not consume a hedge token.&lt;/p&gt;

&lt;p&gt;Every additional request does.&lt;/p&gt;

&lt;p&gt;We also use &lt;code&gt;ShouldHedge&lt;/code&gt; to distinguish errors that should terminate the operation from errors that may allow another attempt.&lt;/p&gt;

&lt;p&gt;For example, a validation error should not trigger another request.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Implement the executor
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt; &lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;policy&lt;/span&gt; &lt;span class="n"&gt;Policy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;operation&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;zero&lt;/span&gt; &lt;span class="n"&gt;T&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Delay&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;operation&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ShouldHedge&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;zero&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ErrInvalidPolicy&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;zero&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;hedgeCtx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WithCancel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;launched&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
    &lt;span class="n"&gt;inFlight&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;

    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;lastErr&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;

    &lt;span class="n"&gt;startAttempt&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;launched&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;
        &lt;span class="n"&gt;inFlight&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;

        &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;launched&lt;/span&gt;

        &lt;span class="k"&gt;go&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;operation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hedgeCtx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;]{&lt;/span&gt;
                &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;   &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="c"&gt;// The primary request starts immediately.&lt;/span&gt;
    &lt;span class="n"&gt;startAttempt&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;timer&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Timer&lt;/span&gt;
    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;timerC&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Time&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;timer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewTimer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;timerC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;C&lt;/span&gt;
        &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;zero&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;inFlight&lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="c"&gt;// First successful response wins.&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="n"&gt;lastErr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ShouldHedge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;zero&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;inFlight&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
                &lt;span class="n"&gt;launched&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;zero&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lastErr&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;timerC&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;launched&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;timerC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Budget&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
                &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TryTake&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="c"&gt;// No more additional requests.&lt;/span&gt;
                &lt;span class="n"&gt;timerC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;

                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;inFlight&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;lastErr&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;zero&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lastErr&lt;/span&gt;
                    &lt;span class="p"&gt;}&lt;/span&gt;

                    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;zero&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ErrBudgetDenied&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;

                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="n"&gt;startAttempt&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;launched&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Reset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;timerC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Let's examine the important decisions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The primary request starts immediately.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;We don't wait for a timer before sending the first request.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Additional requests start only after the configured delay.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If the primary finishes successfully before the delay expires, the executor returns without sending a hedge.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The first successful response wins.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;As soon as a successful result arrives, the function returns and cancels the shared context.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Errors are classified.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A fatal error can terminate the operation immediately. A retryable error can be treated as non-final while another attempt is still running.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The number of attempts is bounded.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The implementation intentionally limits the example to three total attempts, including the original request.&lt;/p&gt;

&lt;p&gt;This is a conservative default for the example, not a universal limit.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Use the executor
&lt;/h3&gt;

&lt;p&gt;Imagine a service that reads market data from multiple replicas.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Price&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Symbol&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;
    &lt;span class="n"&gt;Value&lt;/span&gt;  &lt;span class="kt"&gt;float64&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;fetchPrice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Price&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;replica&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;selectReplica&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;replica&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"AAPL"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We can now use the hedging executor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;policy&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;hedge&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Policy&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;Delay&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;       &lt;span class="m"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;MaxAttempts&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Budget&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;      &lt;span class="n"&gt;retryBudget&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;ShouldHedge&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;isRetryableError&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;price&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;hedge&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;fetchPrice&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"%s: %.2f&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;price&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Symbol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;price&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The primary request goes to the first replica.&lt;/p&gt;

&lt;p&gt;If it hasn't completed after 100ms, the executor checks the budget and starts another request.&lt;/p&gt;

&lt;p&gt;If the second replica responds first, the result is returned and the first request is canceled.&lt;/p&gt;

&lt;p&gt;The example assumes that &lt;code&gt;selectReplica&lt;/code&gt; routes attempts to distinct, suitable replicas. Sending both copies to the same overloaded instance may provide little benefit.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Cancellation Is Not the Same as Stopping Work
&lt;/h2&gt;

&lt;p&gt;This is one of the most important details in a hedging implementation.&lt;/p&gt;

&lt;p&gt;When the winning request completes, the executor calls &lt;code&gt;cancel()&lt;/code&gt; on the shared context.&lt;/p&gt;

&lt;p&gt;The losing requests receive cancellation through their contexts.&lt;/p&gt;

&lt;p&gt;But cancellation is cooperative.&lt;/p&gt;

&lt;p&gt;It does not forcibly terminate a goroutine, an HTTP request, a database query, or server-side processing.&lt;/p&gt;

&lt;p&gt;For example, a dependency may continue executing a query even after the client disconnects.&lt;/p&gt;

&lt;p&gt;A correct operation should propagate its context:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewRequestWithContext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MethodGet&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s"&gt;"https://api.example.com/prices/AAPL"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="no"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Body&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The server should also respect request cancellation where possible.&lt;/p&gt;

&lt;p&gt;Otherwise, the client may stop waiting while the dependency continues doing unnecessary work.&lt;/p&gt;

&lt;p&gt;This is especially important for expensive database queries and CPU-intensive operations.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Why a Hedge Budget Matters
&lt;/h2&gt;

&lt;p&gt;Hedging creates additional traffic.&lt;/p&gt;

&lt;p&gt;Imagine a service receiving 10,000 requests per second.&lt;/p&gt;

&lt;p&gt;If 5% reach the hedging threshold, that could mean approximately 500 additional attempts per second.&lt;/p&gt;

&lt;p&gt;Now imagine the dependency is already degraded.&lt;/p&gt;

&lt;p&gt;More concurrent requests may increase queueing, which increases latency, which triggers more hedges.&lt;/p&gt;

&lt;p&gt;The mechanism can create the same kind of amplification problem that we discussed in the Retry Budgets article.&lt;/p&gt;

&lt;p&gt;This is why the hedge budget should be shared across requests.&lt;/p&gt;

&lt;p&gt;A simple rule:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The original request is always allowed, subject to normal admission controls.&lt;/li&gt;
&lt;li&gt;Each additional hedge requires a token.&lt;/li&gt;
&lt;li&gt;Successful normal traffic replenishes the budget.&lt;/li&gt;
&lt;li&gt;If the budget is exhausted, no new hedges are launched.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;budget&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;retry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewRetryBudget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="m"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="m"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact budget should be tuned according to the dependency's spare capacity and the service's traffic patterns.&lt;/p&gt;

&lt;p&gt;The important part is that hedging is not an unlimited permission to duplicate requests.&lt;/p&gt;

&lt;p&gt;It is an additional workload that must be controlled.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Hedging and Adaptive Concurrency
&lt;/h2&gt;

&lt;p&gt;Hedging and adaptive concurrency work particularly well together, but they need to share a consistent view of downstream capacity.&lt;/p&gt;

&lt;p&gt;Adaptive concurrency adjusts how many requests are allowed in flight based on observed dependency behavior.&lt;/p&gt;

&lt;p&gt;Hedging creates additional in-flight requests.&lt;/p&gt;

&lt;p&gt;If the adaptive limiter counts only original requests, hedged requests can bypass the intended concurrency ceiling.&lt;/p&gt;

&lt;p&gt;That defeats the purpose of the limiter.&lt;/p&gt;

&lt;p&gt;A safer integration is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The primary request acquires a concurrency permit.&lt;/li&gt;
&lt;li&gt;A hedge must acquire its own permit.&lt;/li&gt;
&lt;li&gt;If no permit is available, the hedge is skipped.&lt;/li&gt;
&lt;li&gt;The winning request returns its result.&lt;/li&gt;
&lt;li&gt;Every attempt releases its own permit when it actually completes.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is an important implementation detail here.&lt;/p&gt;

&lt;p&gt;Do not release the losing request's concurrency permit merely because the caller stopped waiting for it.&lt;/p&gt;

&lt;p&gt;The permit must remain held until the underlying operation actually completes or otherwise confirms that the protected work has stopped.&lt;/p&gt;

&lt;p&gt;Otherwise, the limiter undercounts active downstream work.&lt;/p&gt;

&lt;p&gt;This is also why a shared cancellation context alone is not enough to guarantee that hedged work has disappeared.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Hedging Requires Idempotency
&lt;/h2&gt;

&lt;p&gt;Sending the same operation more than once is not always safe.&lt;/p&gt;

&lt;p&gt;Consider these operations:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Hedging suitability&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Read a market price&lt;/td&gt;
&lt;td&gt;Generally suitable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Read a portfolio&lt;/td&gt;
&lt;td&gt;Generally suitable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fetch cached configuration&lt;/td&gt;
&lt;td&gt;Generally suitable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create a payment&lt;/td&gt;
&lt;td&gt;Unsafe without idempotency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Submit an order&lt;/td&gt;
&lt;td&gt;Unsafe without deduplication&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Send an email&lt;/td&gt;
&lt;td&gt;May produce duplicates&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Update an account balance&lt;/td&gt;
&lt;td&gt;Unsafe without strong safeguards&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A hedge can reach two different servers before either has completed.&lt;/p&gt;

&lt;p&gt;Both servers may execute the operation.&lt;/p&gt;

&lt;p&gt;Canceling one request does not guarantee that its side effects are rolled back.&lt;/p&gt;

&lt;p&gt;For operations with side effects, use an idempotency key or another server-side deduplication mechanism.&lt;/p&gt;

&lt;p&gt;Even then, the server must correctly enforce the idempotency contract.&lt;/p&gt;

&lt;p&gt;Hedging should generally be introduced first for read-only operations where duplicate execution is safe.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Where Should Hedged Requests Be Sent?
&lt;/h2&gt;

&lt;p&gt;Sending a duplicate request to the same server is often ineffective.&lt;/p&gt;

&lt;p&gt;If that server is experiencing CPU contention, a long queue, or an overloaded database connection pool, the second request may suffer from the same problem.&lt;/p&gt;

&lt;p&gt;Ideally, the hedge should target an independent replica.&lt;/p&gt;

&lt;p&gt;But independence is not guaranteed.&lt;/p&gt;

&lt;p&gt;Two replicas may share:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The same database&lt;/li&gt;
&lt;li&gt;The same network path&lt;/li&gt;
&lt;li&gt;The same cache&lt;/li&gt;
&lt;li&gt;The same storage system&lt;/li&gt;
&lt;li&gt;The same overloaded region&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is called correlated failure or correlated latency.&lt;/p&gt;

&lt;p&gt;If both requests depend on the same struggling resource, hedging may increase load without meaningfully reducing latency.&lt;/p&gt;

&lt;p&gt;Replica selection should consider failure domains and actual dependency topology.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Observability: Measure the Cost of Hedging
&lt;/h2&gt;

&lt;p&gt;A hedging system needs more than a request-latency dashboard.&lt;/p&gt;

&lt;p&gt;You need to understand whether the reduction in tail latency justifies the additional work.&lt;/p&gt;

&lt;p&gt;Useful metrics include:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;hedged_requests_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Number of additional attempts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;hedge_wins_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Requests where a hedge returned first&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;hedge_budget_denied_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Hedges rejected by the budget&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;hedge_canceled_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Losing attempts canceled by the client&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;hedge_attempts_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Total attempts, including originals&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;hedge_delay_seconds&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Histogram&lt;/td&gt;
&lt;td&gt;Time before a hedge is launched&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;request_latency_seconds&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Histogram&lt;/td&gt;
&lt;td&gt;End-to-end request latency&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A useful derived metric is the hedge win rate:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
HedgeWinRate =&lt;br&gt;
\frac{HedgeWins}{HedgedRequests}&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;If hedges rarely win, you may be generating additional traffic without much benefit.&lt;/p&gt;

&lt;p&gt;Also compare:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;p50 latency&lt;/li&gt;
&lt;li&gt;p95 latency&lt;/li&gt;
&lt;li&gt;p99 latency&lt;/li&gt;
&lt;li&gt;Downstream request volume&lt;/li&gt;
&lt;li&gt;Dependency CPU and memory usage&lt;/li&gt;
&lt;li&gt;Cancellation effectiveness&lt;/li&gt;
&lt;li&gt;Error rate&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The goal is not to minimize p99 at any cost.&lt;/p&gt;

&lt;p&gt;It is to improve tail latency while keeping the additional workload within the dependency's capacity.&lt;/p&gt;
&lt;h2&gt;
  
  
  12. Testing Hedged Requests
&lt;/h2&gt;

&lt;p&gt;Concurrency behavior needs deterministic tests.&lt;/p&gt;

&lt;p&gt;At minimum, test these scenarios:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The primary request succeeds before the hedging delay.&lt;/li&gt;
&lt;li&gt;The primary request is slow and the hedge wins.&lt;/li&gt;
&lt;li&gt;The primary request fails with a retryable error.&lt;/li&gt;
&lt;li&gt;A fatal error cancels outstanding attempts.&lt;/li&gt;
&lt;li&gt;The hedge budget prevents additional requests.&lt;/li&gt;
&lt;li&gt;The parent context is canceled.&lt;/li&gt;
&lt;li&gt;The maximum number of attempts is respected.&lt;/li&gt;
&lt;li&gt;Losing operations receive cancellation.&lt;/li&gt;
&lt;li&gt;A losing operation that ignores cancellation does not release its concurrency permit prematurely.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A simple test for the winning hedge could use controlled channels instead of arbitrary sleeps:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestHedgeWins&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;primaryStarted&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{})&lt;/span&gt;
    &lt;span class="n"&gt;hedgeStarted&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nb"&gt;make&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;chan&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{})&lt;/span&gt;

    &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WithTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Background&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;policy&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;Policy&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;Delay&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;       &lt;span class="m"&gt;20&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;MaxAttempts&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;ShouldHedge&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;true&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="nb"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;primaryStarted&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

                &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="nb"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hedgeStarted&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s"&gt;"hedge response"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s"&gt;"hedge response"&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatalf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="s"&gt;"unexpected result: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;primaryStarted&lt;/span&gt;
    &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;hedgeStarted&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The test verifies that the second request can win and that the operation completes successfully.&lt;/p&gt;

&lt;p&gt;In a complete test suite, also verify that the primary operation observes cancellation.&lt;/p&gt;

&lt;p&gt;Run the race detector:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go &lt;span class="nb"&gt;test&lt;/span&gt; &lt;span class="nt"&gt;-race&lt;/span&gt; ./...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Finally, use load tests with a dependency that deliberately introduces variable latency.&lt;/p&gt;

&lt;p&gt;A useful experiment compares:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Hedging disabled.&lt;/li&gt;
&lt;li&gt;Hedging enabled with a conservative delay.&lt;/li&gt;
&lt;li&gt;Hedging enabled with an aggressive delay.&lt;/li&gt;
&lt;li&gt;Hedging enabled with a shared budget.&lt;/li&gt;
&lt;li&gt;Hedging enabled with adaptive concurrency.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Compare both latency improvements and additional downstream traffic.&lt;/p&gt;

&lt;h2&gt;
  
  
  13. Production Checklist
&lt;/h2&gt;

&lt;p&gt;Before introducing hedged requests into a production service:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Hedge only safe, idempotent operations.&lt;/li&gt;
&lt;li&gt;[ ] Select a delay from measured latency distributions.&lt;/li&gt;
&lt;li&gt;[ ] Use a strict maximum attempt count.&lt;/li&gt;
&lt;li&gt;[ ] Require a shared budget for additional attempts.&lt;/li&gt;
&lt;li&gt;[ ] Ensure hedges target independent replicas where possible.&lt;/li&gt;
&lt;li&gt;[ ] Apply concurrency limits to every attempt.&lt;/li&gt;
&lt;li&gt;[ ] Propagate context cancellation.&lt;/li&gt;
&lt;li&gt;[ ] Ensure permits remain held while work is still active.&lt;/li&gt;
&lt;li&gt;[ ] Respect the original request deadline.&lt;/li&gt;
&lt;li&gt;[ ] Measure hedge win rate and additional traffic.&lt;/li&gt;
&lt;li&gt;[ ] Test correlated latency and dependency overload.&lt;/li&gt;
&lt;li&gt;[ ] Roll out gradually and compare against a no-hedging baseline.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Tail latency is one of the most difficult problems in distributed systems because a small number of unusually slow operations can dominate the response time of an entire request.&lt;/p&gt;

&lt;p&gt;Hedged requests provide a way to reduce that impact.&lt;/p&gt;

&lt;p&gt;By launching a second request after a carefully selected delay, a client can avoid waiting for an unusually slow replica and use the first successful response.&lt;/p&gt;

&lt;p&gt;But hedging is not free.&lt;/p&gt;

&lt;p&gt;Every additional request consumes capacity, and uncontrolled duplication can make an already degraded dependency even slower.&lt;/p&gt;

&lt;p&gt;The right approach combines measured hedging delays, idempotent operations, cancellation, retry budgets, and adaptive concurrency limits.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The goal isn't to send more requests. It's to get the response sooner without making the system slower for everyone else.&lt;/strong&gt;&lt;/p&gt;




&lt;h3&gt;
  
  
  References
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://research.google/pubs/the-tail-at-scale/" rel="noopener noreferrer"&gt;The Tail at Scale — Google Research&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://grpc.io/docs/guides/request-hedging/" rel="noopener noreferrer"&gt;Request Hedging — gRPC Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/grpc/proposal/blob/master/A6-client-retries.md" rel="noopener noreferrer"&gt;gRPC Retry and Hedging Design&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>go</category>
      <category>backend</category>
      <category>productivity</category>
      <category>programming</category>
    </item>
    <item>
      <title>AG-UI &amp; A2UI: Building Agent-Powered Interfaces Beyond Chat</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Sun, 01 Nov 2026 14:09:45 +0000</pubDate>
      <link>https://dev.to/serifcolakel/ag-ui-a2ui-building-agent-powered-interfaces-beyond-chat-doi</link>
      <guid>https://dev.to/serifcolakel/ag-ui-a2ui-building-agent-powered-interfaces-beyond-chat-doi</guid>
      <description>&lt;p&gt;&lt;em&gt;How event-driven agent interactions and generative UI are changing the way we build AI-powered frontend applications.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Why Are Chat Interfaces No Longer Enough?
&lt;/h2&gt;

&lt;p&gt;Most AI applications start with a familiar interface: a text input, a message history, and a streaming response.&lt;/p&gt;

&lt;p&gt;This works well for question answering, summarization, and conversational experiences. However, agent-based applications introduce a different set of interaction requirements.&lt;/p&gt;

&lt;p&gt;Consider an AI assistant that analyzes a user's investment portfolio, investigates market movements, or prepares a technical report.&lt;/p&gt;

&lt;p&gt;A conventional chat interface might return a response containing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A summary of the analysis&lt;/li&gt;
&lt;li&gt;A list of recommendations&lt;/li&gt;
&lt;li&gt;Several data points&lt;/li&gt;
&lt;li&gt;A suggested next action&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;But what if the user needs more than a textual response?&lt;/p&gt;

&lt;p&gt;What if the application needs to display:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Interactive charts&lt;/li&gt;
&lt;li&gt;Dynamic forms&lt;/li&gt;
&lt;li&gt;Data tables&lt;/li&gt;
&lt;li&gt;Approval workflows&lt;/li&gt;
&lt;li&gt;Progress indicators&lt;/li&gt;
&lt;li&gt;Context-aware actions&lt;/li&gt;
&lt;li&gt;Multi-step interfaces that evolve as the agent works&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We could build every possible interface in advance and ask the model to select the appropriate components. This approach works for predictable workflows, but it becomes restrictive when the interaction itself is dynamic.&lt;/p&gt;

&lt;p&gt;Another option is to allow the model to generate HTML, React components, or JavaScript directly. However, executing arbitrary model-generated code introduces significant security, maintainability, and governance challenges.&lt;/p&gt;

&lt;p&gt;This is where two interesting technologies enter the picture:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AG-UI (Agent-User Interaction Protocol)&lt;/strong&gt; and &lt;strong&gt;A2UI (Agent-to-User Interface)&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Although their names are similar, they address different problems.&lt;/p&gt;

&lt;p&gt;AG-UI standardizes how agents communicate with user-facing applications. A2UI provides a declarative format for describing interfaces that client applications can safely render using their own component libraries.&lt;/p&gt;

&lt;p&gt;Together, they provide an interesting foundation for building agent-powered applications beyond traditional chat interfaces.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://docs.ag-ui.com/" rel="noopener noreferrer"&gt;AG-UI Documentation&lt;/a&gt; · &lt;a href="https://a2ui.org/" rel="noopener noreferrer"&gt;Google A2UI&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. What Is AG-UI?
&lt;/h2&gt;

&lt;p&gt;AG-UI stands for Agent-User Interaction Protocol.&lt;/p&gt;

&lt;p&gt;It is an open, lightweight, event-based protocol designed to standardize communication between AI agents and frontend applications.&lt;/p&gt;

&lt;p&gt;The underlying problem is straightforward.&lt;/p&gt;

&lt;p&gt;An agent does not always produce one response and finish. It may execute multiple tools, delegate work, update its state, request additional input, and continue processing.&lt;/p&gt;

&lt;p&gt;A traditional request-response model does not necessarily represent these interactions well.&lt;/p&gt;

&lt;p&gt;For example, imagine an agent performing a database analysis:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The user requests an analysis.&lt;/li&gt;
&lt;li&gt;The agent starts executing the task.&lt;/li&gt;
&lt;li&gt;It calls a database tool.&lt;/li&gt;
&lt;li&gt;The query completes.&lt;/li&gt;
&lt;li&gt;The agent processes the returned data.&lt;/li&gt;
&lt;li&gt;It generates a chart.&lt;/li&gt;
&lt;li&gt;It asks the user to confirm an additional filter.&lt;/li&gt;
&lt;li&gt;It completes the analysis.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If the frontend only receives the final response, the user has no visibility into what happened during execution.&lt;/p&gt;

&lt;p&gt;We could implement custom WebSocket messages, HTTP streaming events, and application-specific state synchronization. But every application would need to define and maintain its own communication contract.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AG-UI provides a standardized event model for this interaction.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It allows agent backends to communicate execution events, messages, tool calls, state changes, and activity updates to the frontend.&lt;/p&gt;

&lt;p&gt;The protocol is intentionally not tied to one agent framework or frontend technology.&lt;/p&gt;

&lt;p&gt;You can build the backend using LangGraph, Google ADK, CrewAI, or a custom runtime, while the frontend uses React or another supported client implementation.&lt;/p&gt;

&lt;p&gt;The important point is that AG-UI does not define how an agent reasons or executes its internal logic. It defines how that execution is communicated to the user-facing application.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://docs.ag-ui.com/" rel="noopener noreferrer"&gt;AG-UI — Introduction&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. How Does AG-UI Work?
&lt;/h2&gt;

&lt;p&gt;AG-UI follows an event-driven architecture.&lt;/p&gt;

&lt;p&gt;Instead of treating an agent interaction as a single response, the backend emits a sequence of typed events that the frontend consumes.&lt;/p&gt;

&lt;p&gt;A simplified architecture looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 User
                   |
                   v
             React Application
                   |
                   v
               AG-UI Client
                   |
            Event Stream / SSE
                   |
                   v
              Agent Runtime
                   |
          +--------+--------+
          |        |        |
          v        v        v
        LLM      Tools    Subagents
          |        |        |
          +--------+--------+
                   |
                   v
             Agent Events
                   |
                   v
              AG-UI Client
                   |
                   v
              Update UI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The frontend and backend have separate responsibilities.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Backend responsibilities:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Execute agent logic.&lt;/li&gt;
&lt;li&gt;Handle model interactions.&lt;/li&gt;
&lt;li&gt;Execute tools.&lt;/li&gt;
&lt;li&gt;Maintain execution state.&lt;/li&gt;
&lt;li&gt;Emit standardized events.&lt;/li&gt;
&lt;li&gt;Handle additional user input and resume interactions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Frontend responsibilities:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Initiate agent runs.&lt;/li&gt;
&lt;li&gt;Consume event streams.&lt;/li&gt;
&lt;li&gt;Render messages and activity updates.&lt;/li&gt;
&lt;li&gt;Synchronize application state.&lt;/li&gt;
&lt;li&gt;Present tool activity.&lt;/li&gt;
&lt;li&gt;Handle user interactions and approvals.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;AG-UI is transport-agnostic. Common integration patterns include Server-Sent Events (SSE), WebSockets, and webhooks.&lt;/p&gt;

&lt;p&gt;The official &lt;code&gt;HttpAgent&lt;/code&gt; client supports HTTP SSE and an HTTP binary transport.&lt;/p&gt;

&lt;p&gt;This separation means we do not have to make frontend components understand the implementation details of the backend agent framework.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://github.com/ag-ui-protocol/ag-ui/blob/main/docs/concepts/architecture.mdx" rel="noopener noreferrer"&gt;AG-UI — Core Architecture&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Understanding AG-UI Events
&lt;/h2&gt;

&lt;p&gt;Events are the foundation of AG-UI.&lt;/p&gt;

&lt;p&gt;Rather than sending an unstructured text response, an agent can communicate different types of events depending on what is happening during execution.&lt;/p&gt;

&lt;p&gt;The main event categories include:&lt;/p&gt;

&lt;h3&gt;
  
  
  4.1. Lifecycle Events
&lt;/h3&gt;

&lt;p&gt;These events describe the execution lifecycle.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;RUN_STARTED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;An agent run has started&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;RUN_FINISHED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;An agent run has finished&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;RUN_ERROR&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;An execution error occurred&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;STEP_STARTED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;A new execution step started&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;STEP_FINISHED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;An execution step finished&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This allows the frontend to display meaningful execution states instead of relying on a generic loading indicator.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.2. Text Message Events
&lt;/h3&gt;

&lt;p&gt;These events represent streaming messages.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;TEXT_MESSAGE_START&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Starts a message&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;TEXT_MESSAGE_CONTENT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sends a text chunk&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;TEXT_MESSAGE_END&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Completes a message&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The frontend can append each text chunk as it arrives.&lt;/p&gt;

&lt;p&gt;This provides a familiar streaming experience while keeping the underlying event structure explicit.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.3. Tool Call Events
&lt;/h3&gt;

&lt;p&gt;Tool execution is an important part of agent workflows.&lt;/p&gt;

&lt;p&gt;AG-UI defines events such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;TOOL_CALL_START&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;TOOL_CALL_ARGS&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;TOOL_CALL_END&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;TOOL_CALL_RESULT&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These events allow the frontend to display which tool is being used and follow its execution lifecycle.&lt;/p&gt;

&lt;p&gt;For example, an interface might display:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Analyzing repository...
    |
    v
GitLab: Fetching merge request
    |
    v
Code Review: Analyzing changes
    |
    v
Security Check: Completed
    |
    v
Generating report
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One distinction is important: receiving a tool-call event does not automatically mean the frontend should execute that tool. Tool execution and authorization remain responsibilities of the appropriate runtime or application layer.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.4. State Management Events
&lt;/h3&gt;

&lt;p&gt;AG-UI also defines events for synchronizing application and agent state.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;STATE_SNAPSHOT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sends a complete state snapshot&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;STATE_DELTA&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sends incremental state changes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;MESSAGES_SNAPSHOT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Synchronizes conversation history&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;For example, an agent might update the current analysis status:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"STATE_SNAPSHOT"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"snapshot"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"analyzing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"completedSteps"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"totalSteps"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"currentTask"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Evaluating transaction data"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The frontend can consume this state and update progress indicators, dashboards, or other components.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.5. Activity and Subagent Events
&lt;/h3&gt;

&lt;p&gt;Modern agent systems often involve multiple agents working together.&lt;/p&gt;

&lt;p&gt;AG-UI includes activity and subagent-related events, allowing a frontend to represent more complex execution patterns.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A research agent starts working.&lt;/li&gt;
&lt;li&gt;A data analysis agent receives a delegated task.&lt;/li&gt;
&lt;li&gt;The security agent completes its review.&lt;/li&gt;
&lt;li&gt;The coordinator produces the final result.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This makes it possible to build interfaces that expose useful execution activity without requiring users to understand the internal implementation of each agent.&lt;/p&gt;

&lt;p&gt;The current AG-UI v1.0 schema documents event families for lifecycle, text, tools, state, activity, subagents, reasoning, and custom events.&lt;/p&gt;

&lt;p&gt;A critical implementation consideration: reasoning-related events should not be treated as permission to expose raw model chain-of-thought. User-facing applications should expose approved progress summaries and operational status instead.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://github.com/ag-ui-protocol/ag-ui/blob/main/docs/concepts/events.mdx" rel="noopener noreferrer"&gt;AG-UI — Event Types and Specification&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  5. How to Integrate AG-UI with a React Application
&lt;/h2&gt;

&lt;p&gt;AG-UI provides a TypeScript SDK that can be used to connect frontend applications to compatible agent endpoints.&lt;/p&gt;

&lt;p&gt;The official package is &lt;code&gt;@ag-ui/client&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Install the package
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm add @ag-ui/client @ag-ui/core
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 2: Create an HTTP agent
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;HttpAgent&lt;/code&gt; class provides a client for communicating with an AG-UI-compatible backend.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;HttpAgent&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@ag-ui/client&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;agent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;HttpAgent&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/api/agent&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;agentId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;portfolio-assistant&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;threadId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The configuration defines the backend endpoint and identifies the agent and conversation thread.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Send a message and consume events
&lt;/h3&gt;

&lt;p&gt;We can use the agent's &lt;code&gt;addMessage&lt;/code&gt; and &lt;code&gt;runAgent&lt;/code&gt; methods to start an interaction.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;HttpAgent&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@ag-ui/client&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;EventType&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@ag-ui/core&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;agent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;HttpAgent&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/api/agent&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;agentId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;portfolio-assistant&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;threadId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="nx"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;addMessage&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
  &lt;span class="na"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;user&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Analyze my portfolio and summarize its performance.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;runAgent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt;
    &lt;span class="na"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;onEvent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;switch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;EventType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="na"&gt;RUN_STARTED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Agent execution started&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
          &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;EventType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="na"&gt;TEXT_MESSAGE_CONTENT&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;delta&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
          &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;EventType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="na"&gt;TOOL_CALL_START&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Tool started:&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;toolCallName&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
          &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;EventType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="na"&gt;RUN_FINISHED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Agent execution completed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
          &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="nx"&gt;EventType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="na"&gt;RUN_ERROR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Agent execution failed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
          &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The example demonstrates the client-side event handling pattern. In a complete application, event handlers would update React state, render tool activity, and maintain conversation state.&lt;/p&gt;

&lt;p&gt;The agent SDK also supports shared state, event subscriptions, middleware, and cancellation through &lt;code&gt;abortRun()&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Synchronize agent state with React
&lt;/h3&gt;

&lt;p&gt;One of the most useful capabilities is the ability to keep frontend and agent state synchronized.&lt;/p&gt;

&lt;p&gt;For example, an agent can update task progress while the frontend renders the current status.&lt;/p&gt;

&lt;p&gt;With a framework integration such as CopilotKit, this can be implemented through hooks such as &lt;code&gt;useAgent&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;useAgent&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@copilotkit/react-core/v2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;TaskProgress&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;agent&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;useAgent&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;progress&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;agent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;progress&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;completed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;total&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;

  &lt;span class="k"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;section&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;h3&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;Agent Progress&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;h3&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;

      &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;p&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;progress&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;completed&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt; / &lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;progress&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;total&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt; completed
      &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;p&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;section&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The component reacts to changes in the agent's shared state.&lt;/p&gt;

&lt;p&gt;The important architectural detail is that AG-UI defines the communication contract, while libraries such as CopilotKit provide additional frontend integration and application-level abstractions.&lt;/p&gt;

&lt;p&gt;We are not required to use CopilotKit to implement AG-UI. A custom frontend can consume the protocol directly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://github.com/ag-ui-protocol/ag-ui" rel="noopener noreferrer"&gt;AG-UI TypeScript SDK&lt;/a&gt; · &lt;a href="https://docs.copilotkit.ai/shared-state" rel="noopener noreferrer"&gt;CopilotKit Shared State&lt;/a&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  6. What Is A2UI?
&lt;/h2&gt;

&lt;p&gt;AG-UI solves the communication problem, but there is another question:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What if an agent needs to generate an actual user interface instead of returning text?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A2UI stands for Agent-to-User Interface.&lt;/p&gt;

&lt;p&gt;It is an open-source project initiated by Google that provides a declarative protocol for generating and updating user interfaces from structured JSON messages.&lt;/p&gt;

&lt;p&gt;The central idea is simple.&lt;/p&gt;

&lt;p&gt;Instead of asking an LLM to generate HTML, JavaScript, or React code, we let it describe what the interface should contain.&lt;/p&gt;

&lt;p&gt;For example, an agent may want to display:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A title&lt;/li&gt;
&lt;li&gt;A summary card&lt;/li&gt;
&lt;li&gt;A table&lt;/li&gt;
&lt;li&gt;A date picker&lt;/li&gt;
&lt;li&gt;A confirmation button&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It can describe these elements using A2UI messages.&lt;/p&gt;

&lt;p&gt;The frontend then maps those descriptions to its own approved components.&lt;/p&gt;

&lt;p&gt;This creates a separation between UI generation and UI execution.&lt;/p&gt;

&lt;p&gt;The agent describes the interface. The application decides how that interface is rendered.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why is this important?
&lt;/h3&gt;

&lt;p&gt;Imagine allowing an LLM to generate the following:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;Dashboard&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// Arbitrary model-generated code&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Executing model-generated React code directly would create substantial security and maintenance problems.&lt;/p&gt;

&lt;p&gt;Instead, we can let the model return structured data:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"component"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Card"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"children"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"component"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Portfolio Summary"&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The frontend maps &lt;code&gt;Card&lt;/code&gt; and &lt;code&gt;Text&lt;/code&gt; to components that already exist in its design system.&lt;/p&gt;

&lt;p&gt;The model never needs to execute arbitrary JavaScript to create the interface.&lt;/p&gt;

&lt;p&gt;This is the main architectural idea behind A2UI.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://a2ui.org/" rel="noopener noreferrer"&gt;Google A2UI — Official Documentation&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  7. How Does A2UI Work?
&lt;/h2&gt;

&lt;p&gt;A2UI uses a JSON-based streaming protocol.&lt;/p&gt;

&lt;p&gt;The agent sends messages to the frontend, and the frontend processes those messages to create or update UI surfaces.&lt;/p&gt;

&lt;p&gt;A2UI v0.9.1 defines four primary message types.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.1. createSurface
&lt;/h3&gt;

&lt;p&gt;Creates a new UI surface.&lt;/p&gt;

&lt;p&gt;A surface represents a UI region that contains components and associated data.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"v0.9.1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"createSurface"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"surfaceId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"portfolio-dashboard"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"catalogId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://a2ui.org/specification/v0_9_1/catalogs/basic/catalog.json"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  7.2. updateComponents
&lt;/h3&gt;

&lt;p&gt;Defines or updates the components within a surface.&lt;/p&gt;

&lt;p&gt;The components are represented as a flat list with references that define their relationships.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"v0.9.1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"updateComponents"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"surfaceId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"portfolio-dashboard"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"components"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"root"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"component"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Column"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"children"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
          &lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
          &lt;/span&gt;&lt;span class="s2"&gt;"summary"&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"component"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Portfolio Overview"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"variant"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"h1"&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"summary"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"component"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="nl"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Your portfolio analysis is ready."&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The renderer understands the component definitions and constructs the corresponding UI tree.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.3. updateDataModel
&lt;/h3&gt;

&lt;p&gt;Updates the data displayed by the components.&lt;/p&gt;

&lt;p&gt;A2UI separates component structure from application data.&lt;/p&gt;

&lt;p&gt;This is particularly useful because changing data does not necessarily require sending the entire component tree again.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"v0.9.1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"updateDataModel"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"surfaceId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"portfolio-dashboard"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"path"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"/portfolio"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"totalValue"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;125000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"currency"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"USD"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"change"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;4.2&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Components can reference values in the data model using JSON Pointer paths.&lt;/p&gt;

&lt;p&gt;When the data model changes, the renderer can update the relevant parts of the interface.&lt;/p&gt;

&lt;h3&gt;
  
  
  7.4. deleteSurface
&lt;/h3&gt;

&lt;p&gt;Removes a previously created surface and its associated components and data.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"v0.9.1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"deleteSurface"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"surfaceId"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"portfolio-dashboard"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is useful for managing dynamic UI regions as an interaction evolves.&lt;/p&gt;

&lt;h3&gt;
  
  
  The complete flow
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User Request
     |
     v
Agent Runtime
     |
     v
LLM Generates A2UI Messages
     |
     v
    JSON Stream
     |
     v
A2UI Message Processor
     |
     v
Component Catalog
     |
     v
React / Flutter / Angular Renderer
     |
     v
Interactive User Interface
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The process can happen incrementally.&lt;/p&gt;

&lt;p&gt;The agent can create a surface first, add components later, and update the data model as more information becomes available.&lt;/p&gt;

&lt;p&gt;This is different from generating an entire interface in one response.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://github.com/a2ui-project/a2ui/blob/main/specification/v0_9_1/docs/a2ui_protocol.md" rel="noopener noreferrer"&gt;A2UI Protocol v0.9.1&lt;/a&gt; · &lt;a href="https://github.com/a2ui-project/a2ui/blob/main/docs/public/quickstart.md" rel="noopener noreferrer"&gt;A2UI Quickstart&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Integrating A2UI with React
&lt;/h2&gt;

&lt;p&gt;A2UI provides an official React renderer and a shared web core library.&lt;/p&gt;

&lt;p&gt;The relevant packages are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;@a2ui/react&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;@a2ui/web_core&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Step 1: Install the packages
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm add @a2ui/react @a2ui/web_core
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 2: Create a MessageProcessor
&lt;/h3&gt;

&lt;p&gt;The &lt;code&gt;MessageProcessor&lt;/code&gt; handles protocol messages, surface lifecycle, component definitions, and data models.&lt;/p&gt;

&lt;p&gt;For the v0.9 implementation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;useState&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;react&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;MessageProcessor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@a2ui/web_core/v0_9&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;A2uiSurface&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;basicCatalog&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@a2ui/react/v0_9&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The processor is initialized with the component catalog that the application allows the agent to use.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;processor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;MessageProcessor&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
  &lt;span class="nx"&gt;basicCatalog&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Step 3: Process agent-generated messages
&lt;/h3&gt;

&lt;p&gt;For example, we can create a surface and populate it with components:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="nx"&gt;processor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;processMessages&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;v0.9&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;createSurface&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;surfaceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;main&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;catalogId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;basicCatalog&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;v0.9&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;updateComponents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;surfaceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;main&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;components&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
          &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;root&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;component&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Column&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;children&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;heading&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;description&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
          &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;heading&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;component&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Text&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Hello from A2UI&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;variant&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;h1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
          &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;description&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;component&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Text&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;This interface was described by an agent.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;MessageProcessor&lt;/code&gt; maintains the surface and its component model.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Render the surface
&lt;/h3&gt;

&lt;p&gt;Once the surface has been created, the React renderer can display it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;DynamicUI&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;processor&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;useState&lt;/span&gt;&lt;span class="p"&gt;(()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;instance&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;MessageProcessor&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
      &lt;span class="nx"&gt;basicCatalog&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]);&lt;/span&gt;

    &lt;span class="nx"&gt;instance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;processMessages&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;v0.9&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;createSurface&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
          &lt;span class="na"&gt;surfaceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;main&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;catalogId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;basicCatalog&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="na"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;v0.9&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;updateComponents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
          &lt;span class="na"&gt;surfaceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;main&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="na"&gt;components&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
              &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;root&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="na"&gt;component&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Column&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="na"&gt;children&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;title&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
            &lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
              &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;title&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="na"&gt;component&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Text&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="na"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Generated Interface&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="na"&gt;variant&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;h1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;},&lt;/span&gt;
          &lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;]);&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;instance&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;surface&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;processor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;model&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;surfacesMap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;main&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;surface&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;p&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;Preparing interface...&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;p&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;A2uiSurface&lt;/span&gt; &lt;span class="na"&gt;surface&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;surface&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt; &lt;span class="p"&gt;/&amp;gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a minimal rendering example. A production implementation should create and manage the processor at the appropriate application or session scope, rather than reconstructing it for each update.&lt;/p&gt;

&lt;p&gt;As new A2UI messages arrive, the application can pass them to the same processor. The surface model drives the React renderer's updates.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Handle user interactions
&lt;/h3&gt;

&lt;p&gt;An interactive UI needs a return path.&lt;/p&gt;

&lt;p&gt;For example, the agent may generate a form containing a confirmation button.&lt;/p&gt;

&lt;p&gt;When the user clicks that button, the frontend captures the associated action and sends it back to the agent through the selected transport.&lt;/p&gt;

&lt;p&gt;The flow becomes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Agent Generates UI
       |
       v
A2UI Renderer
       |
       v
User Interacts
       |
       v
Frontend Captures Action
       |
       v
Application / Agent Endpoint
       |
       v
Agent Processes Input
       |
       v
New A2UI Messages
       |
       v
Updated Interface
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A2UI v0.9.1 uses an external communication channel for these return interactions. Its v1.0 candidate specification introduces additional structured function-call and response mechanisms.&lt;/p&gt;

&lt;p&gt;This distinction matters when designing the frontend-backend contract.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://github.com/a2ui-project/a2ui/tree/main/renderers/react" rel="noopener noreferrer"&gt;A2UI React Renderer&lt;/a&gt; · &lt;a href="https://github.com/a2ui-project/a2ui/blob/main/docs/public/guides/client-setup.md" rel="noopener noreferrer"&gt;A2UI Client Setup&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  9. What Is the Difference Between AG-UI and A2UI?
&lt;/h2&gt;

&lt;p&gt;This is probably the most important distinction in the article.&lt;/p&gt;

&lt;p&gt;AG-UI and A2UI are complementary, but they are not interchangeable.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Area&lt;/th&gt;
&lt;th&gt;AG-UI&lt;/th&gt;
&lt;th&gt;A2UI&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Primary purpose&lt;/td&gt;
&lt;td&gt;Agent-frontend communication&lt;/td&gt;
&lt;td&gt;Declarative UI generation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Main abstraction&lt;/td&gt;
&lt;td&gt;Typed events&lt;/td&gt;
&lt;td&gt;UI surfaces and components&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data flow&lt;/td&gt;
&lt;td&gt;Run events and state updates&lt;/td&gt;
&lt;td&gt;UI structure and data updates&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Main responsibility&lt;/td&gt;
&lt;td&gt;Interaction lifecycle&lt;/td&gt;
&lt;td&gt;Interface representation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Frontend role&lt;/td&gt;
&lt;td&gt;Consume and handle events&lt;/td&gt;
&lt;td&gt;Render approved components&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;User interaction&lt;/td&gt;
&lt;td&gt;Agent runs, tools, state and interrupts&lt;/td&gt;
&lt;td&gt;UI actions and data context&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Consider an agent that needs to create a dynamic approval dashboard.&lt;/p&gt;

&lt;p&gt;AG-UI can communicate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The agent has started processing.&lt;/li&gt;
&lt;li&gt;The data retrieval step is complete.&lt;/li&gt;
&lt;li&gt;A tool has returned results.&lt;/li&gt;
&lt;li&gt;The agent has updated its state.&lt;/li&gt;
&lt;li&gt;A user approval is required.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A2UI can describe:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;An approval summary card.&lt;/li&gt;
&lt;li&gt;A table of requested changes.&lt;/li&gt;
&lt;li&gt;A confirmation button.&lt;/li&gt;
&lt;li&gt;A detailed information panel.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The frontend can render both the interaction lifecycle and the generated interface.&lt;/p&gt;

&lt;h3&gt;
  
  
  Can we use them together?
&lt;/h3&gt;

&lt;p&gt;Yes.&lt;/p&gt;

&lt;p&gt;A possible architecture looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                User
                  |
                  v
            React Frontend
                  |
                  v
               AG-UI
             Event Stream
                  |
                  v
             Agent Runtime
                  |
                  v
                 LLM
                  |
            +-----+-----+
            |           |
            v           v
        AG-UI Events   A2UI JSON
            |           |
            +-----+-----+
                  |
                  v
            Frontend Host
                  |
                  v
            A2UI Renderer
                  |
                  v
           Native React UI
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;AG-UI manages the interaction boundary. A2UI describes the interface that should be rendered.&lt;/p&gt;

&lt;p&gt;One protocol does not replace the other.&lt;/p&gt;

&lt;p&gt;A2UI can also be transported over A2A, MCP-related integrations, SSE, or WebSockets. It is designed to remain independent of a particular transport.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://github.com/ag-ui-protocol/ag-ui/blob/main/docs/concepts/architecture.mdx" rel="noopener noreferrer"&gt;AG-UI Architecture&lt;/a&gt; · &lt;a href="https://github.com/a2ui-project/a2ui/blob/main/specification/v0_9_1/docs/a2ui_protocol.md" rel="noopener noreferrer"&gt;A2UI Transport Decoupling&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  10. How Do They Fit Into the Existing AI Ecosystem?
&lt;/h2&gt;

&lt;p&gt;We have already discussed MCP, ACP, and A2A in previous articles.&lt;/p&gt;

&lt;p&gt;Adding AG-UI and A2UI gives us another important architectural layer.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Technology&lt;/th&gt;
&lt;th&gt;Responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;MCP&lt;/td&gt;
&lt;td&gt;Agent ↔ Tools and external data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ACP&lt;/td&gt;
&lt;td&gt;Coding Agent ↔ Client Application&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A2A&lt;/td&gt;
&lt;td&gt;Agent ↔ Agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AG-UI&lt;/td&gt;
&lt;td&gt;Agent ↔ Frontend interaction&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A2UI&lt;/td&gt;
&lt;td&gt;Agent → Declarative UI&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A typical enterprise AI platform could use several of these technologies simultaneously.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;An application communicates with an agent using AG-UI.&lt;/li&gt;
&lt;li&gt;The agent accesses internal services through MCP.&lt;/li&gt;
&lt;li&gt;The agent delegates a specialized task to another agent using A2A.&lt;/li&gt;
&lt;li&gt;The agent returns dynamic interface definitions using A2UI.&lt;/li&gt;
&lt;li&gt;The application renders those definitions through its own component system.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;ACP can be used independently when we need to integrate coding agents with a client application.&lt;/p&gt;

&lt;p&gt;This is not about adopting every protocol. It is about keeping architectural boundaries clear.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. How Do These Approaches Compare With Traditional Generative UI?
&lt;/h2&gt;

&lt;p&gt;AG-UI and A2UI are not the only ways to build agent-powered interfaces.&lt;/p&gt;

&lt;p&gt;There are several approaches worth understanding.&lt;/p&gt;

&lt;h3&gt;
  
  
  AI SDK Tool-Based UI
&lt;/h3&gt;

&lt;p&gt;Vercel AI SDK already allows applications to render custom UI based on tool calls and structured message parts.&lt;/p&gt;

&lt;p&gt;For example, a model can call a weather tool, and the frontend can render a weather card instead of displaying only raw text.&lt;/p&gt;

&lt;p&gt;This is a good approach when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The application has a known set of components.&lt;/li&gt;
&lt;li&gt;UI behavior is closely tied to model tool calls.&lt;/li&gt;
&lt;li&gt;The application owns the entire interaction contract.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;However, it does not automatically provide the same cross-framework UI specification and agent interaction model as AG-UI and A2UI.&lt;/p&gt;

&lt;h3&gt;
  
  
  MCP Apps
&lt;/h3&gt;

&lt;p&gt;MCP Apps extend MCP tools with interactive UI experiences.&lt;/p&gt;

&lt;p&gt;A tool can expose a &lt;code&gt;ui://&lt;/code&gt; resource containing its interface, which an MCP host can render in a sandboxed iframe.&lt;/p&gt;

&lt;p&gt;This is particularly useful when a tool needs to provide its own interactive experience, such as a dashboard, form, or specialized workflow.&lt;/p&gt;

&lt;p&gt;It solves a different problem from A2UI: the UI is associated with a tool rather than being a general declarative UI surface generated by an agent.&lt;/p&gt;

&lt;h3&gt;
  
  
  A2UI
&lt;/h3&gt;

&lt;p&gt;A2UI focuses on agent-generated interface descriptions and framework-specific rendering.&lt;/p&gt;

&lt;p&gt;The same declarative description can be consumed by compatible clients with different native component implementations.&lt;/p&gt;

&lt;h3&gt;
  
  
  AG-UI
&lt;/h3&gt;

&lt;p&gt;AG-UI focuses on the interaction protocol rather than the visual representation.&lt;/p&gt;

&lt;p&gt;It can be used with ordinary chat interfaces, custom React components, or declarative generative UI approaches.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;Best suited for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;AI SDK Tool UI&lt;/td&gt;
&lt;td&gt;Application-defined component rendering&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MCP Apps&lt;/td&gt;
&lt;td&gt;Interactive UIs provided by individual tools&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A2UI&lt;/td&gt;
&lt;td&gt;Agent-generated, declarative interfaces&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AG-UI&lt;/td&gt;
&lt;td&gt;Standardized agent-frontend communication&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These approaches are not mutually exclusive.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://ai-sdk.dev/docs/reference/ai-sdk-ui/use-chat" rel="noopener noreferrer"&gt;AI SDK — useChat&lt;/a&gt; · &lt;a href="https://vercel.com/kb/guide/ai-sdk-mcp-apps" rel="noopener noreferrer"&gt;Vercel — MCP Apps Integration&lt;/a&gt; · &lt;a href="https://a2ui.org/" rel="noopener noreferrer"&gt;Google A2UI&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  12. Advantages and Trade-offs
&lt;/h2&gt;

&lt;p&gt;Introducing these technologies can improve architecture and developer experience, but they also introduce new responsibilities.&lt;/p&gt;

&lt;h3&gt;
  
  
  AG-UI Advantages
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Standardized event communication&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Teams can consume a consistent event model instead of maintaining separate custom streaming contracts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Frontend-backend separation&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The frontend does not need to understand the internal implementation of each agent framework.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Better visibility into execution&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Tool calls, progress updates, and agent state can be represented in the UI.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Framework interoperability&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Different backend agent implementations can expose a common interaction contract.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Support for human-in-the-loop workflows&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Applications can build approval, clarification, and user-steering experiences around agent execution.&lt;/p&gt;

&lt;h3&gt;
  
  
  AG-UI Trade-offs
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Existing agent frameworks may require adapters or event translation.&lt;/li&gt;
&lt;li&gt;Teams must manage protocol versions and event compatibility.&lt;/li&gt;
&lt;li&gt;Application state and agent state can become inconsistent if ownership is unclear.&lt;/li&gt;
&lt;li&gt;Authentication, authorization, and event validation remain application responsibilities.&lt;/li&gt;
&lt;li&gt;For simple chat applications, a custom endpoint or ordinary AI SDK integration may be sufficient.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  A2UI Advantages
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Declarative and safer UI generation&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Agents send structured data rather than executable React or JavaScript code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Framework independence&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The same UI description can be mapped to different client-side component systems.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reuse of existing design systems&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Applications can expose their own approved component catalog.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Incremental rendering&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The UI structure and data can be updated without resending everything.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Separation of structure and data&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Components can remain stable while the agent updates the underlying data model.&lt;/p&gt;

&lt;h3&gt;
  
  
  A2UI Trade-offs
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;A component catalog must be designed and maintained.&lt;/li&gt;
&lt;li&gt;The renderer must support the required component types.&lt;/li&gt;
&lt;li&gt;Complex application-specific interactions require additional integration.&lt;/li&gt;
&lt;li&gt;Schema and protocol version compatibility require attention.&lt;/li&gt;
&lt;li&gt;Debugging streamed component updates can be more difficult than debugging ordinary React components.&lt;/li&gt;
&lt;li&gt;Cross-platform rendering does not guarantee identical visual behavior on every platform.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An important point: declarative JSON is safer than arbitrary executable code, but it is not automatically trustworthy.&lt;/p&gt;

&lt;p&gt;The application must still validate messages, restrict available components, and authorize actions.&lt;/p&gt;

&lt;h2&gt;
  
  
  13. Security and Production Considerations
&lt;/h2&gt;

&lt;p&gt;This is where the difference between a successful demo and a maintainable production implementation becomes important.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Never execute arbitrary generated code
&lt;/h3&gt;

&lt;p&gt;An agent should not be able to send arbitrary JavaScript, HTML, or React code and have the application execute it.&lt;/p&gt;

&lt;p&gt;A2UI's declarative component catalog provides a safer approach.&lt;/p&gt;

&lt;p&gt;The application should only render components it explicitly supports.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Validate incoming messages
&lt;/h3&gt;

&lt;p&gt;AG-UI events and A2UI messages must be treated as untrusted input.&lt;/p&gt;

&lt;p&gt;Validate event structures, protocol versions, component definitions, data paths, and payload sizes before processing them.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Control available components
&lt;/h3&gt;

&lt;p&gt;Not every component should be available to an agent.&lt;/p&gt;

&lt;p&gt;For example, an internal banking application might allow:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Summary cards&lt;/li&gt;
&lt;li&gt;Data tables&lt;/li&gt;
&lt;li&gt;Charts&lt;/li&gt;
&lt;li&gt;Read-only forms&lt;/li&gt;
&lt;li&gt;Approval request components&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;However, a sensitive transaction operation should not become available merely because the agent can generate a button.&lt;/p&gt;

&lt;p&gt;The backend must authorize the actual business operation.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Keep application state under application control
&lt;/h3&gt;

&lt;p&gt;There should be a clear distinction between UI state, agent state, and authoritative business state.&lt;/p&gt;

&lt;p&gt;An agent-generated UI should not be considered a source of truth for financial transactions, permissions, or other business-critical information.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Require human approval for sensitive actions
&lt;/h3&gt;

&lt;p&gt;A dynamically generated interface can make it easier for an agent to request user confirmation.&lt;/p&gt;

&lt;p&gt;However, the approval must be verified on the backend. Hiding or disabling a frontend button is not an authorization mechanism.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Handle interrupted streams and reconnection
&lt;/h3&gt;

&lt;p&gt;Streaming applications need predictable recovery behavior.&lt;/p&gt;

&lt;p&gt;Consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reconnecting after network failures&lt;/li&gt;
&lt;li&gt;Handling duplicate events&lt;/li&gt;
&lt;li&gt;Recovering from incomplete UI updates&lt;/li&gt;
&lt;li&gt;Preserving task and surface identifiers&lt;/li&gt;
&lt;li&gt;Handling cancellation&lt;/li&gt;
&lt;li&gt;Expiring stale sessions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;AG-UI and A2UI define communication structures, but the application still needs an operational strategy for stream recovery and state consistency.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Audit agent-generated interactions
&lt;/h3&gt;

&lt;p&gt;For enterprise applications, it should be possible to understand:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which agent generated a UI.&lt;/li&gt;
&lt;li&gt;Which components were requested.&lt;/li&gt;
&lt;li&gt;Which actions the user performed.&lt;/li&gt;
&lt;li&gt;Which backend operations were executed.&lt;/li&gt;
&lt;li&gt;Whether an approval was obtained.&lt;/li&gt;
&lt;li&gt;What the final result was.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is particularly important when AI interfaces are used to interact with internal business systems.&lt;/p&gt;

&lt;h2&gt;
  
  
  14. What Can an AI Chapter Lead Contribute?
&lt;/h2&gt;

&lt;p&gt;I believe this is where these technologies become particularly valuable.&lt;/p&gt;

&lt;p&gt;The opportunity is not simply introducing generative UI into an application. It is establishing a reusable frontend-agent architecture that product teams can build on.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Establish a Shared Agent-Frontend Integration Layer
&lt;/h3&gt;

&lt;p&gt;Instead of every team implementing its own event handling and agent-state synchronization, we can define a shared integration layer based on AG-UI.&lt;/p&gt;

&lt;p&gt;It can provide:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Event subscription and processing&lt;/li&gt;
&lt;li&gt;Streaming message handling&lt;/li&gt;
&lt;li&gt;Tool execution indicators&lt;/li&gt;
&lt;li&gt;State synchronization&lt;/li&gt;
&lt;li&gt;Cancellation and error handling&lt;/li&gt;
&lt;li&gt;Human approval workflows&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This reduces duplicated infrastructure across applications.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Create a Governed Component Catalog
&lt;/h3&gt;

&lt;p&gt;A2UI introduces an interesting opportunity to connect agent capabilities with an organization's existing design system.&lt;/p&gt;

&lt;p&gt;For example, we can define an approved component catalog:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SummaryCard&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Display structured summaries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DataTable&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Render tabular information&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;LineChart&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Display time-series data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ApprovalPanel&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Present approval requests&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DateRangePicker&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Collect date filters&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;StatusIndicator&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Display execution status&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The agent can request these components, while the frontend maps them to existing React components.&lt;/p&gt;

&lt;p&gt;This means teams can introduce dynamic interfaces without giving the model unrestricted access to the frontend codebase.&lt;/p&gt;

&lt;p&gt;For organizations with established design systems, this could be particularly valuable.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Define Reusable Frontend Patterns
&lt;/h3&gt;

&lt;p&gt;We can establish standard patterns for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agent progress indicators&lt;/li&gt;
&lt;li&gt;Interactive forms&lt;/li&gt;
&lt;li&gt;Approval interfaces&lt;/li&gt;
&lt;li&gt;Dynamic dashboards&lt;/li&gt;
&lt;li&gt;Multi-step workflows&lt;/li&gt;
&lt;li&gt;Tool execution views&lt;/li&gt;
&lt;li&gt;Subagent activity panels&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These patterns can become shared components and examples that product teams reuse.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Standardize the Security Model
&lt;/h3&gt;

&lt;p&gt;A shared approach should define which components agents can request, which actions require approval, and how the backend validates every operation.&lt;/p&gt;

&lt;p&gt;This is especially important for applications interacting with sensitive business systems.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Build a Reference Implementation
&lt;/h3&gt;

&lt;p&gt;A small reference application could demonstrate the complete interaction:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A user asks an agent to analyze a dataset.&lt;/li&gt;
&lt;li&gt;The agent starts execution and sends AG-UI events.&lt;/li&gt;
&lt;li&gt;The frontend displays progress and activity.&lt;/li&gt;
&lt;li&gt;The agent generates an A2UI dashboard.&lt;/li&gt;
&lt;li&gt;The React renderer maps the description to approved components.&lt;/li&gt;
&lt;li&gt;The user modifies a filter or submits an action.&lt;/li&gt;
&lt;li&gt;The frontend sends the interaction back to the agent.&lt;/li&gt;
&lt;li&gt;The agent updates the interface with new results.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This would help developers understand how the protocols work together in a real application.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Measure the Developer and User Experience
&lt;/h3&gt;

&lt;p&gt;We should not evaluate generative UI only by how quickly an agent can produce a visual result.&lt;/p&gt;

&lt;p&gt;Useful metrics include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Time to interactive UI&lt;/li&gt;
&lt;li&gt;Agent-to-frontend event latency&lt;/li&gt;
&lt;li&gt;UI schema validation failure rate&lt;/li&gt;
&lt;li&gt;Unsupported component requests&lt;/li&gt;
&lt;li&gt;Rendering errors&lt;/li&gt;
&lt;li&gt;User interaction completion rate&lt;/li&gt;
&lt;li&gt;Human approval rate&lt;/li&gt;
&lt;li&gt;Frontend integration effort&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These metrics can help determine whether generative UI actually improves the application experience.&lt;/p&gt;

&lt;h2&gt;
  
  
  15. What Is the Current State of A2UI and AG-UI?
&lt;/h2&gt;

&lt;p&gt;Both projects are evolving, so protocol versioning matters.&lt;/p&gt;

&lt;p&gt;As of November 1, 2026:&lt;/p&gt;

&lt;h3&gt;
  
  
  AG-UI
&lt;/h3&gt;

&lt;p&gt;AG-UI publishes a versioned v1.0 specification and provides TypeScript client libraries, event schemas, and integrations with different agent frameworks.&lt;/p&gt;

&lt;p&gt;Its ecosystem includes integrations with projects such as LangGraph, CrewAI, Google ADK, and other agent runtimes.&lt;/p&gt;

&lt;h3&gt;
  
  
  A2UI
&lt;/h3&gt;

&lt;p&gt;A2UI currently identifies v0.9.1 as its production protocol release.&lt;/p&gt;

&lt;p&gt;The v1.0 specification is a candidate release, not the production version.&lt;/p&gt;

&lt;p&gt;The project also provides an official React renderer, shared web core, Lit renderer, Angular renderer, and Flutter-based support.&lt;/p&gt;

&lt;p&gt;There is an important migration consideration: the v1.0 candidate introduces changes to the protocol model and should not be assumed to be wire-compatible with v0.9.1.&lt;/p&gt;

&lt;p&gt;For production applications, I would start with the stable v0.9.1 protocol and its matching renderer, and evaluate v1.0 separately before planning a migration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://github.com/a2ui-project/a2ui" rel="noopener noreferrer"&gt;A2UI Repository and Status&lt;/a&gt; · &lt;a href="https://github.com/a2ui-project/a2ui/blob/main/docs/public/roadmap.md" rel="noopener noreferrer"&gt;A2UI Roadmap&lt;/a&gt; · &lt;a href="https://github.com/ag-ui-protocol/ag-ui" rel="noopener noreferrer"&gt;AG-UI Specification&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;AG-UI and A2UI address two important challenges in building modern agent-powered applications.&lt;/p&gt;

&lt;p&gt;AG-UI standardizes the interaction between agents and frontend applications through a structured, event-driven communication model.&lt;/p&gt;

&lt;p&gt;A2UI allows agents to describe dynamic interfaces using declarative JSON messages, while applications remain in control of the actual components and rendering process.&lt;/p&gt;

&lt;p&gt;Together, they make it possible to move beyond traditional chat interfaces and build applications where AI agents can communicate their progress, update application state, and generate context-aware user interfaces.&lt;/p&gt;

&lt;p&gt;For engineering teams, the opportunity is to establish reusable interaction patterns, govern agent-accessible components, maintain security boundaries, and integrate generative UI into existing design systems.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;My recommendation:&lt;/strong&gt; Start with a small React application that uses AG-UI for streaming and state synchronization, then introduce A2UI for one clearly defined dynamic interface. Evaluate rendering reliability, security, and user experience before expanding the approach.&lt;/p&gt;

&lt;p&gt;The goal should not be to make every screen dynamic.&lt;/p&gt;

&lt;p&gt;It should be to make the interface adapt when a conversation or workflow genuinely requires it.&lt;/p&gt;




&lt;h2&gt;
  
  
  References and Further Reading
&lt;/h2&gt;

&lt;h3&gt;
  
  
  AG-UI
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://docs.ag-ui.com/" rel="noopener noreferrer"&gt;AG-UI Official Documentation&lt;/a&gt; — Introduction, architecture, events, and integration.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/ag-ui-protocol/ag-ui" rel="noopener noreferrer"&gt;AG-UI GitHub Repository&lt;/a&gt; — Protocol implementation and SDKs.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/ag-ui-protocol/ag-ui/blob/main/docs/concepts/architecture.mdx" rel="noopener noreferrer"&gt;AG-UI Architecture&lt;/a&gt; — Event-driven architecture and client-server communication.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/ag-ui-protocol/ag-ui/blob/main/docs/concepts/events.mdx" rel="noopener noreferrer"&gt;AG-UI Event Reference&lt;/a&gt; — Lifecycle, messages, tools, and state events.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/ag-ui-protocol/ag-ui/tree/main/sdks/typescript" rel="noopener noreferrer"&gt;AG-UI TypeScript SDK&lt;/a&gt; — Client implementation and event handling.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  A2UI
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://a2ui.org/" rel="noopener noreferrer"&gt;Google A2UI Official Documentation&lt;/a&gt; — Project overview and developer resources.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/a2ui-project/a2ui/blob/main/specification/v0_9_1/docs/a2ui_protocol.md" rel="noopener noreferrer"&gt;A2UI Protocol v0.9.1&lt;/a&gt; — JSON protocol, surfaces, components, and data models.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/a2ui-project/a2ui/blob/main/docs/public/quickstart.md" rel="noopener noreferrer"&gt;A2UI Quickstart&lt;/a&gt; — Full-stack example and generated UI workflow.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/a2ui-project/a2ui/tree/main/renderers/react" rel="noopener noreferrer"&gt;A2UI React Renderer&lt;/a&gt; — React rendering implementation.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/a2ui-project/a2ui/blob/main/docs/public/guides/client-setup.md" rel="noopener noreferrer"&gt;A2UI Client Setup&lt;/a&gt; — Framework integration and component catalogs.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/a2ui-project/a2ui/blob/main/docs/public/guides/renderer-development.md" rel="noopener noreferrer"&gt;A2UI Renderer Development&lt;/a&gt; — Building custom renderers and integrating design systems.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/a2ui-project/a2ui/blob/main/specification/v1_0/docs/a2ui_protocol.md" rel="noopener noreferrer"&gt;A2UI v1.0 Candidate Specification&lt;/a&gt; — Upcoming protocol capabilities and changes.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Related Technologies
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://ai-sdk.dev/docs/reference/ai-sdk-ui/use-chat" rel="noopener noreferrer"&gt;Vercel AI SDK — useChat&lt;/a&gt; — Conversational streaming and frontend state.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://vercel.com/kb/guide/ai-sdk-mcp-apps" rel="noopener noreferrer"&gt;Vercel — MCP Apps Integration&lt;/a&gt; — Interactive tool UIs in AI SDK applications.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://docs.copilotkit.ai/shared-state" rel="noopener noreferrer"&gt;CopilotKit — Shared State&lt;/a&gt; — Bidirectional state synchronization between React and agents.&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>productivity</category>
      <category>programming</category>
      <category>agents</category>
    </item>
    <item>
      <title>Adaptive Concurrency Limits in Go: Dynamically Protecting Dependencies Under Load</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Sun, 01 Nov 2026 13:51:31 +0000</pubDate>
      <link>https://dev.to/serifcolakel/adaptive-concurrency-limits-in-go-dynamically-protecting-dependencies-under-load-32gf</link>
      <guid>https://dev.to/serifcolakel/adaptive-concurrency-limits-in-go-dynamically-protecting-dependencies-under-load-32gf</guid>
      <description>&lt;p&gt;Most services eventually need a concurrency limit.&lt;/p&gt;

&lt;p&gt;You might start with something simple:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;const&lt;/span&gt; &lt;span class="n"&gt;maxConcurrentRequests&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;100&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A semaphore ensures that no more than 100 requests execute simultaneously.&lt;/p&gt;

&lt;p&gt;It works. Until the conditions change.&lt;/p&gt;

&lt;p&gt;Your service gets more CPU resources after scaling. A downstream database becomes slower. A new deployment changes request processing time. Traffic patterns shift throughout the day.&lt;/p&gt;

&lt;p&gt;The limit that worked yesterday might be too restrictive today. Or worse, it might be too high and allow a dependency to become overloaded.&lt;/p&gt;

&lt;p&gt;Static concurrency limits are easy to implement, but they require continuous tuning.&lt;/p&gt;

&lt;p&gt;What if a service could adjust its concurrency limit automatically based on how the dependency behaves?&lt;/p&gt;

&lt;p&gt;That's the idea behind &lt;strong&gt;adaptive concurrency limiting&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Instead of relying entirely on a predefined number, the system observes request latency and failure signals, estimates when queueing is happening, and adjusts the number of requests it allows in flight.&lt;/p&gt;

&lt;p&gt;In this article, we'll explore:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Why fixed concurrency limits eventually become difficult to maintain&lt;/li&gt;
&lt;li&gt;The relationship between concurrency, throughput, and latency&lt;/li&gt;
&lt;li&gt;How TCP congestion control inspired adaptive concurrency algorithms&lt;/li&gt;
&lt;li&gt;How to implement a latency-aware concurrency limiter in Go&lt;/li&gt;
&lt;li&gt;How to integrate it into an HTTP client&lt;/li&gt;
&lt;li&gt;How adaptive limits interact with backpressure, bulkheads, and retry budgets&lt;/li&gt;
&lt;li&gt;What to monitor and test before using adaptive concurrency in production&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  1. The Problem with Fixed Concurrency Limits
&lt;/h2&gt;

&lt;p&gt;Imagine a Go service that calls an external market data API.&lt;/p&gt;

&lt;p&gt;You initially configure a concurrency limit of 50.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;limiter&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;NewFixedLimiter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;50&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Under normal traffic, the dependency responds in 40ms.&lt;/p&gt;

&lt;p&gt;Everything looks healthy.&lt;/p&gt;

&lt;p&gt;Then traffic increases.&lt;/p&gt;

&lt;p&gt;The dependency starts processing requests more slowly. Its response time grows to 150ms, then 300ms.&lt;/p&gt;

&lt;p&gt;Your service still allows 50 concurrent requests.&lt;/p&gt;

&lt;p&gt;The problem is that concurrency and latency are connected.&lt;/p&gt;

&lt;p&gt;When requests take longer to complete, existing requests occupy concurrency slots for longer. New requests accumulate, queues grow, and latency increases further.&lt;/p&gt;

&lt;p&gt;A fixed limit cannot automatically distinguish between:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A dependency that has additional capacity&lt;/li&gt;
&lt;li&gt;A dependency approaching saturation&lt;/li&gt;
&lt;li&gt;A dependency already experiencing queueing&lt;/li&gt;
&lt;li&gt;A dependency recovering from an overload&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can increase the limit manually, but that risks overloading the dependency.&lt;/p&gt;

&lt;p&gt;You can decrease it, but that might unnecessarily reduce throughput.&lt;/p&gt;

&lt;p&gt;The challenge is finding a concurrency level that allows high throughput without creating excessive queueing.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Why Concurrency Matters More Than RPS Alone
&lt;/h2&gt;

&lt;p&gt;Rate limiting and concurrency limiting solve different problems.&lt;/p&gt;

&lt;p&gt;A rate limiter controls how many requests can start during a period.&lt;/p&gt;

&lt;p&gt;A concurrency limiter controls how many requests can be in progress simultaneously.&lt;/p&gt;

&lt;p&gt;Consider two situations:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Request rate&lt;/th&gt;
&lt;th&gt;Average latency&lt;/th&gt;
&lt;th&gt;Approx. concurrency&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Healthy dependency&lt;/td&gt;
&lt;td&gt;500 req/s&lt;/td&gt;
&lt;td&gt;40ms&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Slow dependency&lt;/td&gt;
&lt;td&gt;500 req/s&lt;/td&gt;
&lt;td&gt;200ms&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Both scenarios have the same incoming request rate.&lt;/p&gt;

&lt;p&gt;But the second requires five times as many concurrent requests to sustain that throughput.&lt;/p&gt;

&lt;p&gt;This relationship is described by Little's Law:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
L = \lambda W&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;(L) is the average number of items in the system&lt;/li&gt;
&lt;li&gt;(\lambda) is the average arrival rate&lt;/li&gt;
&lt;li&gt;(W) is the average time spent in the system&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For request processing:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
Concurrency \approx Throughput \times Latency&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;This explains why a fixed RPS limit is not always sufficient to protect a dependency.&lt;/p&gt;

&lt;p&gt;A service can receive an acceptable number of requests per second while still accumulating too many in-flight operations because response times have increased.&lt;/p&gt;

&lt;p&gt;Adaptive concurrency focuses on this relationship.&lt;/p&gt;
&lt;h2&gt;
  
  
  3. The Idea Behind Adaptive Concurrency
&lt;/h2&gt;

&lt;p&gt;The central idea is straightforward:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Measure request latency.&lt;/li&gt;
&lt;li&gt;Maintain an estimate of normal, unloaded latency.&lt;/li&gt;
&lt;li&gt;Compare current latency against that baseline.&lt;/li&gt;
&lt;li&gt;Increase concurrency when the dependency has spare capacity.&lt;/li&gt;
&lt;li&gt;Reduce concurrency when latency indicates queueing or failures.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This resembles congestion control in TCP.&lt;/p&gt;

&lt;p&gt;TCP does not blindly send unlimited data. It maintains a congestion window and adjusts how much data can be in flight based on network conditions.&lt;/p&gt;

&lt;p&gt;Adaptive concurrency applies a similar idea to service requests.&lt;/p&gt;

&lt;p&gt;The concurrency limit becomes a dynamic window.&lt;/p&gt;


  &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdummyimage.com%2F1200x500%2F101827%2F00eaff.png%26text%3DAdaptive%2BConcurrency%253A%2BMeasure%2B%25E2%2586%2592%2BAdjust%2B%25E2%2586%2592%2BProtect" alt="Adaptive concurrency control loop" width="1200" height="500"&gt;


&lt;p&gt;There are several algorithms worth understanding.&lt;/p&gt;
&lt;h3&gt;
  
  
  AIMD
&lt;/h3&gt;

&lt;p&gt;Additive Increase, Multiplicative Decrease.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Increase the limit gradually when requests succeed.&lt;/li&gt;
&lt;li&gt;Reduce it significantly when failures or overload signals appear.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Current limit: 50

Successful samples:
50 → 51 → 52 → 53

Overload detected:
53 → 37
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;AIMD is relatively simple and works well when the system has reliable failure signals.&lt;/p&gt;

&lt;p&gt;However, it may react only after a dependency starts rejecting requests or timing out.&lt;/p&gt;

&lt;h3&gt;
  
  
  Vegas
&lt;/h3&gt;

&lt;p&gt;Vegas uses latency as an early signal of congestion.&lt;/p&gt;

&lt;p&gt;It estimates queueing using the difference between observed latency and a no-load baseline.&lt;/p&gt;

&lt;p&gt;A simplified queue estimate is:&lt;/p&gt;

&lt;p&gt;[&lt;br&gt;
Q = L \times \left(1 - \frac{RTT_{baseline}}{RTT_{current}}\right)&lt;br&gt;
]&lt;/p&gt;

&lt;p&gt;Where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;(Q) is estimated queue size&lt;/li&gt;
&lt;li&gt;(L) is the current concurrency limit&lt;/li&gt;
&lt;li&gt;(RTT_{baseline}) is the observed no-load latency&lt;/li&gt;
&lt;li&gt;(RTT_{current}) is the current measured latency&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When latency stays close to the baseline, the algorithm can increase concurrency.&lt;/p&gt;

&lt;p&gt;When latency grows significantly, it reduces the limit.&lt;/p&gt;
&lt;h3&gt;
  
  
  Gradient2
&lt;/h3&gt;

&lt;p&gt;Gradient2 compares short-term and long-term latency averages to detect changes in the latency trend.&lt;/p&gt;

&lt;p&gt;It is useful when latency baselines drift over time or workloads are bursty.&lt;/p&gt;

&lt;p&gt;Instead of relying only on a minimum latency measurement, it uses smoothed averages to identify growing queueing pressure.&lt;/p&gt;

&lt;p&gt;For a deeper implementation reference, Netflix's concurrency-limits project is worth exploring:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/Netflix/concurrency-limits" rel="noopener noreferrer"&gt;https://github.com/Netflix/concurrency-limits&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For this article, we'll build a smaller Vegas-inspired controller. It is an educational implementation, not a complete reproduction of Netflix's Vegas or Gradient2 algorithms.&lt;/p&gt;
&lt;h2&gt;
  
  
  4. Building an Adaptive Limiter in Go
&lt;/h2&gt;

&lt;p&gt;Let's implement a limiter that:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tracks in-flight requests&lt;/li&gt;
&lt;li&gt;Enforces a dynamic concurrency limit&lt;/li&gt;
&lt;li&gt;Measures request duration&lt;/li&gt;
&lt;li&gt;Collects latency samples in windows&lt;/li&gt;
&lt;li&gt;Estimates queueing using a baseline latency&lt;/li&gt;
&lt;li&gt;Increases or decreases the limit based on observed behavior&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We'll use a fail-fast admission strategy.&lt;/p&gt;

&lt;p&gt;When the current concurrency limit is reached, the limiter rejects the new request instead of allowing an unbounded queue to form.&lt;/p&gt;
&lt;h3&gt;
  
  
  Step 1: Define the limiter
&lt;/h3&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;adaptive&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"errors"&lt;/span&gt;
    &lt;span class="s"&gt;"sync"&lt;/span&gt;
    &lt;span class="s"&gt;"time"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;ErrLimitExceeded&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"adaptive concurrency limit exceeded"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Limiter&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;mu&lt;/span&gt; &lt;span class="n"&gt;sync&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Mutex&lt;/span&gt;

    &lt;span class="n"&gt;limit&lt;/span&gt;    &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;minLimit&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;maxLimit&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;

    &lt;span class="n"&gt;inflight&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;peak&lt;/span&gt;     &lt;span class="kt"&gt;int&lt;/span&gt;

    &lt;span class="n"&gt;baseline&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;

    &lt;span class="n"&gt;sampleCount&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;sampleTotal&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;
    &lt;span class="n"&gt;dropped&lt;/span&gt;     &lt;span class="kt"&gt;bool&lt;/span&gt;

    &lt;span class="n"&gt;windowSize&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;

    &lt;span class="n"&gt;alpha&lt;/span&gt; &lt;span class="kt"&gt;float64&lt;/span&gt;
    &lt;span class="n"&gt;beta&lt;/span&gt;  &lt;span class="kt"&gt;float64&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;initialLimit&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;minLimit&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;maxLimit&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;minLimit&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;initialLimit&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;minLimit&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;maxLimit&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;initialLimit&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nb"&gt;panic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"invalid concurrency limits"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;      &lt;span class="n"&gt;initialLimit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;minLimit&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;   &lt;span class="n"&gt;minLimit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;maxLimit&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;   &lt;span class="n"&gt;maxLimit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;windowSize&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;      &lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;beta&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;       &lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The limiter maintains two important pieces of information:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Current limit:&lt;/strong&gt; The maximum number of requests allowed simultaneously.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;In-flight count:&lt;/strong&gt; The number of requests currently executing.&lt;/p&gt;

&lt;p&gt;We also maintain latency statistics and a baseline.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;windowSize&lt;/code&gt; determines how many completed requests we collect before updating the concurrency limit.&lt;/p&gt;

&lt;p&gt;Using a window helps avoid changing the limit after every individual request.&lt;/p&gt;
&lt;h3&gt;
  
  
  Step 2: Acquire a permit
&lt;/h3&gt;

&lt;p&gt;The limiter needs to decide whether a new request can execute.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Acquire&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inflight&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ErrLimitExceeded&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inflight&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inflight&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;peak&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;peak&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inflight&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;startedAt&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;once&lt;/span&gt; &lt;span class="n"&gt;sync&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Once&lt;/span&gt;

    &lt;span class="n"&gt;done&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;once&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="n"&gt;elapsed&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Since&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;startedAt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

            &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

            &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inflight&lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;

            &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;recordSample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;elapsed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dropped&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The returned function releases the permit and reports the outcome.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;limiter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Acquire&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;false&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;callDependency&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the operation experiences a dependency-level overload signal, we can report that separately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;limiter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Acquire&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;isOverloadError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="p"&gt;}()&lt;/span&gt;

&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;callDependency&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;sync.Once&lt;/code&gt; protects against accidentally releasing the same permit twice.&lt;/p&gt;

&lt;p&gt;The caller must invoke the returned function exactly once in normal control flow. The &lt;code&gt;sync.Once&lt;/code&gt; makes repeated invocations harmless.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Record latency samples
&lt;/h3&gt;

&lt;p&gt;Now we need to collect completed request samples.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;recordSample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;latency&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sampleCount&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;
    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sampleTotal&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;latency&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;true&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sampleCount&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;windowSize&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;average&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sampleTotal&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sampleCount&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;updateLimit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;average&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sampleCount&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sampleTotal&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;false&lt;/span&gt;
    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;peak&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Instead of adjusting concurrency after every request, we wait until a complete sample window is collected.&lt;/p&gt;

&lt;p&gt;This reduces sensitivity to individual latency spikes.&lt;/p&gt;

&lt;p&gt;A production implementation may use more sophisticated rolling windows, exponential moving averages, or percentile-based measurements.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Implement the control algorithm
&lt;/h3&gt;

&lt;p&gt;Now comes the interesting part.&lt;/p&gt;

&lt;p&gt;We compare the current average latency against the baseline.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;updateLimit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;currentLatency&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;currentLatency&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;baseline&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;currentLatency&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;baseline&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;baseline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;currentLatency&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;decrease&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;queueEstimate&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;baseline&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;currentLatency&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

    &lt;span class="c"&gt;// Increase only when the previous window&lt;/span&gt;
    &lt;span class="c"&gt;// demonstrated meaningful concurrency usage.&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;peak&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;switch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;queueEstimate&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;alpha&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;increase&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;queueEstimate&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;beta&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;decrease&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The queue estimate is an approximation of the queueing pressure observed at the dependency.&lt;/p&gt;

&lt;p&gt;When current latency is close to the baseline, the estimated queue remains small.&lt;/p&gt;

&lt;p&gt;When current latency increases, the estimated queue grows.&lt;/p&gt;

&lt;p&gt;We use two thresholds:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;alpha&lt;/code&gt;: Below this value, the dependency appears to have spare capacity.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;beta&lt;/code&gt;: Above this value, queueing pressure is considered too high.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The gap between the thresholds creates a stability region where the limit does not change.&lt;/p&gt;

&lt;p&gt;This helps avoid unnecessary oscillation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Increase and decrease concurrency
&lt;/h3&gt;

&lt;p&gt;The final methods enforce the configured boundaries.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;increase&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;maxLimit&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;decrease&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;next&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="m"&gt;0.7&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;next&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;minLimit&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;next&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;minLimit&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;next&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The limiter uses:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Additive increase: Grow gradually.&lt;/li&gt;
&lt;li&gt;Multiplicative decrease: Reduce more aggressively when congestion appears.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is intentionally conservative.&lt;/p&gt;

&lt;p&gt;Increasing the limit by one allows the dependency to demonstrate additional capacity without creating a large traffic spike.&lt;/p&gt;

&lt;p&gt;Reducing the limit by 30% provides a faster response to overload.&lt;/p&gt;

&lt;p&gt;These values are configuration choices, not universal production defaults.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 6: Expose the current limit
&lt;/h3&gt;

&lt;p&gt;We should expose the current state for observability.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Stats&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;limit&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;inflight&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;baseline&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inflight&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;baseline&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now we can inspect the limiter without exposing its internal synchronization.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;inflight&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;baseline&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;limiter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Stats&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Printf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"adaptive limiter: limit=%d inflight=%d baseline=%s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;inflight&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;baseline&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  5. Integrating the Limiter with an HTTP Client
&lt;/h2&gt;

&lt;p&gt;Let's apply the limiter to an external dependency.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;MarketDataClient&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;httpClient&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Client&lt;/span&gt;
    &lt;span class="n"&gt;limiter&lt;/span&gt;    &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;adaptive&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;MarketDataClient&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;Fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;symbol&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;limiter&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Acquire&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;

    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;done&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dropped&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}()&lt;/span&gt;

    &lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewRequestWithContext&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MethodGet&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s"&gt;"https://api.example.com/market/"&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;symbol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="no"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;httpClient&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;isDependencyOverload&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Body&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StatusCode&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StatusTooManyRequests&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StatusCode&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;http&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StatusServiceUnavailable&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;dropped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;true&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="s"&gt;"market data dependency returned %d"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StatusCode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StatusCode&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;200&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StatusCode&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="m"&gt;300&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fmt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Errorf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="s"&gt;"unexpected status: %d"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;StatusCode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The example classifies overload responses separately from other errors.&lt;/p&gt;

&lt;p&gt;This distinction matters.&lt;/p&gt;

&lt;p&gt;A validation error, for example, should not necessarily cause the adaptive limiter to reduce its concurrency limit.&lt;/p&gt;

&lt;p&gt;Likewise, a network timeout might be an overload signal in one system but a network infrastructure issue in another.&lt;/p&gt;

&lt;p&gt;The failure classifier should reflect the dependency's actual behavior.&lt;/p&gt;

&lt;p&gt;The HTTP client should also have appropriate timeouts and connection settings. An adaptive concurrency limiter is not a replacement for transport-level protection.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. What Happens During an Overload?
&lt;/h2&gt;

&lt;p&gt;Let's walk through a simplified scenario.&lt;/p&gt;

&lt;p&gt;Assume the limiter starts with a concurrency limit of 50.&lt;/p&gt;

&lt;p&gt;The dependency initially responds in 40ms.&lt;/p&gt;

&lt;p&gt;As traffic increases, the limiter observes the following:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Average latency&lt;/th&gt;
&lt;th&gt;Concurrency limit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Initial&lt;/td&gt;
&lt;td&gt;40ms&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Increased traffic&lt;/td&gt;
&lt;td&gt;45ms&lt;/td&gt;
&lt;td&gt;51&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Higher utilization&lt;/td&gt;
&lt;td&gt;55ms&lt;/td&gt;
&lt;td&gt;52&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Queueing begins&lt;/td&gt;
&lt;td&gt;120ms&lt;/td&gt;
&lt;td&gt;36&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dependency recovers&lt;/td&gt;
&lt;td&gt;45ms&lt;/td&gt;
&lt;td&gt;Gradually increases&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The important behavior is not the exact numbers.&lt;/p&gt;

&lt;p&gt;The limiter detects that latency is increasing relative to the baseline and reduces the amount of work it allows in flight.&lt;/p&gt;

&lt;p&gt;As the dependency recovers, the limiter can gradually explore additional capacity again.&lt;/p&gt;

&lt;p&gt;This is a feedback control loop.&lt;/p&gt;

&lt;p&gt;However, it is important to understand that the limiter does not instantly discover the perfect concurrency limit. It continuously estimates a useful operating point based on observed signals.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Adaptive Concurrency vs. Other Resilience Patterns
&lt;/h2&gt;

&lt;p&gt;Adaptive concurrency is not a replacement for the patterns we've discussed in previous articles.&lt;/p&gt;

&lt;p&gt;It complements them.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mechanism&lt;/th&gt;
&lt;th&gt;Responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Backpressure&lt;/td&gt;
&lt;td&gt;Controls incoming work&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Load shedding&lt;/td&gt;
&lt;td&gt;Rejects work when capacity is exceeded&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bulkhead&lt;/td&gt;
&lt;td&gt;Isolates resources and dependencies&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retry budget&lt;/td&gt;
&lt;td&gt;Limits additional retry traffic&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Adaptive concurrency&lt;/td&gt;
&lt;td&gt;Adjusts the concurrency ceiling based on observed conditions&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Consider a service that communicates with three dependencies:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Market Data API&lt;/li&gt;
&lt;li&gt;Portfolio Database&lt;/li&gt;
&lt;li&gt;Reporting Service&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each dependency has different latency characteristics and capacity.&lt;/p&gt;

&lt;p&gt;Using one global concurrency limit would couple their behavior.&lt;/p&gt;

&lt;p&gt;A better approach is to create an independent adaptive limiter for each dependency.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;DependencyLimiters&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;MarketData&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;adaptive&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;
    &lt;span class="n"&gt;Portfolio&lt;/span&gt;  &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;adaptive&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;
    &lt;span class="n"&gt;Reporting&lt;/span&gt;  &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;adaptive&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Limiter&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This gives each dependency its own feedback loop.&lt;/p&gt;

&lt;p&gt;If the reporting service becomes slow, its concurrency limit can decrease without automatically reducing the capacity available to market data requests.&lt;/p&gt;

&lt;p&gt;This is where adaptive concurrency and bulkheads work particularly well together.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Important Production Considerations
&lt;/h2&gt;

&lt;p&gt;The example implementation demonstrates the core idea, but a production adaptive limiter needs additional safeguards.&lt;/p&gt;

&lt;h3&gt;
  
  
  Baseline latency can become stale
&lt;/h3&gt;

&lt;p&gt;A latency baseline is only useful when it represents a reasonably unloaded system.&lt;/p&gt;

&lt;p&gt;If the baseline is established while the dependency is already overloaded, the algorithm may incorrectly treat unhealthy latency as normal.&lt;/p&gt;

&lt;p&gt;Production implementations should carefully manage baseline updates and resets.&lt;/p&gt;

&lt;h3&gt;
  
  
  Traffic must be representative
&lt;/h3&gt;

&lt;p&gt;A dependency may expose different latency characteristics for different operations.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Reading a cached market price&lt;/li&gt;
&lt;li&gt;Fetching historical prices&lt;/li&gt;
&lt;li&gt;Calculating a portfolio&lt;/li&gt;
&lt;li&gt;Executing a database aggregation&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Combining all these operations into one latency signal can produce misleading results.&lt;/p&gt;

&lt;p&gt;Consider separate limiters or traffic partitions when workloads have significantly different behavior.&lt;/p&gt;

&lt;h3&gt;
  
  
  The algorithm needs enough samples
&lt;/h3&gt;

&lt;p&gt;Adaptive control depends on observations.&lt;/p&gt;

&lt;p&gt;A low-traffic service might not generate enough samples to update its limit reliably.&lt;/p&gt;

&lt;p&gt;A new instance also needs a warm-up strategy.&lt;/p&gt;

&lt;p&gt;Slow start, conservative initial limits, and minimum sample requirements can help.&lt;/p&gt;

&lt;h3&gt;
  
  
  Avoid uncontrolled oscillation
&lt;/h3&gt;

&lt;p&gt;A limiter that constantly increases and decreases its capacity can introduce instability.&lt;/p&gt;

&lt;p&gt;Useful techniques include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Sampling windows&lt;/li&gt;
&lt;li&gt;Exponential smoothing&lt;/li&gt;
&lt;li&gt;Separate increase and decrease thresholds&lt;/li&gt;
&lt;li&gt;Bounded increase and decrease rates&lt;/li&gt;
&lt;li&gt;Minimum and maximum concurrency limits&lt;/li&gt;
&lt;li&gt;Jitter when multiple independent clients share a dependency&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Local limits multiply across replicas
&lt;/h3&gt;

&lt;p&gt;This is especially important in distributed deployments.&lt;/p&gt;

&lt;p&gt;Imagine ten replicas, each with an adaptive concurrency limit of 100.&lt;/p&gt;

&lt;p&gt;The dependency may receive up to 1,000 concurrent requests.&lt;/p&gt;

&lt;p&gt;Each instance is making decisions based on its own observations. The aggregate behavior is not automatically coordinated.&lt;/p&gt;

&lt;p&gt;Local adaptive concurrency is often sufficient for protecting individual instances and reacting to local conditions.&lt;/p&gt;

&lt;p&gt;But strict global dependency quotas may require a shared dispatcher, centralized coordination, or provider-side quota enforcement.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do not confuse concurrency with connection limits
&lt;/h3&gt;

&lt;p&gt;An HTTP transport's connection pool and an application's concurrency limiter protect different resources.&lt;/p&gt;

&lt;p&gt;A request may hold an application permit while waiting for a connection. Conversely, HTTP/2 can multiplex multiple requests over one connection.&lt;/p&gt;

&lt;p&gt;Configure both based on the actual workload.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Testing Adaptive Concurrency
&lt;/h2&gt;

&lt;p&gt;Adaptive algorithms should be tested as control systems, not only as ordinary utility functions.&lt;/p&gt;

&lt;p&gt;At minimum, test these behaviors:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The initial concurrency limit is respected.&lt;/li&gt;
&lt;li&gt;Requests are rejected when the current limit is reached.&lt;/li&gt;
&lt;li&gt;Permits are released exactly once.&lt;/li&gt;
&lt;li&gt;Successful low-latency windows allow gradual growth.&lt;/li&gt;
&lt;li&gt;Rising latency causes the limit to decrease.&lt;/li&gt;
&lt;li&gt;Overload signals trigger a decrease.&lt;/li&gt;
&lt;li&gt;The limit never exceeds its configured maximum.&lt;/li&gt;
&lt;li&gt;The limit never falls below its configured minimum.&lt;/li&gt;
&lt;li&gt;Concurrent acquisitions are race-safe.&lt;/li&gt;
&lt;li&gt;Insufficient sample windows do not trigger premature changes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Run the race detector:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go &lt;span class="nb"&gt;test&lt;/span&gt; &lt;span class="nt"&gt;-race&lt;/span&gt; ./...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Also consider deterministic algorithm tests.&lt;/p&gt;

&lt;p&gt;Instead of using real HTTP requests and sleeping for arbitrary durations, separate the control algorithm from the admission mechanism.&lt;/p&gt;

&lt;p&gt;Feed known latency samples into the controller and verify the resulting limit.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Baseline: 40ms

Window 1: 42ms → increase
Window 2: 45ms → increase
Window 3: 120ms → decrease
Window 4: 130ms → decrease
Window 5: 45ms → gradual recovery
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This makes algorithm behavior easier to reproduce and debug.&lt;/p&gt;

&lt;p&gt;Finally, run load tests with changing dependency latency.&lt;/p&gt;

&lt;p&gt;A useful test should simulate:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A healthy dependency.&lt;/li&gt;
&lt;li&gt;Gradually increasing latency.&lt;/li&gt;
&lt;li&gt;Sudden overload.&lt;/li&gt;
&lt;li&gt;Partial recovery.&lt;/li&gt;
&lt;li&gt;A complete dependency outage.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Measure throughput, p95/p99 latency, rejection rate, and concurrency limit changes.&lt;/p&gt;

&lt;p&gt;The goal is not simply to maximize throughput.&lt;/p&gt;

&lt;p&gt;The goal is to maintain useful throughput without allowing latency and queueing to grow uncontrollably.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Observability
&lt;/h2&gt;

&lt;p&gt;An adaptive limiter introduces a new operational component.&lt;/p&gt;

&lt;p&gt;You need visibility into how it makes decisions.&lt;/p&gt;

&lt;p&gt;Useful metrics include:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adaptive_concurrency_limit&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Gauge&lt;/td&gt;
&lt;td&gt;Current concurrency ceiling&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adaptive_inflight_requests&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Gauge&lt;/td&gt;
&lt;td&gt;Active requests&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adaptive_request_latency_seconds&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Histogram&lt;/td&gt;
&lt;td&gt;Observed dependency latency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adaptive_rejected_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Requests rejected at admission&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adaptive_limit_increase_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Limit increase events&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adaptive_limit_decrease_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Limit decrease events&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;adaptive_baseline_latency_seconds&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Gauge&lt;/td&gt;
&lt;td&gt;Estimated unloaded latency&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These metrics help answer important questions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Is the limiter constantly operating at its maximum?&lt;/li&gt;
&lt;li&gt;Is the concurrency limit frequently decreasing?&lt;/li&gt;
&lt;li&gt;Is the dependency latency increasing?&lt;/li&gt;
&lt;li&gt;Are requests being rejected before reaching the dependency?&lt;/li&gt;
&lt;li&gt;Is the limiter recovering too slowly?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A particularly useful dashboard plots three signals together:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Current concurrency limit&lt;/li&gt;
&lt;li&gt;In-flight requests&lt;/li&gt;
&lt;li&gt;Dependency latency&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Looking at them independently can hide the relationship between capacity and latency.&lt;/p&gt;

&lt;p&gt;Avoid high-cardinality labels such as raw URLs, user IDs, and request IDs.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. When Should You Use Adaptive Concurrency?
&lt;/h2&gt;

&lt;p&gt;Adaptive concurrency is not necessary for every service.&lt;/p&gt;

&lt;p&gt;A fixed limit is often the better choice when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Dependency capacity is stable and well understood.&lt;/li&gt;
&lt;li&gt;Traffic patterns are predictable.&lt;/li&gt;
&lt;li&gt;The service has low request volume.&lt;/li&gt;
&lt;li&gt;The cost of an adaptive control loop is not justified.&lt;/li&gt;
&lt;li&gt;A simple semaphore already provides sufficient protection.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Adaptive concurrency becomes more attractive when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Dependency capacity changes frequently.&lt;/li&gt;
&lt;li&gt;Traffic is bursty or unpredictable.&lt;/li&gt;
&lt;li&gt;Services scale horizontally.&lt;/li&gt;
&lt;li&gt;Latency is a reliable indicator of queueing.&lt;/li&gt;
&lt;li&gt;Manual concurrency tuning is difficult.&lt;/li&gt;
&lt;li&gt;You need to protect dependencies before they start rejecting requests.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Start with a fixed limit if it is sufficient. Introduce adaptive behavior when operational evidence shows that static configuration is becoming a limitation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;A fixed concurrency limit assumes that system capacity is relatively predictable.&lt;/p&gt;

&lt;p&gt;In distributed systems, that assumption does not always hold.&lt;/p&gt;

&lt;p&gt;Dependencies slow down, traffic changes, instances scale, and resource availability evolves.&lt;/p&gt;

&lt;p&gt;Adaptive concurrency provides a feedback loop that adjusts the amount of work allowed in flight based on observed latency and failure signals.&lt;/p&gt;

&lt;p&gt;In Go, the core mechanism is straightforward: track active requests, collect latency samples, estimate congestion, and adjust the concurrency ceiling.&lt;/p&gt;

&lt;p&gt;The difficult part is not implementing a semaphore.&lt;/p&gt;

&lt;p&gt;It is building a stable controller that reacts to real overload without constantly oscillating or making incorrect decisions from noisy measurements.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Don't just limit how much work your service can handle. Let it learn when it should slow down.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>go</category>
      <category>backend</category>
      <category>productivity</category>
      <category>beginners</category>
    </item>
    <item>
      <title>A2A (Agent2Agent Protocol): Building Interoperable Multi-Agent Systems</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Fri, 30 Oct 2026 18:53:51 +0000</pubDate>
      <link>https://dev.to/serifcolakel/a2a-agent2agent-protocol-building-interoperable-multi-agent-systems-1nnn</link>
      <guid>https://dev.to/serifcolakel/a2a-agent2agent-protocol-building-interoperable-multi-agent-systems-1nnn</guid>
      <description>&lt;p&gt;&lt;em&gt;Understanding how AI agents discover, communicate, delegate tasks, and collaborate across different frameworks and platforms.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  1. What Is A2A, and What Problem Does It Solve?
&lt;/h2&gt;

&lt;p&gt;As AI applications become more capable, we are moving beyond simple interactions between a user and a single AI model.&lt;/p&gt;

&lt;p&gt;Today, an AI agent can search documentation, query databases, interact with APIs, analyze code, and execute multi-step workflows. However, as these capabilities grow, a new architectural question emerges:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How can independently developed AI agents communicate and work together?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Consider an application that needs to analyze a GitLab merge request. The task may involve several different responsibilities:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A security agent reviews potential vulnerabilities.&lt;/li&gt;
&lt;li&gt;A testing agent analyzes test coverage and identifies missing cases.&lt;/li&gt;
&lt;li&gt;An architecture agent evaluates design decisions and code structure.&lt;/li&gt;
&lt;li&gt;A review agent combines the findings and prepares a final report.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each agent might use a different framework, model provider, toolset, or programming language.&lt;/p&gt;

&lt;p&gt;Without a standardized communication mechanism, integrating these agents usually requires custom APIs, shared data contracts, and application-specific orchestration logic.&lt;/p&gt;

&lt;p&gt;This is where &lt;strong&gt;Agent2Agent (A2A) Protocol&lt;/strong&gt; becomes relevant.&lt;/p&gt;

&lt;p&gt;A2A is an open protocol designed to enable communication and collaboration between independent AI agent systems. It provides a common interaction model regardless of the framework, programming language, or vendor used to build an agent.&lt;/p&gt;

&lt;p&gt;The protocol was originally developed by Google and contributed to the Linux Foundation. It is designed around interoperability without requiring agents to expose their internal reasoning, memory, tools, or implementation details.&lt;/p&gt;

&lt;p&gt;The important distinction is that A2A does not define how an agent thinks or performs its internal tasks. It defines how one agent can communicate with another.&lt;/p&gt;

&lt;p&gt;In other words, an agent can be implemented using LangGraph, Google ADK, CrewAI, Semantic Kernel, or a completely custom solution and still communicate with another compatible agent through A2A.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://a2a-protocol.org/latest/" rel="noopener noreferrer"&gt;A2A Protocol — Official Documentation&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Why Do We Need A2A?
&lt;/h2&gt;

&lt;p&gt;Most agent frameworks provide mechanisms for calling tools, delegating work, or coordinating multiple agents.&lt;/p&gt;

&lt;p&gt;However, these mechanisms are generally designed around the framework's own execution model.&lt;/p&gt;

&lt;p&gt;For example, an agent built with one framework may have its own task execution logic, state management, and internal tool definitions. Connecting it directly to an agent developed using another framework can require custom integration work.&lt;/p&gt;

&lt;p&gt;A2A addresses the interoperability problem.&lt;/p&gt;

&lt;p&gt;Instead of requiring agents to understand each other's internal implementations, they communicate through a standardized protocol.&lt;/p&gt;

&lt;p&gt;This gives us several important capabilities:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Discover agents and understand their capabilities.&lt;/li&gt;
&lt;li&gt;Delegate tasks to remote agents.&lt;/li&gt;
&lt;li&gt;Exchange messages and structured data.&lt;/li&gt;
&lt;li&gt;Track the lifecycle of long-running tasks.&lt;/li&gt;
&lt;li&gt;Receive progress updates through streaming.&lt;/li&gt;
&lt;li&gt;Exchange files and generated artifacts.&lt;/li&gt;
&lt;li&gt;Support authentication and authorization across agent boundaries.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A2A is particularly useful when agents belong to different teams, applications, organizations, or technology stacks.&lt;/p&gt;

&lt;p&gt;The goal is not to turn every agent into the same implementation. The goal is to allow different implementations to collaborate.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. A2A vs. MCP vs. ACP
&lt;/h2&gt;

&lt;p&gt;These three protocols are often discussed together, but they solve different architectural problems.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Technology&lt;/th&gt;
&lt;th&gt;Main responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;MCP&lt;/td&gt;
&lt;td&gt;Agent ↔ Tools and external context&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ACP&lt;/td&gt;
&lt;td&gt;Coding Agent ↔ Client Application&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A2A&lt;/td&gt;
&lt;td&gt;Agent ↔ Agent&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Let's consider a practical example.&lt;/p&gt;

&lt;p&gt;Imagine a research agent that needs to investigate a topic, retrieve information from internal documentation, and delegate a part of its work to a specialized analysis agent.&lt;/p&gt;

&lt;p&gt;In this scenario:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;MCP provides access to documentation, APIs, databases, and other tools.&lt;/li&gt;
&lt;li&gt;A2A allows the research agent to communicate with the analysis agent.&lt;/li&gt;
&lt;li&gt;ACP could be used separately when integrating a coding agent with an editor or another client application.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A simplified architecture:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                   User
                     |
                     v
              Main AI Application
                     |
                     v
               Coordinator Agent
                 /           \
                /             \
             A2A               A2A
              /                 \
             v                   v
      Research Agent       Analysis Agent
             |                   |
            MCP                 MCP
             |                   |
       Internal APIs        Database / APIs
             |                   |
             v                   v
          Results             Results
                \             /
                 \           /
                  Coordinator
                       |
                       v
                 Final Response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each agent can use its own tools and internal architecture. A2A provides the communication layer between them.&lt;/p&gt;

&lt;p&gt;One important point: A2A is not a replacement for an agent framework or an internal sub-agent mechanism. It focuses on communication between independent agent applications. Internal orchestration can still be handled by the framework that runs the agent.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://a2a-protocol.org/latest/" rel="noopener noreferrer"&gt;A2A and MCP&lt;/a&gt; · &lt;a href="https://modelcontextprotocol.io/" rel="noopener noreferrer"&gt;MCP Documentation&lt;/a&gt; · &lt;a href="https://agentclientprotocol.com/" rel="noopener noreferrer"&gt;ACP Documentation&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  4. How Does A2A Work?
&lt;/h2&gt;

&lt;p&gt;To understand A2A, we need to understand a few core concepts.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.1. Agent Card: Discovering Other Agents
&lt;/h3&gt;

&lt;p&gt;One of the most important concepts in A2A is the Agent Card.&lt;/p&gt;

&lt;p&gt;An Agent Card is a JSON document that describes an agent and its capabilities.&lt;/p&gt;

&lt;p&gt;It acts as a machine-readable description of what an agent can do and how another agent can communicate with it.&lt;/p&gt;

&lt;p&gt;An Agent Card can contain:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agent name and description&lt;/li&gt;
&lt;li&gt;Service endpoint&lt;/li&gt;
&lt;li&gt;Supported protocol versions and transports&lt;/li&gt;
&lt;li&gt;Available capabilities&lt;/li&gt;
&lt;li&gt;Supported input and output formats&lt;/li&gt;
&lt;li&gt;Skills and example tasks&lt;/li&gt;
&lt;li&gt;Authentication requirements&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;By default, an A2A server exposes its Agent Card at:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://agent.example.com/.well-known/agent-card.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For example, a research agent might expose the following description:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Research Agent"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"An agent specialized in research and source analysis."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"version"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1.0.0"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"supportedInterfaces"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://research.example.com/a2a"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"protocolBinding"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"JSONRPC"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"protocolVersion"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1.0"&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"capabilities"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"streaming"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"pushNotifications"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"extensions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"defaultInputModes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"text/plain"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"defaultOutputModes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"text/plain"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"skills"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"market-research"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Market Research"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Researches topics and produces structured findings."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"tags"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"research"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"analysis"&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"examples"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"Research the latest developments in AI agents."&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"inputModes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"text/plain"&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="nl"&gt;"outputModes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"text/plain"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
        &lt;/span&gt;&lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a simplified public Agent Card example. Authentication and additional security requirements can also be described through the protocol's security fields.&lt;/p&gt;

&lt;p&gt;A client can retrieve this document and understand whether the remote agent is suitable for a particular task.&lt;/p&gt;

&lt;p&gt;For example, a coordinator agent looking for an agent capable of analyzing financial documents can inspect registered Agent Cards and evaluate their declared skills.&lt;/p&gt;

&lt;p&gt;Agent discovery can happen through:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Well-known URLs&lt;/li&gt;
&lt;li&gt;Private or public agent registries&lt;/li&gt;
&lt;li&gt;Directly configured Agent Card URLs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An Agent Card describes an agent's advertised capabilities; it does not prove that the agent can successfully perform every advertised task. Capability selection should be combined with authorization, availability checks, and runtime evaluation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://a2a-protocol.org/latest/topics/agent-discovery/" rel="noopener noreferrer"&gt;A2A — Agent Discovery&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  4.2. Messages and Tasks
&lt;/h3&gt;

&lt;p&gt;A2A distinguishes between a message and a task.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Message&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A message represents communication between a client and an agent. It can contain instructions, clarification requests, or additional information.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Task&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A task represents a stateful unit of work with its own lifecycle.&lt;/p&gt;

&lt;p&gt;For example, when an agent receives a request to analyze a large document, it may need several minutes to complete the work. Instead of treating the interaction as a simple request-response operation, A2A allows the agent to create a task and report its progress.&lt;/p&gt;

&lt;p&gt;A task has a lifecycle that can include:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Submitted&lt;/td&gt;
&lt;td&gt;Task has been accepted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Working&lt;/td&gt;
&lt;td&gt;Agent is processing the task&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Input Required&lt;/td&gt;
&lt;td&gt;Additional information is needed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Auth Required&lt;/td&gt;
&lt;td&gt;Additional authentication is required&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Completed&lt;/td&gt;
&lt;td&gt;Task finished successfully&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failed&lt;/td&gt;
&lt;td&gt;Task execution failed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Canceled&lt;/td&gt;
&lt;td&gt;Task was canceled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rejected&lt;/td&gt;
&lt;td&gt;Agent declined the task&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Some states represent interruptions that can be resolved through further interaction, while others represent terminal states.&lt;/p&gt;

&lt;p&gt;This is important for long-running workflows because the client does not need to keep a single request open until every operation is complete.&lt;/p&gt;

&lt;p&gt;A2A also provides two useful identifiers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;taskId&lt;/code&gt;: Identifies an individual task.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;contextId&lt;/code&gt;: Groups related messages and tasks within a broader interaction context.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The task identifier is generated by the agent server. The context identifier helps maintain continuity across multiple related interactions.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.3. Artifacts: Sharing Task Results
&lt;/h3&gt;

&lt;p&gt;A2A also introduces the concept of Artifacts.&lt;/p&gt;

&lt;p&gt;An artifact represents the output produced by an agent while completing a task.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A research agent can produce a research report.&lt;/li&gt;
&lt;li&gt;A coding agent can return a generated file.&lt;/li&gt;
&lt;li&gt;An analysis agent can return structured JSON.&lt;/li&gt;
&lt;li&gt;A document processing agent can return an extracted document.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An artifact can contain multiple parts, including text, binary content, URLs, or structured data.&lt;/p&gt;

&lt;p&gt;This distinction is important:&lt;/p&gt;

&lt;p&gt;Messages are primarily used for communication and interaction, while artifacts represent the actual outputs of a task.&lt;/p&gt;

&lt;p&gt;It allows agents to exchange more than just text-based responses.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.4. Streaming and Asynchronous Communication
&lt;/h3&gt;

&lt;p&gt;Not every agent task finishes immediately.&lt;/p&gt;

&lt;p&gt;A research process, document analysis, or code review can take a considerable amount of time.&lt;/p&gt;

&lt;p&gt;A2A supports multiple ways of receiving task updates:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Streaming&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The client can subscribe to task updates and receive events while the agent is working.&lt;/p&gt;

&lt;p&gt;Depending on the protocol binding, streaming is supported through Server-Sent Events (SSE) or gRPC streaming.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Polling&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The client can periodically request the current task state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Push Notifications&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For long-running tasks where maintaining a persistent connection is not practical, the agent can send updates to a client-provided webhook.&lt;/p&gt;

&lt;p&gt;These mechanisms allow applications to support different execution patterns without inventing custom communication protocols for each agent.&lt;/p&gt;

&lt;p&gt;A2A also defines operations for sending messages, retrieving tasks, canceling tasks, subscribing to updates, and managing push notification configurations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://a2a-protocol.org/latest/specification/" rel="noopener noreferrer"&gt;A2A Specification&lt;/a&gt; · &lt;a href="https://a2a-protocol.org/latest/topics/life-of-a-task/" rel="noopener noreferrer"&gt;Task Lifecycle&lt;/a&gt; · &lt;a href="https://a2a-protocol.org/latest/topics/streaming-and-async/" rel="noopener noreferrer"&gt;Streaming and Asynchronous Operations&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  5. How to Integrate A2A?
&lt;/h2&gt;

&lt;p&gt;A2A provides official SDKs for several programming languages, including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Python&lt;/li&gt;
&lt;li&gt;JavaScript / TypeScript&lt;/li&gt;
&lt;li&gt;Java&lt;/li&gt;
&lt;li&gt;Go&lt;/li&gt;
&lt;li&gt;C# / .NET&lt;/li&gt;
&lt;li&gt;Rust&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For TypeScript applications, the official package is &lt;code&gt;@a2a-js/sdk&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The current SDK implements the A2A v1.0 specification and supports client and server development.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Install the package
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm add @a2a-js/sdk
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The official JavaScript SDK requires Node.js 20 or later.&lt;/p&gt;

&lt;p&gt;For this example, we will implement a client that communicates with an existing remote research agent.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Connect to an A2A agent
&lt;/h3&gt;

&lt;p&gt;The SDK provides &lt;code&gt;ClientFactory&lt;/code&gt; to create a client.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;ClientFactory&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@a2a-js/sdk/client&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;factory&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ClientFactory&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;factory&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createFromUrl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://research-agent.example.com&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The SDK retrieves the Agent Card and selects a compatible transport based on the agent's advertised interfaces and the client's supported transports.&lt;/p&gt;

&lt;p&gt;This means we do not need to hardcode every protocol detail into our application.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Send a message to the agent
&lt;/h3&gt;

&lt;p&gt;Once the connection is established, we can send a task request.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;Role&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;Message&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;SendMessageRequest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@a2a-js/sdk&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Message&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;messageId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
  &lt;span class="na"&gt;role&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Role&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ROLE_USER&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;content&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="na"&gt;$case&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;text&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="na"&gt;value&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Research the latest developments in multi-agent systems.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
      &lt;span class="na"&gt;mediaType&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;text/plain&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;],&lt;/span&gt;
  &lt;span class="na"&gt;taskId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;contextId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;extensions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt;
  &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{},&lt;/span&gt;
  &lt;span class="na"&gt;referenceTaskIds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;request&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;SendMessageRequest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;tenant&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;message&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;configuration&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{},&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendMessage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;request&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The example uses the SDK's generated TypeScript message types. Depending on the agent's behavior, the response can contain a message or a task.&lt;/p&gt;

&lt;p&gt;For production integrations, authentication and request timeouts should be configured as well.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendMessage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;serviceParameters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;Authorization&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`Bearer &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;A2A_TOKEN&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="na"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;AbortSignal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="nx"&gt;_000&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The client can now communicate with a remote agent without needing to know its internal framework or model implementation.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Stream task updates
&lt;/h3&gt;

&lt;p&gt;For long-running tasks, we can use &lt;code&gt;sendMessageStream&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;stream&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sendMessageStream&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;request&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="k"&gt;await &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;stream&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// Handle task events:&lt;/span&gt;
  &lt;span class="c1"&gt;// - task&lt;/span&gt;
  &lt;span class="c1"&gt;// - statusUpdate&lt;/span&gt;
  &lt;span class="c1"&gt;// - artifactUpdate&lt;/span&gt;

  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The actual event union should be handled using the SDK's generated types.&lt;/p&gt;

&lt;p&gt;This allows us to display task progress, process generated artifacts, and update the application as the remote agent works.&lt;/p&gt;

&lt;p&gt;The SDK also supports task retrieval, cancellation, and push notification configuration.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Build your own A2A agent
&lt;/h3&gt;

&lt;p&gt;A2A is not limited to consuming existing agents. We can also expose our own agent through the protocol.&lt;/p&gt;

&lt;p&gt;The official JavaScript SDK separates server-side responsibilities into three main components:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AgentExecutor&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Contains the agent's business logic. It receives the request context, executes the task, and publishes messages, status updates, and artifacts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DefaultRequestHandler&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Handles protocol-level responsibilities such as message routing, task storage, cancellation, and push notifications.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Transport Adapters&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Expose the agent through supported communication protocols.&lt;/p&gt;

&lt;p&gt;The SDK provides adapters for JSON-RPC and HTTP+JSON/REST through Express, as well as gRPC for Node.js.&lt;/p&gt;

&lt;p&gt;The architecture looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Incoming A2A Request
        |
        v
 Transport Adapter
        |
        v
DefaultRequestHandler
        |
        +----&amp;gt; Task Store
        |
        +----&amp;gt; Execution Event Bus
        |
        v
   AgentExecutor
        |
        v
   Agent Logic
        |
        +----&amp;gt; LLM
        +----&amp;gt; MCP Tools
        +----&amp;gt; Internal APIs
        +----&amp;gt; Business Logic
        |
        v
 Messages / Status / Artifacts
        |
        v
   A2A Response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The SDK also supports custom task stores and event buses, which allows us to integrate persistent databases, caches, or message brokers instead of relying exclusively on in-memory implementations.&lt;/p&gt;

&lt;p&gt;For production, this separation is particularly useful because protocol handling, task execution, and persistence can evolve independently.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://github.com/a2aproject/a2a-js" rel="noopener noreferrer"&gt;Official A2A JavaScript SDK&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Integrating A2A with the Vercel AI SDK
&lt;/h2&gt;

&lt;p&gt;There is another interesting integration worth looking at.&lt;/p&gt;

&lt;p&gt;The Vercel AI SDK lists an A2A community provider called &lt;code&gt;a2a-ai-provider&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This package allows an A2A-compatible agent to be consumed through familiar AI SDK functions such as &lt;code&gt;generateText&lt;/code&gt; and &lt;code&gt;streamText&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This is conceptually similar to the ACP provider we discussed previously.&lt;/p&gt;

&lt;p&gt;Instead of implementing every A2A interaction manually, we can use the community provider as an adapter between the A2A protocol and the AI SDK.&lt;/p&gt;

&lt;h3&gt;
  
  
  Installation
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm add ai a2a-ai-provider
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Provider configuration
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;a2a&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;a2a-ai-provider&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;generateText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ai&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateText&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;a2a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://research-agent.example.com/.well-known/agent-card.json&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
  &lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="na"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Research the latest developments in agentic AI.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;text&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Streaming responses
&lt;/h3&gt;

&lt;p&gt;We can also use &lt;code&gt;streamText&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;a2a&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;a2a-ai-provider&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;streamText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ai&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;streamText&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;a2a&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://research-agent.example.com/.well-known/agent-card.json&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;
  &lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="na"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;Analyze the current multi-agent ecosystem.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;providerOptions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;a2a&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;contextId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;consumeStream&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;content&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here, &lt;code&gt;contextId&lt;/code&gt; is used to associate related interactions with the same context.&lt;/p&gt;

&lt;p&gt;This integration makes it easier to use A2A agents in existing AI SDK applications.&lt;/p&gt;

&lt;p&gt;However, there is an important distinction:&lt;/p&gt;

&lt;p&gt;The official A2A SDK is the protocol implementation, whereas &lt;code&gt;a2a-ai-provider&lt;/code&gt; is a community-maintained adapter.&lt;/p&gt;

&lt;p&gt;The community provider's documentation describes it as under active development, and its GitHub repository currently labels it alpha and advises against production workloads.&lt;/p&gt;

&lt;p&gt;For that reason, I would use it for experimentation and evaluate the official SDK directly for production-critical integrations until the provider's maturity and compatibility are established.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://ai-sdk.dev/providers/community-providers/a2a" rel="noopener noreferrer"&gt;AI SDK — A2A Community Provider&lt;/a&gt; · &lt;a href="https://github.com/dracoblue/a2a-ai-provider" rel="noopener noreferrer"&gt;a2a-ai-provider GitHub&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Supported Communication Protocols
&lt;/h2&gt;

&lt;p&gt;A2A separates its common data model and operations from the transport used to communicate.&lt;/p&gt;

&lt;p&gt;The current specification defines three standard protocol bindings:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Protocol&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;JSON-RPC&lt;/td&gt;
&lt;td&gt;JSON-based remote procedure calls&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP+JSON / REST&lt;/td&gt;
&lt;td&gt;Resource-oriented HTTP communication&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gRPC&lt;/td&gt;
&lt;td&gt;RPC communication using Protocol Buffers and gRPC&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;An agent can support more than one binding.&lt;/p&gt;

&lt;p&gt;For example, a service could expose JSON-RPC and REST interfaces using the same underlying task execution logic.&lt;/p&gt;

&lt;p&gt;This separation is valuable because clients and servers can select a compatible transport without changing the fundamental meaning of tasks, messages, and artifacts.&lt;/p&gt;

&lt;p&gt;The official JavaScript SDK supports all three transports, with its gRPC integration requiring a Node.js environment and additional dependencies.&lt;/p&gt;

&lt;p&gt;A2A also defines protocol versioning. Clients and agents use the &lt;code&gt;A2A-Version&lt;/code&gt; header to identify the version they are communicating with.&lt;/p&gt;

&lt;p&gt;The current v1.0 specification includes changes from the earlier v0.3 interface, so compatibility and migration should be considered when connecting existing implementations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://a2a-protocol.org/v1.0.0/specification/" rel="noopener noreferrer"&gt;A2A Protocol Specification&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Advantages of A2A
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Interoperability across frameworks
&lt;/h3&gt;

&lt;p&gt;One of A2A's biggest advantages is the ability to connect agents built using different frameworks and programming languages.&lt;/p&gt;

&lt;p&gt;We do not have to standardize every team's internal agent implementation just to enable collaboration.&lt;/p&gt;

&lt;p&gt;A Python-based research agent can communicate with a TypeScript-based coordinator, while another agent may be implemented using Go or .NET.&lt;/p&gt;

&lt;p&gt;This reduces framework-level coupling.&lt;/p&gt;

&lt;h3&gt;
  
  
  Independent agent implementations
&lt;/h3&gt;

&lt;p&gt;Agents do not need to expose their internal prompts, model configurations, memory, or tools.&lt;/p&gt;

&lt;p&gt;They only need to expose their supported capabilities and implement the agreed communication contract.&lt;/p&gt;

&lt;p&gt;This allows teams to evolve their internal implementations without forcing all consumers to change their integration code.&lt;/p&gt;

&lt;h3&gt;
  
  
  Better support for long-running workflows
&lt;/h3&gt;

&lt;p&gt;A2A is designed around stateful tasks.&lt;/p&gt;

&lt;p&gt;Instead of implementing custom job tracking, status polling, event streaming, and result retrieval for every remote agent, we can use standardized operations.&lt;/p&gt;

&lt;p&gt;This is particularly useful for research, document processing, code analysis, and other workflows that may take several minutes or longer.&lt;/p&gt;

&lt;h3&gt;
  
  
  More flexible output handling
&lt;/h3&gt;

&lt;p&gt;Agents can exchange text, files, structured data, and other supported content types through message parts and artifacts.&lt;/p&gt;

&lt;p&gt;This makes the protocol useful beyond conventional text-based chat applications.&lt;/p&gt;

&lt;h3&gt;
  
  
  Centralized agent discovery and integration
&lt;/h3&gt;

&lt;p&gt;Agent Cards allow applications to discover what an agent offers and how to communicate with it.&lt;/p&gt;

&lt;p&gt;Combined with a curated internal agent registry, this can provide a common discovery layer for enterprise AI platforms.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Limitations and Trade-offs
&lt;/h2&gt;

&lt;p&gt;A2A solves a real interoperability problem, but it also introduces architectural complexity.&lt;/p&gt;

&lt;h3&gt;
  
  
  It does not provide orchestration by itself
&lt;/h3&gt;

&lt;p&gt;A2A defines how agents communicate, not how an entire multi-agent system should be planned and executed.&lt;/p&gt;

&lt;p&gt;It does not decide which agent should receive a task, how many agents should participate, or how conflicting results should be resolved.&lt;/p&gt;

&lt;p&gt;Those responsibilities remain with an orchestrator or an agent framework.&lt;/p&gt;

&lt;p&gt;For example, LangGraph, Google ADK, or a custom coordinator can decide when to delegate a task through A2A.&lt;/p&gt;

&lt;h3&gt;
  
  
  Distributed execution introduces more failure scenarios
&lt;/h3&gt;

&lt;p&gt;When agents communicate across network boundaries, new failure scenarios appear:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A remote agent may become unavailable.&lt;/li&gt;
&lt;li&gt;A request may time out while the task continues running.&lt;/li&gt;
&lt;li&gt;A task may fail after partial execution.&lt;/li&gt;
&lt;li&gt;A client may disconnect before receiving the final artifact.&lt;/li&gt;
&lt;li&gt;A downstream agent may return an incomplete or unexpected result.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Therefore, retries, cancellation, idempotency, timeout policies, and persistent task storage need to be considered carefully.&lt;/p&gt;

&lt;h3&gt;
  
  
  More communication can increase latency and cost
&lt;/h3&gt;

&lt;p&gt;Every agent handoff introduces additional work.&lt;/p&gt;

&lt;p&gt;A complex multi-agent workflow may involve several model calls, network requests, and tool executions.&lt;/p&gt;

&lt;p&gt;Using more agents does not automatically improve the result.&lt;/p&gt;

&lt;p&gt;We should introduce agent delegation only when the additional specialization or parallelism justifies the extra latency, cost, and operational complexity.&lt;/p&gt;

&lt;h3&gt;
  
  
  Agent compatibility does not guarantee equivalent capabilities
&lt;/h3&gt;

&lt;p&gt;Two agents may both implement A2A but provide very different skills and execution behavior.&lt;/p&gt;

&lt;p&gt;A2A standardizes communication, not the quality or accuracy of an agent.&lt;/p&gt;

&lt;p&gt;Clients still need to evaluate supported capabilities, input/output formats, and the reliability of the remote service.&lt;/p&gt;

&lt;h3&gt;
  
  
  Version compatibility requires attention
&lt;/h3&gt;

&lt;p&gt;The transition from earlier protocol versions to v1.0 introduced changes to operation names, Agent Cards, and other protocol details.&lt;/p&gt;

&lt;p&gt;The official TypeScript SDK provides an opt-in v0.3 compatibility layer, which can help during staged migration.&lt;/p&gt;

&lt;p&gt;However, version compatibility should be tested explicitly rather than assumed.&lt;/p&gt;

&lt;h3&gt;
  
  
  Security does not come automatically
&lt;/h3&gt;

&lt;p&gt;A2A supports authentication and authorization mechanisms, but the protocol does not automatically make remote agents trustworthy.&lt;/p&gt;

&lt;p&gt;The application must decide which agents can communicate, what data can be shared, and which actions are permitted.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://a2a-protocol.org/v1.0.0/specification/" rel="noopener noreferrer"&gt;A2A v1.0 Specification&lt;/a&gt; · &lt;a href="https://a2a-protocol.org/latest/specification/" rel="noopener noreferrer"&gt;A2A Security Considerations&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Security and Production Considerations
&lt;/h2&gt;

&lt;p&gt;This is one of the most important parts of adopting A2A in an enterprise environment.&lt;/p&gt;

&lt;p&gt;An agent may receive data from another agent, process documents, invoke tools, and return results that influence subsequent decisions.&lt;/p&gt;

&lt;p&gt;We should not treat every remote agent as trusted simply because it implements the same protocol.&lt;/p&gt;

&lt;p&gt;Some important considerations:&lt;/p&gt;

&lt;h3&gt;
  
  
  Authentication
&lt;/h3&gt;

&lt;p&gt;A2A relies on established authentication mechanisms such as HTTP authentication, OAuth 2.0, OpenID Connect, API keys, and mutual TLS where appropriate.&lt;/p&gt;

&lt;p&gt;The agent can advertise supported authentication schemes through its Agent Card.&lt;/p&gt;

&lt;p&gt;Credentials are handled at the transport layer, not embedded directly into protocol messages.&lt;/p&gt;

&lt;h3&gt;
  
  
  Authorization
&lt;/h3&gt;

&lt;p&gt;Authentication and authorization are different responsibilities.&lt;/p&gt;

&lt;p&gt;Knowing who is calling an agent does not automatically mean that the caller should be allowed to invoke every skill.&lt;/p&gt;

&lt;p&gt;Authorization should consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which application or user initiated the task?&lt;/li&gt;
&lt;li&gt;Which agent is allowed to communicate with the target agent?&lt;/li&gt;
&lt;li&gt;Which skills can be invoked?&lt;/li&gt;
&lt;li&gt;Which business operations are permitted?&lt;/li&gt;
&lt;li&gt;Which data can be accessed or shared?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For financial or other sensitive enterprise environments, authorization should be enforced at the skill and business-action levels.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data isolation
&lt;/h3&gt;

&lt;p&gt;Agents should receive only the information required to complete their tasks.&lt;/p&gt;

&lt;p&gt;A research agent should not automatically receive the entire conversation history, database access, or credentials of another agent.&lt;/p&gt;

&lt;p&gt;Task and context storage should also respect user and tenant boundaries.&lt;/p&gt;

&lt;h3&gt;
  
  
  Agent Card validation
&lt;/h3&gt;

&lt;p&gt;Agent Cards describe service identity and capabilities.&lt;/p&gt;

&lt;p&gt;For sensitive environments, clients should verify the origin of Agent Cards and validate their signatures when available.&lt;/p&gt;

&lt;p&gt;A discovery result should not automatically be treated as a trusted execution target.&lt;/p&gt;

&lt;h3&gt;
  
  
  Observability and auditing
&lt;/h3&gt;

&lt;p&gt;A multi-agent system needs visibility beyond individual model calls.&lt;/p&gt;

&lt;p&gt;We should record:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Caller and target agent identifiers&lt;/li&gt;
&lt;li&gt;Task and context identifiers&lt;/li&gt;
&lt;li&gt;Correlation and trace identifiers&lt;/li&gt;
&lt;li&gt;Task state transitions&lt;/li&gt;
&lt;li&gt;Request duration and failure reasons&lt;/li&gt;
&lt;li&gt;Artifact metadata and access&lt;/li&gt;
&lt;li&gt;Sensitive action approvals&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;OpenTelemetry and distributed tracing can help follow a request across multiple agents.&lt;/p&gt;

&lt;h3&gt;
  
  
  Push notification security
&lt;/h3&gt;

&lt;p&gt;Push notification webhooks need their own security controls.&lt;/p&gt;

&lt;p&gt;For example, the receiver should authenticate the notification source, validate the task identifier, and process duplicate notifications safely.&lt;/p&gt;

&lt;p&gt;Webhook endpoints should also be protected against SSRF and unauthorized access.&lt;/p&gt;

&lt;p&gt;These controls are application and infrastructure responsibilities that must be designed alongside the protocol integration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://a2a-protocol.org/latest/topics/enterprise-ready/" rel="noopener noreferrer"&gt;A2A — Enterprise Implementation&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  11. What Can an AI Chapter Lead Contribute?
&lt;/h2&gt;

&lt;p&gt;I think this is where A2A becomes especially interesting from an engineering leadership perspective.&lt;/p&gt;

&lt;p&gt;The value is not simply adding another protocol to the technology stack. It is establishing a reusable architecture for teams that want to build specialized agents independently and allow them to collaborate.&lt;/p&gt;

&lt;p&gt;There are several areas where we can contribute.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Establish a common agent communication standard
&lt;/h3&gt;

&lt;p&gt;Different teams may build agents using different languages and frameworks.&lt;/p&gt;

&lt;p&gt;Instead of forcing everyone to adopt the same framework, we can define a common communication contract based on A2A.&lt;/p&gt;

&lt;p&gt;This allows teams to own their agent implementations while maintaining a consistent integration approach.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Build an internal agent registry
&lt;/h3&gt;

&lt;p&gt;We can create a centralized catalog of available agents and their capabilities.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Agent&lt;/th&gt;
&lt;th&gt;Responsibility&lt;/th&gt;
&lt;th&gt;Available skills&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Research Agent&lt;/td&gt;
&lt;td&gt;Information gathering&lt;/td&gt;
&lt;td&gt;Research, source analysis&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Database Agent&lt;/td&gt;
&lt;td&gt;Data analysis&lt;/td&gt;
&lt;td&gt;SQL analysis, reporting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Code Review Agent&lt;/td&gt;
&lt;td&gt;Code quality&lt;/td&gt;
&lt;td&gt;Review, recommendations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Security Agent&lt;/td&gt;
&lt;td&gt;Security analysis&lt;/td&gt;
&lt;td&gt;Vulnerability assessment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Documentation Agent&lt;/td&gt;
&lt;td&gt;Documentation&lt;/td&gt;
&lt;td&gt;Search, summarization&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each agent can publish an Agent Card, while the internal registry handles discovery, ownership, versioning, authorization, and availability.&lt;/p&gt;

&lt;p&gt;This would help teams reuse existing capabilities instead of implementing the same functionality repeatedly.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Define a reusable orchestration architecture
&lt;/h3&gt;

&lt;p&gt;A2A should be combined with a clear orchestration strategy.&lt;/p&gt;

&lt;p&gt;We can establish a reference architecture that separates:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agent discovery&lt;/li&gt;
&lt;li&gt;Task delegation&lt;/li&gt;
&lt;li&gt;Agent communication&lt;/li&gt;
&lt;li&gt;Execution state management&lt;/li&gt;
&lt;li&gt;Tool and data access&lt;/li&gt;
&lt;li&gt;Result aggregation&lt;/li&gt;
&lt;li&gt;Human approval&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The coordinator should remain responsible for task planning and result aggregation. Specialized agents should focus on their assigned responsibilities.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Establish security and governance standards
&lt;/h3&gt;

&lt;p&gt;A2A introduces communication between independently managed systems.&lt;/p&gt;

&lt;p&gt;We should define clear standards for agent authentication, authorization, data sharing, tenant isolation, and audit logging.&lt;/p&gt;

&lt;p&gt;This is especially important when agents interact with internal systems or execute business operations.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Run a practical multi-agent pilot
&lt;/h3&gt;

&lt;p&gt;Rather than introducing A2A across every team immediately, we can start with a well-defined workflow.&lt;/p&gt;

&lt;p&gt;A good example would be an automated merge request review process.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example workflow:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A developer submits a merge request.&lt;/li&gt;
&lt;li&gt;The Coordinator Agent receives the review request.&lt;/li&gt;
&lt;li&gt;The Security Agent analyzes potential vulnerabilities.&lt;/li&gt;
&lt;li&gt;The Testing Agent evaluates test coverage and identifies missing cases.&lt;/li&gt;
&lt;li&gt;The Architecture Agent reviews design and implementation decisions.&lt;/li&gt;
&lt;li&gt;Each specialist returns findings through A2A.&lt;/li&gt;
&lt;li&gt;The Coordinator combines the results into a structured review report.&lt;/li&gt;
&lt;li&gt;A developer reviews the recommendations before any changes are merged.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each agent can use its own MCP servers to access GitLab, documentation, test tools, or other required services.&lt;/p&gt;

&lt;p&gt;This allows us to evaluate the actual value of agent specialization and interoperability.&lt;/p&gt;

&lt;h3&gt;
  
  
  6. Measure the results
&lt;/h3&gt;

&lt;p&gt;A successful pilot should be evaluated using real measurements.&lt;/p&gt;

&lt;p&gt;Some useful metrics:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Task completion rate&lt;/li&gt;
&lt;li&gt;Agent-to-agent communication latency&lt;/li&gt;
&lt;li&gt;Overall workflow duration&lt;/li&gt;
&lt;li&gt;Failed or timed-out tasks&lt;/li&gt;
&lt;li&gt;Duplicate execution rate&lt;/li&gt;
&lt;li&gt;Token and infrastructure costs&lt;/li&gt;
&lt;li&gt;Human intervention rate&lt;/li&gt;
&lt;li&gt;Quality of the final aggregated result&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The objective is to determine whether distributing work across specialized agents produces better results than a single-agent implementation.&lt;/p&gt;

&lt;h3&gt;
  
  
  7. Create reusable learning resources
&lt;/h3&gt;

&lt;p&gt;A practical reference implementation, architecture documentation, and internal workshops can help developers understand when A2A is useful.&lt;/p&gt;

&lt;p&gt;Teams should learn not only how to implement the protocol, but also when not to use it.&lt;/p&gt;

&lt;p&gt;Not every internal function needs to become a remote agent. For simple operations within the same runtime, a regular function or framework-native tool call may be more appropriate.&lt;/p&gt;

&lt;h2&gt;
  
  
  12. When Should We Use A2A?
&lt;/h2&gt;

&lt;p&gt;A2A is most useful when communication between independent agents is an actual architectural requirement.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Recommendation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Agents built with different frameworks need to collaborate&lt;/td&gt;
&lt;td&gt;A2A&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Different teams expose independently maintained agents&lt;/td&gt;
&lt;td&gt;A2A&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Remote, long-running agent tasks need tracking&lt;/td&gt;
&lt;td&gt;A2A&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agents need to exchange structured outputs and files&lt;/td&gt;
&lt;td&gt;A2A&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;An agent needs access to APIs and databases&lt;/td&gt;
&lt;td&gt;MCP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A single agent needs internal tool execution&lt;/td&gt;
&lt;td&gt;Native tools or MCP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Multiple agents operate within one tightly coupled workflow&lt;/td&gt;
&lt;td&gt;Framework-native orchestration may be sufficient&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The important thing is to avoid introducing network boundaries and protocol complexity when a simpler implementation would work.&lt;/p&gt;

&lt;p&gt;A2A should solve an interoperability requirement, not become an additional abstraction by default.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;A2A addresses an important challenge in the evolution of agentic AI systems: enabling independently developed agents to communicate and collaborate.&lt;/p&gt;

&lt;p&gt;As AI applications become more specialized, a single agent will not necessarily be responsible for every task. Different agents may own different capabilities, use different frameworks, and operate in different environments.&lt;/p&gt;

&lt;p&gt;A2A provides a standardized way for these systems to discover each other, delegate tasks, exchange information, and track execution.&lt;/p&gt;

&lt;p&gt;Combined with MCP, it creates a useful architectural separation: MCP connects agents to tools and external context, while A2A connects agents to other agents.&lt;/p&gt;

&lt;p&gt;For an AI Chapter Lead, the opportunity is to establish common integration standards, build an internal agent ecosystem, define security boundaries, and help teams develop reusable capabilities.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;My recommendation:&lt;/strong&gt; Start with one practical multi-agent workflow, implement it using the official A2A SDK, and evaluate the operational benefits against a conventional single-agent or framework-native orchestration approach.&lt;/p&gt;

&lt;p&gt;The real question is not how many agents we can connect. It is whether connecting specialized agents makes the system more reliable, maintainable, and effective.&lt;/p&gt;




&lt;h2&gt;
  
  
  References and Further Reading
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://a2a-protocol.org/latest/" rel="noopener noreferrer"&gt;A2A Protocol — Official Website&lt;/a&gt; — Overview, architecture, ecosystem, and documentation.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://a2a-protocol.org/v1.0.0/specification/" rel="noopener noreferrer"&gt;A2A Protocol — Specification v1.0&lt;/a&gt; — Protocol operations, data models, task lifecycle, transports, and security.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://a2a-protocol.org/latest/topics/agent-discovery/" rel="noopener noreferrer"&gt;A2A — Agent Discovery&lt;/a&gt; — Agent Cards, discovery mechanisms, and capability descriptions.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://a2a-protocol.org/latest/topics/life-of-a-task/" rel="noopener noreferrer"&gt;A2A — Life of a Task&lt;/a&gt; — Task lifecycle and interaction patterns.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://a2a-protocol.org/latest/topics/streaming-and-async/" rel="noopener noreferrer"&gt;A2A — Streaming and Asynchronous Operations&lt;/a&gt; — Streaming, task updates, and push notifications.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://a2a-protocol.org/latest/topics/enterprise-ready/" rel="noopener noreferrer"&gt;A2A — Enterprise Implementation&lt;/a&gt; — Authentication, authorization, observability, and enterprise security.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/a2aproject/a2a-js" rel="noopener noreferrer"&gt;A2A — Official JavaScript SDK&lt;/a&gt; — TypeScript client/server implementation, transport adapters, and runnable examples.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/a2aproject" rel="noopener noreferrer"&gt;A2A — Official GitHub Organization&lt;/a&gt; — SDKs, samples, and project development.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://ai-sdk.dev/providers/community-providers/a2a" rel="noopener noreferrer"&gt;AI SDK — A2A Community Provider&lt;/a&gt; — Integrating A2A agents with Vercel AI SDK.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/dracoblue/a2a-ai-provider" rel="noopener noreferrer"&gt;A2A AI Provider&lt;/a&gt; — Community provider implementation and current maturity status.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://modelcontextprotocol.io/" rel="noopener noreferrer"&gt;Model Context Protocol&lt;/a&gt; — Complementary agent-to-tool integration protocol.&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>architecture</category>
      <category>agents</category>
      <category>systemdesign</category>
    </item>
    <item>
      <title>Retry Budgets in Go: Preventing Retries from Becoming an Outage</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Fri, 30 Oct 2026 18:34:52 +0000</pubDate>
      <link>https://dev.to/serifcolakel/retry-budgets-in-go-preventing-retries-from-becoming-an-outage-6p4</link>
      <guid>https://dev.to/serifcolakel/retry-budgets-in-go-preventing-retries-from-becoming-an-outage-6p4</guid>
      <description>&lt;p&gt;Retries are one of the most common ways to make distributed systems more resilient.&lt;/p&gt;

&lt;p&gt;A request fails because of a temporary network issue. The client waits briefly and tries again. The second attempt succeeds, and the failure never reaches the user.&lt;/p&gt;

&lt;p&gt;That sounds reasonable.&lt;/p&gt;

&lt;p&gt;But what happens when the dependency is already overloaded?&lt;/p&gt;

&lt;p&gt;Requests start failing. Every caller retries. Those retries create additional traffic, which causes more failures, which trigger even more retries.&lt;/p&gt;

&lt;p&gt;A temporary slowdown can turn into a retry storm.&lt;/p&gt;

&lt;p&gt;Retries are not free. Every attempt consumes resources, occupies connections, increases queue pressure, and adds work to an already struggling dependency.&lt;/p&gt;

&lt;p&gt;This is where &lt;strong&gt;retry budgets&lt;/strong&gt; become useful.&lt;/p&gt;

&lt;p&gt;A retry budget places a limit on how much additional traffic a service can generate through retries. Instead of allowing every failed request to retry independently, it establishes a shared budget across requests.&lt;/p&gt;

&lt;p&gt;In this article, we'll explore:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Why per-request retry limits are not enough&lt;/li&gt;
&lt;li&gt;How retry amplification creates cascading failures&lt;/li&gt;
&lt;li&gt;The difference between retry limits and retry budgets&lt;/li&gt;
&lt;li&gt;How to implement a concurrency-safe retry budget in Go&lt;/li&gt;
&lt;li&gt;How to combine budgets with exponential backoff and jitter&lt;/li&gt;
&lt;li&gt;How retries interact with timeouts, bulkheads, and circuit breakers&lt;/li&gt;
&lt;li&gt;What to measure and test before using retries in production&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  1. The Problem: Retries Can Amplify Failures
&lt;/h2&gt;

&lt;p&gt;Imagine a service that sends requests to an external market data provider.&lt;/p&gt;

&lt;p&gt;Under normal conditions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The service receives 1,000 requests per second.&lt;/li&gt;
&lt;li&gt;Each request makes one provider call.&lt;/li&gt;
&lt;li&gt;The provider responds successfully.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Now the provider starts experiencing problems. Its error rate increases, and requests become slower.&lt;/p&gt;

&lt;p&gt;The service retries failed requests up to three times.&lt;/p&gt;

&lt;p&gt;If every original request fails, the provider may receive:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Attempt&lt;/th&gt;
&lt;th&gt;Requests&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Original requests&lt;/td&gt;
&lt;td&gt;1,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;First retries&lt;/td&gt;
&lt;td&gt;1,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Second retries&lt;/td&gt;
&lt;td&gt;1,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Third retries&lt;/td&gt;
&lt;td&gt;1,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4,000&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The service has turned 1,000 incoming requests into as many as 4,000 downstream attempts.&lt;/p&gt;

&lt;p&gt;This is retry amplification.&lt;/p&gt;

&lt;p&gt;The actual situation can be worse. Retries may overlap with new incoming traffic, other services may retry independently, and clients may have their own retry policies.&lt;/p&gt;

&lt;p&gt;The dependency is already struggling, but the system keeps sending it more work.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdummyimage.com%2F1200x520%2F101827%2F00eaff.png%26text%3DRetry%2BAmplification%253A%2B1000%2BRequests%2B%25E2%2586%2592%2B4000%2BAttempts" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdummyimage.com%2F1200x520%2F101827%2F00eaff.png%26text%3DRetry%2BAmplification%253A%2B1000%2BRequests%2B%25E2%2586%2592%2B4000%2BAttempts" alt="Retry amplification diagram" width="1200" height="520"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The important distinction is that retries can improve the success rate of individual requests while making the overall system less stable.&lt;/p&gt;

&lt;p&gt;A retry policy should not only ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How many times should this request retry?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It should also ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How much additional traffic can this service afford to generate?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  2. Why Max Retries Is Not Enough
&lt;/h2&gt;

&lt;p&gt;A typical retry implementation has a maximum attempt count:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;callDependency&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This limits retries for an individual request.&lt;/p&gt;

&lt;p&gt;But it does not limit retries across the entire service.&lt;/p&gt;

&lt;p&gt;Consider a service receiving 10,000 requests per second. Even with only one retry per request, it can generate another 10,000 downstream attempts per second during an outage.&lt;/p&gt;

&lt;p&gt;A per-request retry limit provides a local constraint, not a global traffic constraint.&lt;/p&gt;

&lt;p&gt;There are two separate controls:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Control&lt;/th&gt;
&lt;th&gt;What it limits&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Maximum attempts&lt;/td&gt;
&lt;td&gt;Retries for one request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retry budget&lt;/td&gt;
&lt;td&gt;Retries across a service or dependency&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;You generally want both.&lt;/p&gt;

&lt;p&gt;A maximum attempt count prevents one request from retrying indefinitely. A shared retry budget prevents thousands of requests from independently exhausting a dependency.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. What Is a Retry Budget?
&lt;/h2&gt;

&lt;p&gt;A retry budget is a shared allowance for additional attempts.&lt;/p&gt;

&lt;p&gt;Instead of allowing every failed request to retry automatically, the service must acquire permission from the budget before performing another attempt.&lt;/p&gt;

&lt;p&gt;One practical approach is a token bucket.&lt;/p&gt;

&lt;p&gt;The model is simple:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Successful initial requests add tokens to the budget.&lt;/li&gt;
&lt;li&gt;Each retry consumes one token.&lt;/li&gt;
&lt;li&gt;The budget has a maximum capacity.&lt;/li&gt;
&lt;li&gt;When no tokens are available, additional retries are rejected.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For example, suppose a service uses these settings:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Maximum budget: 20 tokens&lt;/li&gt;
&lt;li&gt;Credit per successful initial request: 0.1 tokens&lt;/li&gt;
&lt;li&gt;Cost per retry: 1 token&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every ten successful initial requests generate one retry token.&lt;/p&gt;

&lt;p&gt;If the service experiences a sudden failure, it can spend its accumulated tokens on retries. Once the budget is exhausted, it stops generating additional retry traffic until successful initial requests replenish it.&lt;/p&gt;

&lt;p&gt;This creates an important feedback mechanism:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Healthy service: successful requests replenish the budget.&lt;/li&gt;
&lt;li&gt;Unhealthy service: fewer successful requests mean less budget replenishment.&lt;/li&gt;
&lt;li&gt;Outage: retries consume the remaining budget and eventually stop.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The budget is not a guarantee that a dependency can handle a particular amount of traffic. It is a mechanism for limiting retry amplification.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Implementing a Retry Budget in Go
&lt;/h2&gt;

&lt;p&gt;Let's start with the shared budget.&lt;/p&gt;

&lt;p&gt;We need a structure that is safe to access concurrently because multiple requests may attempt to consume or replenish tokens at the same time.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;retry&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="s"&gt;"sync"&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;RetryBudget&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;mu&lt;/span&gt; &lt;span class="n"&gt;sync&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Mutex&lt;/span&gt;

    &lt;span class="n"&gt;tokens&lt;/span&gt;          &lt;span class="kt"&gt;float64&lt;/span&gt;
    &lt;span class="n"&gt;capacity&lt;/span&gt;        &lt;span class="kt"&gt;float64&lt;/span&gt;
    &lt;span class="n"&gt;creditPerSuccess&lt;/span&gt; &lt;span class="kt"&gt;float64&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;NewRetryBudget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;capacity&lt;/span&gt; &lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;creditPerSuccess&lt;/span&gt; &lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;RetryBudget&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;RetryBudget&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;capacity&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;         &lt;span class="n"&gt;capacity&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;creditPerSuccess&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;creditPerSuccess&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;RetryBudget&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;RecordSuccess&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;creditPerSuccess&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;capacity&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;capacity&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;RetryBudget&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;TryTake&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mu&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Unlock&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;false&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;true&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The implementation deliberately keeps the budget small and predictable.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;RecordSuccess&lt;/code&gt; adds credits without exceeding the configured capacity.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;TryTake&lt;/code&gt; atomically checks and consumes a token. This prevents concurrent requests from spending the same token.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;budget&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;NewRetryBudget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="m"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;   &lt;span class="c"&gt;// Maximum tokens&lt;/span&gt;
    &lt;span class="m"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c"&gt;// Credit per successful initial request&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A successful initial request adds 0.1 tokens. A retry requires one complete token.&lt;/p&gt;

&lt;p&gt;The budget starts empty. This is intentional: a newly started instance does not immediately receive permission to generate retries. If a service needs an initial retry allowance, that can be configured explicitly.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Building a Retry Executor
&lt;/h2&gt;

&lt;p&gt;A retry budget is only one part of a retry policy.&lt;/p&gt;

&lt;p&gt;A useful retry executor should also support:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A maximum number of attempts&lt;/li&gt;
&lt;li&gt;Exponential backoff&lt;/li&gt;
&lt;li&gt;Jitter&lt;/li&gt;
&lt;li&gt;Context cancellation&lt;/li&gt;
&lt;li&gt;Retryable error classification&lt;/li&gt;
&lt;li&gt;A shared retry budget&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;First, define the policy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;package&lt;/span&gt; &lt;span class="n"&gt;retry&lt;/span&gt;

&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s"&gt;"context"&lt;/span&gt;
    &lt;span class="s"&gt;"errors"&lt;/span&gt;
    &lt;span class="s"&gt;"math/rand"&lt;/span&gt;
    &lt;span class="s"&gt;"time"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;ErrInvalidPolicy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;New&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"invalid retry policy"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Policy&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;MaxAttempts&lt;/span&gt;  &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;InitialDelay&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;
    &lt;span class="n"&gt;MaxDelay&lt;/span&gt;     &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;

    &lt;span class="n"&gt;Budget&lt;/span&gt;      &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;RetryBudget&lt;/span&gt;
    &lt;span class="n"&gt;ShouldRetry&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;MaxAttempts&lt;/code&gt; includes the original attempt.&lt;/p&gt;

&lt;p&gt;For example, &lt;code&gt;MaxAttempts: 3&lt;/code&gt; means one original attempt and at most two retries.&lt;/p&gt;

&lt;p&gt;Next, calculate the exponential backoff.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;backoff&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;initial&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;maximum&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;initial&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;maximum&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;maximum&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;*=&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;maximum&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;maximum&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The delay grows exponentially until it reaches the configured maximum.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Retry&lt;/th&gt;
&lt;th&gt;Maximum delay&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;100ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;200ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;400ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;800ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;1.6s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The next step is jitter.&lt;/p&gt;

&lt;p&gt;Without jitter, clients that fail at approximately the same time may retry at approximately the same time. This creates synchronized traffic spikes.&lt;/p&gt;

&lt;p&gt;Full jitter chooses a random delay between zero and the calculated maximum:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;fullJitter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;maxDelay&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;maxDelay&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Duration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;rand&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Float64&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="kt"&gt;float64&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;maxDelay&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The package-level &lt;code&gt;math/rand&lt;/code&gt; functions are safe for concurrent use. If you need reproducible tests, inject a random source rather than depending on global randomness.&lt;/p&gt;

&lt;p&gt;Now we can implement the retry executor.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;policy&lt;/span&gt; &lt;span class="n"&gt;Policy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;operation&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;InitialDelay&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxDelay&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;InitialDelay&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt;
        &lt;span class="n"&gt;operation&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ErrInvalidPolicy&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ShouldRetry&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ErrInvalidPolicy&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;operation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="c"&gt;// Only successful initial attempts replenish the budget.&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Budget&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RecordSuccess&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxAttempts&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ShouldRetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="c"&gt;// Do not generate more traffic when the shared&lt;/span&gt;
        &lt;span class="c"&gt;// retry budget has been exhausted.&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Budget&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TryTake&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

        &lt;span class="n"&gt;maxDelay&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;backoff&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;InitialDelay&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MaxDelay&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;fullJitter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;maxDelay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="n"&gt;timer&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewTimer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;select&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Stop&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;C&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;

            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Err&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

        &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="n"&gt;timer&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;C&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few design decisions are worth highlighting.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First, the budget is consumed before the retry.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Once the service decides to schedule another attempt, it spends a token. If the context is canceled during the backoff period, that token is not refunded.&lt;/p&gt;

&lt;p&gt;This is a conservative choice. The budget controls permission to generate retry work, not only completed downstream calls.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Second, only successful initial attempts replenish the budget.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A successful retry does not replenish it. Otherwise, a service could keep generating retries and effectively fund its own retry traffic during a degraded period.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Third, the context is checked before each attempt and during backoff.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A canceled request should not continue generating work.&lt;/p&gt;

&lt;p&gt;The executor returns the last operation error when the retry budget is exhausted. This preserves the underlying failure for callers, although production implementations may also expose a separate retry-denied metric or callback.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Choosing Which Errors Are Retryable
&lt;/h2&gt;

&lt;p&gt;Not every failure should trigger a retry.&lt;/p&gt;

&lt;p&gt;A retry policy should distinguish between transient failures and permanent failures.&lt;/p&gt;

&lt;p&gt;Examples of potentially retryable failures:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Temporary network errors&lt;/li&gt;
&lt;li&gt;Connection resets&lt;/li&gt;
&lt;li&gt;Selected HTTP 5xx responses&lt;/li&gt;
&lt;li&gt;HTTP 429 responses, subject to server guidance&lt;/li&gt;
&lt;li&gt;Temporary resource unavailability&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Examples of failures that generally should not be retried:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Invalid input&lt;/li&gt;
&lt;li&gt;Authentication or authorization failures&lt;/li&gt;
&lt;li&gt;Unsupported operations&lt;/li&gt;
&lt;li&gt;Validation errors&lt;/li&gt;
&lt;li&gt;Requests that have already exceeded their deadline&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A simple classifier might look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;shouldRetry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;bool&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="n"&gt;netErr&lt;/span&gt; &lt;span class="n"&gt;net&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Error&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;As&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;netErr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;netErr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Timeout&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;netErr&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Temporary&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;false&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is only an illustration. &lt;code&gt;net.Error.Temporary()&lt;/code&gt; is deprecated, and real applications should classify errors according to their HTTP client, database driver, RPC framework, and dependency-specific semantics.&lt;/p&gt;

&lt;p&gt;For HTTP clients, for example, a 429 response may include &lt;code&gt;Retry-After&lt;/code&gt;. A robust policy should respect that server-provided delay where applicable, while still enforcing the request deadline and retry budget.&lt;/p&gt;

&lt;p&gt;Also remember that retrying a request is not always safe.&lt;/p&gt;

&lt;p&gt;A GET request is generally easier to retry than a payment or order-creation request. For operations with side effects, use idempotency keys or another deduplication mechanism.&lt;/p&gt;

&lt;p&gt;A retry budget controls traffic. It does not make an operation safe to execute twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Putting It Together
&lt;/h2&gt;

&lt;p&gt;Let's use the retry executor with a market data provider.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;budget&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;retry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NewRetryBudget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="m"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="m"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;policy&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;retry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Policy&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;MaxAttempts&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;  &lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;InitialDelay&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="m"&gt;100&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Millisecond&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;MaxDelay&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;     &lt;span class="m"&gt;2&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;Budget&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;       &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;ShouldRetry&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;  &lt;span class="n"&gt;shouldRetry&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WithTimeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Background&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Second&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;cancel&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;retry&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Do&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;marketDataClient&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;"AAPL"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="no"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c"&gt;// Handle the final operation error.&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The behavior is now bounded by multiple independent controls:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The request can make at most three attempts.&lt;/li&gt;
&lt;li&gt;The request cannot exceed its context deadline.&lt;/li&gt;
&lt;li&gt;Each retry uses exponential backoff with jitter.&lt;/li&gt;
&lt;li&gt;The service must have an available retry token.&lt;/li&gt;
&lt;li&gt;Only successful initial attempts replenish the shared budget.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is much safer than allowing every request to retry independently.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Retry Budgets and Other Resilience Patterns
&lt;/h2&gt;

&lt;p&gt;Retry budgets do not replace other resilience mechanisms. They complement them.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mechanism&lt;/th&gt;
&lt;th&gt;Primary responsibility&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Timeout&lt;/td&gt;
&lt;td&gt;Limits how long an operation can run&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backpressure&lt;/td&gt;
&lt;td&gt;Controls how much work enters a system&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Load shedding&lt;/td&gt;
&lt;td&gt;Rejects work when capacity is insufficient&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bulkhead&lt;/td&gt;
&lt;td&gt;Isolates resource consumption&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Circuit breaker&lt;/td&gt;
&lt;td&gt;Stops calls to an unhealthy dependency&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retry budget&lt;/td&gt;
&lt;td&gt;Limits additional traffic generated by retries&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Consider how these mechanisms work together.&lt;/p&gt;

&lt;p&gt;A request enters the service. Backpressure controls admission. A bulkhead limits concurrent calls to the provider. The request has a deadline. If the provider fails, the retry policy classifies the error and checks the retry budget. If the dependency is persistently unhealthy, the circuit breaker can stop further calls.&lt;/p&gt;

&lt;p&gt;Each mechanism addresses a different failure mode.&lt;/p&gt;

&lt;h3&gt;
  
  
  A subtle interaction: Bulkheads and retries
&lt;/h3&gt;

&lt;p&gt;Retry placement matters.&lt;/p&gt;

&lt;p&gt;Suppose a bulkhead permit is acquired once around the entire retry loop. The permit remains occupied during backoff.&lt;/p&gt;

&lt;p&gt;That may be desirable if the goal is to limit the number of complete operations, but it can also waste scarce concurrency capacity while requests are sleeping.&lt;/p&gt;

&lt;p&gt;Alternatively, acquiring a permit for each individual attempt releases capacity during backoff. However, this allows more logical requests to remain in progress simultaneously.&lt;/p&gt;

&lt;p&gt;Neither approach is universally correct. The choice depends on what the bulkhead is protecting:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If it protects active downstream calls, acquire a permit per attempt.&lt;/li&gt;
&lt;li&gt;If it protects the number of in-flight logical operations, hold the permit across the complete retry operation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important thing is to define the resource boundary explicitly.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. Is a Local Retry Budget Enough?
&lt;/h2&gt;

&lt;p&gt;A mutex-protected retry budget is local to one process.&lt;/p&gt;

&lt;p&gt;If you run ten replicas, each replica has its own budget.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;10 replicas&lt;/li&gt;
&lt;li&gt;20 tokens per replica&lt;/li&gt;
&lt;li&gt;200 tokens of aggregate retry capacity&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This may be perfectly acceptable. Local budgets are simple, fast, and do not introduce another distributed dependency.&lt;/p&gt;

&lt;p&gt;But if the external provider enforces a strict global quota, local budgets alone may not be sufficient.&lt;/p&gt;

&lt;p&gt;Possible approaches include:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Local budgets:&lt;/strong&gt; Simple and independent, but aggregate capacity grows with replica count.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Centralized budget:&lt;/strong&gt; Coordinates retry allowance across instances, but introduces network calls and another failure boundary.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Provider-aware limits:&lt;/strong&gt; Use the provider's documented quota and response headers to coordinate traffic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Centralized dispatch:&lt;/strong&gt; Route downstream work through a shared queue or dispatcher that controls the total attempt rate.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Do not introduce a distributed retry-budget service automatically. The coordination mechanism can become more expensive and less reliable than the problem it is trying to solve.&lt;/p&gt;

&lt;p&gt;Start with local budgets when they provide sufficient protection. Move to centralized coordination only when the actual quota or failure model requires it.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. Observability: Measure Retry Behavior
&lt;/h2&gt;

&lt;p&gt;A retry policy that cannot be observed is difficult to operate.&lt;/p&gt;

&lt;p&gt;At minimum, expose metrics that help answer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How often are requests retried?&lt;/li&gt;
&lt;li&gt;How many retries are denied by the budget?&lt;/li&gt;
&lt;li&gt;Which dependencies consume the most retry capacity?&lt;/li&gt;
&lt;li&gt;How often do requests exhaust their attempt limit?&lt;/li&gt;
&lt;li&gt;How much time is spent waiting between attempts?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Useful metrics include:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;retry_attempts_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Total retry attempts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;retry_budget_denied_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Retries rejected by the budget&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;retry_budget_tokens&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Gauge&lt;/td&gt;
&lt;td&gt;Current available tokens&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;retry_exhausted_total&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Counter&lt;/td&gt;
&lt;td&gt;Requests reaching the attempt limit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;retry_delay_seconds&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Histogram&lt;/td&gt;
&lt;td&gt;Time spent in retry backoff&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Avoid using raw URLs, request IDs, user IDs, or other high-cardinality values as metric labels.&lt;/p&gt;

&lt;p&gt;Prefer bounded dimensions such as dependency name, operation, and outcome.&lt;/p&gt;

&lt;p&gt;A particularly useful signal is the ratio of retries to original requests.&lt;/p&gt;

&lt;p&gt;If the retry rate starts increasing while the initial success rate drops, the system may be entering a retry-amplification cycle.&lt;/p&gt;

&lt;p&gt;The retry budget should make that cycle bounded, but the metrics help you understand why it happened.&lt;/p&gt;

&lt;h2&gt;
  
  
  11. Testing Retry Budgets
&lt;/h2&gt;

&lt;p&gt;Concurrency behavior should be tested explicitly.&lt;/p&gt;

&lt;p&gt;The first test verifies that successful initial requests replenish the budget and that retries consume tokens.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestRetryBudget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;budget&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;NewRetryBudget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RecordSuccess&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RecordSuccess&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TryTake&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"expected first token"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TryTake&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"expected second token"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TryTake&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"expected budget to be exhausted"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The next test checks that concurrent callers cannot consume more tokens than are available.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="n"&gt;TestRetryBudgetConcurrent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;const&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;tokens&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;100&lt;/span&gt;
        &lt;span class="n"&gt;workers&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="m"&gt;1000&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;budget&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;NewRetryBudget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RecordSuccess&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;var&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;wg&lt;/span&gt;       &lt;span class="n"&gt;sync&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;WaitGroup&lt;/span&gt;
        &lt;span class="n"&gt;accepted&lt;/span&gt; &lt;span class="n"&gt;atomic&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Int64&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;workers&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;workers&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;go&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;defer&lt;/span&gt; &lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Done&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;budget&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TryTake&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="n"&gt;accepted&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}()&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;wg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Wait&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;got&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;accepted&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Load&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="n"&gt;got&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Fatalf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="s"&gt;"expected %d successful acquisitions, got %d"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;got&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run the tests with the race detector:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;go &lt;span class="nb"&gt;test&lt;/span&gt; &lt;span class="nt"&gt;-race&lt;/span&gt; ./...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should also test the retry executor itself:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A successful first attempt should not retry.&lt;/li&gt;
&lt;li&gt;A transient failure should retry when budget is available.&lt;/li&gt;
&lt;li&gt;A permanent failure should return immediately.&lt;/li&gt;
&lt;li&gt;An exhausted budget should prevent another attempt.&lt;/li&gt;
&lt;li&gt;Context cancellation should interrupt backoff.&lt;/li&gt;
&lt;li&gt;The maximum attempt count should be respected.&lt;/li&gt;
&lt;li&gt;A successful retry should not replenish the budget.&lt;/li&gt;
&lt;li&gt;A failed initial request should not replenish the budget.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Finally, test behavior under load. A retry policy that passes unit tests can still create unexpected traffic patterns when thousands of requests fail together.&lt;/p&gt;

&lt;h2&gt;
  
  
  12. Production Checklist
&lt;/h2&gt;

&lt;p&gt;Before enabling retries in a production service, verify the following:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Maximum attempts include the original request.&lt;/li&gt;
&lt;li&gt;[ ] Retryable errors are explicitly classified.&lt;/li&gt;
&lt;li&gt;[ ] Every request has a deadline or timeout.&lt;/li&gt;
&lt;li&gt;[ ] Backoff is exponential and bounded.&lt;/li&gt;
&lt;li&gt;[ ] Jitter prevents synchronized retries.&lt;/li&gt;
&lt;li&gt;[ ] A shared budget limits aggregate retry traffic.&lt;/li&gt;
&lt;li&gt;[ ] Non-idempotent operations have appropriate protection.&lt;/li&gt;
&lt;li&gt;[ ] Provider-specific rate limits and &lt;code&gt;Retry-After&lt;/code&gt; are respected.&lt;/li&gt;
&lt;li&gt;[ ] Retry behavior is observable.&lt;/li&gt;
&lt;li&gt;[ ] Bulkhead permit ownership is clearly defined.&lt;/li&gt;
&lt;li&gt;[ ] Tests cover cancellation, concurrency, and exhaustion.&lt;/li&gt;
&lt;li&gt;[ ] Replica count and aggregate retry capacity are understood.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Retries are useful, but unlimited retry behavior can turn a temporary dependency failure into a system-wide incident.&lt;/p&gt;

&lt;p&gt;A maximum attempt count protects individual requests. A retry budget protects the service from the combined effect of those requests.&lt;/p&gt;

&lt;p&gt;The key idea is simple: successful traffic can fund a limited amount of additional work, but retries should not be allowed to generate unlimited traffic when the system is already unhealthy.&lt;/p&gt;

&lt;p&gt;In Go, a concurrency-safe token bucket provides a straightforward way to implement this policy. Combined with deadlines, exponential backoff, jitter, bulkheads, and circuit breakers, retry budgets help make failure behavior more predictable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Resilience is not about trying again forever. It's about knowing when another attempt is worth the cost.&lt;/strong&gt;&lt;/p&gt;

</description>
      <category>go</category>
      <category>programming</category>
      <category>productivity</category>
      <category>beginners</category>
    </item>
    <item>
      <title>ACP (Agent Client Protocol): Integrating Coding Agents with the Vercel AI SDK</title>
      <dc:creator>Serif COLAKEL</dc:creator>
      <pubDate>Thu, 29 Oct 2026 21:51:58 +0000</pubDate>
      <link>https://dev.to/serifcolakel/acp-agent-client-protocol-integrating-coding-agents-with-the-vercel-ai-sdk-lb5</link>
      <guid>https://dev.to/serifcolakel/acp-agent-client-protocol-integrating-coding-agents-with-the-vercel-ai-sdk-lb5</guid>
      <description>&lt;p&gt;&lt;em&gt;What ACP is, how it relates to MCP, how to integrate it into a TypeScript application, and what it means for teams building agent-powered developer tools.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  1. What Is ACP, and What Problem Does It Solve?
&lt;/h2&gt;

&lt;p&gt;When building AI-powered applications, the usual approach is to connect to a model provider, send a prompt, process the response, and optionally handle tool calls.&lt;/p&gt;

&lt;p&gt;Coding agents introduce a different set of requirements.&lt;/p&gt;

&lt;p&gt;Tools such as Claude Code, Gemini CLI, and Codex CLI can work with a codebase, inspect files, execute commands, make changes, and perform multi-step development tasks. Integrating these capabilities into our own applications requires more than simply displaying a model's text response.&lt;/p&gt;

&lt;p&gt;Without a common protocol, each client may need a custom integration for every agent it wants to support. Agents may also need to implement different interfaces for different clients.&lt;/p&gt;

&lt;p&gt;The &lt;strong&gt;Agent Client Protocol (ACP)&lt;/strong&gt; addresses this problem by standardizing communication between coding agents and client applications. It was designed primarily for coding agents and code editors, including IDEs, but its architecture can also support other agent-enabled client interfaces.&lt;/p&gt;

&lt;p&gt;ACP defines how a client and an agent communicate, establish sessions, exchange messages, report progress, and handle interactions such as permission requests.&lt;/p&gt;

&lt;p&gt;The key point is that &lt;strong&gt;ACP is not an LLM or a model provider. It is a communication protocol for integrating agents into applications.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;This distinction matters because a coding agent is more than a text-generation endpoint. It may have its own execution loop, tools, filesystem access, authentication requirements, and session lifecycle.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://agentclientprotocol.com/get-started/introduction" rel="noopener noreferrer"&gt;Agent Client Protocol — Introduction&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. ACP vs. MCP: What Is the Difference?
&lt;/h2&gt;

&lt;p&gt;ACP and the Model Context Protocol (MCP) are complementary, not competing, technologies. They address different parts of an agent-based architecture.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;ACP (Agent Client Protocol)&lt;/strong&gt; standardizes communication between an agent and its client.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MCP (Model Context Protocol)&lt;/strong&gt; standardizes how AI applications and agents interact with tools and external context sources.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Consider a developer platform that allows an agent to review a GitLab merge request, inspect the related code, make a correction, and run tests.&lt;/p&gt;

&lt;p&gt;The responsibilities could be divided as follows:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;ACP connects the application to the coding agent and carries the interaction.&lt;/li&gt;
&lt;li&gt;MCP makes the required tools available to the agent, such as GitLab operations or access to other services.&lt;/li&gt;
&lt;li&gt;The Vercel AI SDK provides application-level APIs for consuming agent output, including text generation and streaming through a familiar TypeScript interface.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A simplified architecture looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User
 |
 v
AI Application
 |
 v
Vercel AI SDK
 |
 v
ACP Provider
 |
 v
ACP-Compatible Agent
 |
 +----&amp;gt; Filesystem
 +----&amp;gt; Terminal / Tests
 +----&amp;gt; MCP Servers
           |
           +----&amp;gt; GitLab
           +----&amp;gt; Internal APIs
           +----&amp;gt; Other Services
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This diagram illustrates a possible architecture, not a guarantee that every agent supports every capability shown. The available features depend on the agent, its ACP implementation, session configuration, and execution environment.&lt;/p&gt;

&lt;p&gt;One important implementation detail is that the agent can connect to configured MCP servers directly. The application does not necessarily proxy every MCP request through itself.&lt;/p&gt;

&lt;p&gt;This separation helps us design the system around clear responsibilities: ACP handles agent communication, MCP exposes tools, and the application owns the user experience and its policies.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://agentclientprotocol.com/get-started/architecture" rel="noopener noreferrer"&gt;ACP Architecture&lt;/a&gt; · &lt;a href="https://modelcontextprotocol.io/" rel="noopener noreferrer"&gt;MCP Documentation&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  3. What Does the ACP Provider in the Vercel AI SDK Offer?
&lt;/h2&gt;

&lt;p&gt;The Vercel AI SDK has a community provider that integrates ACP-compatible agents through the &lt;code&gt;LanguageModel&lt;/code&gt; interface.&lt;/p&gt;

&lt;p&gt;The package is &lt;code&gt;@mcpc-tech/acp-ai-provider&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It allows a TypeScript application to use an ACP-compatible agent through familiar AI SDK functions rather than implementing the underlying protocol integration from scratch.&lt;/p&gt;

&lt;p&gt;Its documented capabilities include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Agent integration:&lt;/strong&gt; Connect to supported agents such as Gemini CLI, Claude Code, and Codex CLI.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Text generation:&lt;/strong&gt; Use &lt;code&gt;generateText&lt;/code&gt; to send a task and receive a result.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Streaming:&lt;/strong&gt; Consume output through &lt;code&gt;streamText&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MCP configuration:&lt;/strong&gt; Configure MCP servers for the agent's session.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tool-call reporting:&lt;/strong&gt; Expose provider-defined tool-call information through the AI SDK interface.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Process management:&lt;/strong&gt; Start agent processes and manage their lifecycle through the provider.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important architectural nuance is that exposing an agent through the &lt;code&gt;LanguageModel&lt;/code&gt; interface does not turn that agent into a conventional LLM API.&lt;/p&gt;

&lt;p&gt;The provider offers a common application interface, but the underlying system may still execute multi-step workflows, use external tools, and manage its own state.&lt;/p&gt;

&lt;p&gt;This abstraction is useful, but it does not make all agents behave identically.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://ai-sdk.dev/providers/community-providers/acp" rel="noopener noreferrer"&gt;AI SDK — ACP Community Provider&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  4. How to Integrate ACP
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Step 1: Install the dependencies
&lt;/h3&gt;

&lt;p&gt;For a Node.js and TypeScript project, install the AI SDK and the ACP provider:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pnpm add ai @mcpc-tech/acp-ai-provider
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Installing the package is only the first step. The selected ACP agent must also be installed, configured, and authenticated according to its own requirements.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Create an ACP provider
&lt;/h3&gt;

&lt;p&gt;The following example uses Gemini CLI with its documented ACP command-line option.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;createACPProvider&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@mcpc-tech/acp-ai-provider'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;generateText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'ai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;provider&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;createACPProvider&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'gemini'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'--experimental-acp'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cwd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cwd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;mcpServers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;languageModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Analyze this codebase and identify its main components.'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Let's break down the configuration:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;command&lt;/code&gt; specifies the executable used to start the agent.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;args&lt;/code&gt; passes command-line arguments to that executable.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;cwd&lt;/code&gt; defines the working directory for the agent.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;mcpServers&lt;/code&gt; configures the MCP servers available to the session.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;provider.languageModel()&lt;/code&gt; exposes the configured agent through the AI SDK's model interface.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, setting &lt;code&gt;cwd&lt;/code&gt; to a repository directory allows the agent to work in that context, subject to its capabilities and permissions.&lt;/p&gt;

&lt;p&gt;The command-line arguments are agent-specific. The example assumes a compatible Gemini CLI version is installed and supports the documented ACP mode.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Stream the response
&lt;/h3&gt;

&lt;p&gt;For a long-running coding task, waiting for the entire response before displaying anything can produce a poor user experience.&lt;/p&gt;

&lt;p&gt;The AI SDK provides &lt;code&gt;streamText&lt;/code&gt; to consume output incrementally:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;createACPProvider&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'@mcpc-tech/acp-ai-provider'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;streamText&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s1"&gt;'ai'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;provider&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;createACPProvider&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'claude-code-acp'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cwd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cwd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;mcpServers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;textStream&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;streamText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;languageModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'Review the error handling in this repository.'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="n"&gt;textStream&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stdout&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;chunk&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This pattern lets the application consume text as it becomes available.&lt;/p&gt;

&lt;p&gt;However, text streaming and agent execution events are not the same thing. A coding interface may also need to display tool calls, file changes, permission requests, and execution progress.&lt;/p&gt;

&lt;p&gt;Before building that experience, inspect the provider's supported event mapping and determine which events are exposed through the version you use.&lt;/p&gt;

&lt;h3&gt;
  
  
  Observability and Event Correlation
&lt;/h3&gt;

&lt;p&gt;Text streaming is useful for presenting generated output, but it is not enough to understand the full execution lifecycle of a coding agent.&lt;/p&gt;

&lt;p&gt;For reliable observability, the application should correlate the user request with the ACP session and the individual tool executions. A practical tracing model should capture:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Application trace or request ID:&lt;/strong&gt; Identifies the end-to-end operation in the application.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ACP session ID:&lt;/strong&gt; Identifies the agent session associated with that operation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ACP tool-call ID:&lt;/strong&gt; Identifies an individual tool execution reported by the agent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AI SDK tool-call ID:&lt;/strong&gt; Identifies the corresponding tool call in the AI SDK layer, when exposed by the provider.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Execution events:&lt;/strong&gt; Records tool start/completion, permission requests and decisions, progress updates, errors, and timestamps.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not assume that these identifiers are interchangeable or that every ACP event is automatically forwarded through the AI SDK's &lt;code&gt;textStream&lt;/code&gt;. Where the provider exposes both ACP and AI SDK identifiers, keep an explicit mapping between them. Where required events or identifiers are not exposed, instrument the provider or ACP client boundary rather than trying to reconstruct the execution history from generated text.&lt;/p&gt;

&lt;p&gt;This distinction matters in practice: without event-level correlation, a failed tool execution or a pending permission request can appear in the frontend as an ordinary model response. Preserving the execution context makes failures traceable and gives the application enough information to present agent activity accurately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Provide MCP servers to the agent
&lt;/h3&gt;

&lt;p&gt;ACP does not automatically grant an agent access to GitLab, a database, or another internal service. Those capabilities need to be configured and made available to the agent.&lt;/p&gt;

&lt;p&gt;For example, the following configuration provides a filesystem MCP server restricted to &lt;code&gt;/tmp&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;provider&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;createACPProvider&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'gemini'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'--experimental-acp'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;cwd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cwd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;mcpServers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'stdio'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'filesystem'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'npx'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
          &lt;span class="s1"&gt;'-y'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="s1"&gt;'@modelcontextprotocol/server-filesystem'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="s1"&gt;'/tmp'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This follows the provider documentation's MCP configuration pattern. Verify the selected MCP server's current package, installation instructions, and compatibility before using it.&lt;/p&gt;

&lt;p&gt;A few details are worth keeping in mind:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The agent must support the MCP configuration and transport being used.&lt;/li&gt;
&lt;li&gt;Filesystem and other tool permissions should be restricted to the minimum required.&lt;/li&gt;
&lt;li&gt;Providing an MCP server does not replace authentication, authorization, or execution isolation.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is also a distinction between configuring MCP tools for the agent and exposing tool calls through the AI SDK.&lt;/p&gt;

&lt;p&gt;The provider documentation shows passing &lt;code&gt;provider.tools&lt;/code&gt; to AI SDK functions when tool-call information should be available through the SDK interface:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generateText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;languageModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'List the files in the configured workspace.'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here, &lt;code&gt;provider.tools&lt;/code&gt; refers to the provider-defined tools. It is not a replacement for configuring the MCP servers that supply capabilities to the agent.&lt;/p&gt;

&lt;p&gt;This distinction becomes important when designing tool-call interfaces, logging agent activity, or building an interactive coding environment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reference:&lt;/strong&gt; &lt;a href="https://ai-sdk.dev/providers/community-providers/acp" rel="noopener noreferrer"&gt;AI SDK — ACP Provider: Examples and Tool Usage&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Advantages of Using ACP
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Reuse existing coding agents
&lt;/h3&gt;

&lt;p&gt;Building a capable coding agent from scratch involves much more than prompt engineering. It may require repository context management, file operations, command execution, tool orchestration, and recovery from failures.&lt;/p&gt;

&lt;p&gt;Integrating an existing ACP-compatible agent can reduce the amount of infrastructure we need to implement ourselves.&lt;/p&gt;

&lt;p&gt;This is especially useful when we want to experiment with coding workflows without immediately committing to maintaining our own full agent runtime.&lt;/p&gt;

&lt;h3&gt;
  
  
  Reduce client-agent integration overhead
&lt;/h3&gt;

&lt;p&gt;ACP introduces a common communication interface for compatible clients and agents.&lt;/p&gt;

&lt;p&gt;Instead of writing a completely different integration for every combination, an application can build against the protocol and add support for compatible agents.&lt;/p&gt;

&lt;p&gt;There will still be agent-specific configuration and behavioral differences, but the common protocol can reduce duplicated integration work.&lt;/p&gt;

&lt;h3&gt;
  
  
  Build our own product experience
&lt;/h3&gt;

&lt;p&gt;We do not necessarily have to expose developers to the agent's original terminal or editor interface.&lt;/p&gt;

&lt;p&gt;We can build a custom interface in a web application, desktop client, or internal developer platform.&lt;/p&gt;

&lt;p&gt;Possible features include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Task creation and tracking&lt;/li&gt;
&lt;li&gt;Streaming output and progress indicators&lt;/li&gt;
&lt;li&gt;Repository and session selection&lt;/li&gt;
&lt;li&gt;Permission-request dialogs&lt;/li&gt;
&lt;li&gt;Tool-call visibility&lt;/li&gt;
&lt;li&gt;File-change summaries&lt;/li&gt;
&lt;li&gt;Integration with existing code review workflows&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are application features built on top of the protocol, not turnkey features supplied by ACP itself.&lt;/p&gt;

&lt;h3&gt;
  
  
  Reuse MCP-based capabilities
&lt;/h3&gt;

&lt;p&gt;Organizations that already maintain MCP servers may be able to reuse those integrations with compatible agents.&lt;/p&gt;

&lt;p&gt;For example, an agent could use one set of configured tools to retrieve a merge request, another to read internal documentation, and another to run approved operations.&lt;/p&gt;

&lt;p&gt;The benefit is architectural reuse: tools can be maintained as a separate capability layer rather than being embedded into every agent integration.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Limitations and Trade-offs
&lt;/h2&gt;

&lt;p&gt;ACP simplifies communication, but it does not eliminate the complexity of operating coding agents.&lt;/p&gt;

&lt;h3&gt;
  
  
  The provider is community-maintained
&lt;/h3&gt;

&lt;p&gt;The AI SDK ACP integration is a community provider, not a first-party Vercel model provider.&lt;/p&gt;

&lt;p&gt;Being listed in the AI SDK documentation does not mean it has the same ownership, support commitments, or release lifecycle as a first-party package.&lt;/p&gt;

&lt;p&gt;Before adopting it in production, evaluate maintenance activity, release compatibility, issue response, and the cost of maintaining a fallback or internal adapter.&lt;/p&gt;

&lt;h3&gt;
  
  
  Model selection is currently limited
&lt;/h3&gt;

&lt;p&gt;The provider documentation states that it exposes a single language model representing the configured agent and does not currently support selecting a specific underlying model through its model interface.&lt;/p&gt;

&lt;p&gt;This matters in environments where model choice is a platform-level concern.&lt;/p&gt;

&lt;p&gt;For example, an organization might require specific models for latency, cost, data-handling, or compliance reasons. If model selection is controlled by the agent's own configuration, that control may not be available through the ACP provider itself.&lt;/p&gt;

&lt;p&gt;The current limitations should be checked again before adoption because community provider features can change.&lt;/p&gt;

&lt;h3&gt;
  
  
  Process lifecycle and concurrency need attention
&lt;/h3&gt;

&lt;p&gt;The documented provider runs in Node.js environments with child-process capabilities. Its documentation also notes that each language model instance spawns a new agent process.&lt;/p&gt;

&lt;p&gt;That creates operational questions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;When are agent processes started and terminated?&lt;/li&gt;
&lt;li&gt;How are timeouts and cancellations handled?&lt;/li&gt;
&lt;li&gt;What happens if an agent exits unexpectedly?&lt;/li&gt;
&lt;li&gt;Can sessions be resumed, and under what conditions?&lt;/li&gt;
&lt;li&gt;How are concurrent tasks isolated?&lt;/li&gt;
&lt;li&gt;What limits apply to CPU, memory, and execution time?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Creating a new agent process for every request without a lifecycle strategy can lead to unnecessary startup overhead, resource consumption, and operational instability.&lt;/p&gt;

&lt;p&gt;For a production application, process management should be part of the design rather than an implementation detail left to chance.&lt;/p&gt;

&lt;h3&gt;
  
  
  ACP is not a sandbox
&lt;/h3&gt;

&lt;p&gt;An agent may read files, execute commands, and invoke tools. ACP standardizes the communication involved, but the protocol alone does not isolate those operations.&lt;/p&gt;

&lt;p&gt;The effective security boundary depends on the execution environment, the agent, its tool configuration, and the permissions granted to it.&lt;/p&gt;

&lt;p&gt;A production implementation should consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Isolated workspaces or containers&lt;/li&gt;
&lt;li&gt;Least-privilege filesystem and terminal access&lt;/li&gt;
&lt;li&gt;Explicit approval for sensitive operations&lt;/li&gt;
&lt;li&gt;Secure credential handling&lt;/li&gt;
&lt;li&gt;Auditable command and file-change records&lt;/li&gt;
&lt;li&gt;Resource limits and execution timeouts&lt;/li&gt;
&lt;li&gt;Cancellation and recovery behavior&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The client should not assume that every agent implements the same permission model or supports the same capabilities.&lt;/p&gt;

&lt;h3&gt;
  
  
  Compatibility does not mean identical behavior
&lt;/h3&gt;

&lt;p&gt;Two ACP-compatible agents can differ in supported features, authentication methods, tools, session handling, and their interpretation of a task.&lt;/p&gt;

&lt;p&gt;ACP reduces communication-level coupling, but it does not guarantee identical results when switching agents.&lt;/p&gt;

&lt;p&gt;Applications should define a common baseline of supported capabilities and make agent-specific differences explicit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References:&lt;/strong&gt; &lt;a href="https://ai-sdk.dev/providers/community-providers/acp" rel="noopener noreferrer"&gt;AI SDK — ACP Provider Limitations&lt;/a&gt; · &lt;a href="https://agentclientprotocol.com/protocol/v1/overview" rel="noopener noreferrer"&gt;ACP Protocol Overview&lt;/a&gt; · &lt;a href="https://agentclientprotocol.com/protocol/v1/security-considerations" rel="noopener noreferrer"&gt;ACP Security Considerations&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  7. What Can an AI Chapter Lead Contribute?
&lt;/h2&gt;

&lt;p&gt;The most valuable outcome is not simply introducing another protocol. It is using the protocol to establish a repeatable approach to agent integration, security, and developer experience across teams.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Establish a shared agent integration layer
&lt;/h3&gt;

&lt;p&gt;Without a common approach, teams may implement their own agent lifecycle management, streaming, permission handling, and error recovery.&lt;/p&gt;

&lt;p&gt;A shared integration layer can provide common abstractions for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agent configuration and selection&lt;/li&gt;
&lt;li&gt;Session creation and cleanup&lt;/li&gt;
&lt;li&gt;Output streaming&lt;/li&gt;
&lt;li&gt;Cancellation and timeouts&lt;/li&gt;
&lt;li&gt;Permission handling&lt;/li&gt;
&lt;li&gt;Error reporting and retries&lt;/li&gt;
&lt;li&gt;Logging and observability&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This allows feature teams to focus on their workflows instead of rebuilding the same integration infrastructure.&lt;/p&gt;

&lt;p&gt;The abstraction should not hide meaningful differences between agents. It should provide a common baseline while allowing explicit capability checks and agent-specific configuration.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Define when to use ACP, MCP, or a model provider
&lt;/h3&gt;

&lt;p&gt;A short architecture guide can help teams decide which technology belongs at which layer.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Requirement&lt;/th&gt;
&lt;th&gt;Primary consideration&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Generate text or structured output from a model&lt;/td&gt;
&lt;td&gt;AI SDK model provider&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Give an agent access to internal APIs or tools&lt;/td&gt;
&lt;td&gt;MCP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Connect a coding agent to a client application&lt;/td&gt;
&lt;td&gt;ACP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stream task output and supported progress events&lt;/td&gt;
&lt;td&gt;ACP integration and AI SDK&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enforce permissions and restrict execution&lt;/td&gt;
&lt;td&gt;Application policies and runtime isolation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These are architectural guidelines, not mutually exclusive choices. A single system may use all three.&lt;/p&gt;

&lt;p&gt;The objective is to avoid introducing a protocol simply because it is available. Every component should have a clear responsibility.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Define a secure execution standard
&lt;/h3&gt;

&lt;p&gt;Coding agents introduce security concerns beyond those of a read-only chat interface.&lt;/p&gt;

&lt;p&gt;Before teams use agents to make changes or run commands, establish standards for workspace isolation, tool permissions, credential management, user approvals, and auditability.&lt;/p&gt;

&lt;p&gt;For instance, an agent that reviews a repository may only need read access. An agent that fixes a bug may require write access and permission to run tests. An agent that interacts with a shared environment should have a much stricter policy.&lt;/p&gt;

&lt;p&gt;Permission decisions should be enforced by trusted application and runtime components rather than relying solely on an agent's instructions.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Run a measurable pilot
&lt;/h3&gt;

&lt;p&gt;A working demo shows that an integration is possible. It does not show that the integration is useful or production-ready.&lt;/p&gt;

&lt;p&gt;A good pilot would be a bounded merge request review and correction workflow:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The agent inspects a repository and identifies potential issues.&lt;/li&gt;
&lt;li&gt;It explains the relevant files and findings.&lt;/li&gt;
&lt;li&gt;It proposes a change or modifies permitted files.&lt;/li&gt;
&lt;li&gt;It runs the available tests.&lt;/li&gt;
&lt;li&gt;The application presents a summary of the changes and test results.&lt;/li&gt;
&lt;li&gt;A developer reviews the result before it is merged.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The pilot should be compared with the existing workflow using the same tasks wherever possible.&lt;/p&gt;

&lt;p&gt;Useful metrics include task completion rate, end-to-end duration, process startup overhead, tool-call failures, resource consumption, human intervention, and the number of changes accepted after review.&lt;/p&gt;

&lt;p&gt;Security checks and regression tests should be treated as acceptance criteria, not optional improvements.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Create a reference implementation and share learnings
&lt;/h3&gt;

&lt;p&gt;Instead of asking each team to interpret protocol documentation independently, provide a small reference application.&lt;/p&gt;

&lt;p&gt;It should demonstrate provider configuration, MCP integration, streaming, process lifecycle management, permission handling, and failure scenarios.&lt;/p&gt;

&lt;p&gt;Teams can then reuse the implementation and contribute improvements.&lt;/p&gt;

&lt;p&gt;This approach makes the learning transferable: the organization develops an integration standard and a practical understanding of where the technology helps, where it introduces complexity, and which limitations still need work.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Questions to Answer Before Production Adoption
&lt;/h2&gt;

&lt;p&gt;Before choosing ACP for a production workflow, I would evaluate the following:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Does our selected agent implement ACP, and which version and capabilities does it support?&lt;/li&gt;
&lt;li&gt;Does the provider expose the events and controls required by our application?&lt;/li&gt;
&lt;li&gt;Can application trace IDs, ACP session IDs, and ACP/AI SDK tool-call IDs be correlated end to end?&lt;/li&gt;
&lt;li&gt;Where does the agent process run, and what resources can it access?&lt;/li&gt;
&lt;li&gt;How are credentials, permissions, and approval decisions managed?&lt;/li&gt;
&lt;li&gt;How are sessions, cancellation, timeouts, and process termination handled?&lt;/li&gt;
&lt;li&gt;How are concurrent users and tasks isolated?&lt;/li&gt;
&lt;li&gt;How will we measure reliability, cost, performance, and developer productivity?&lt;/li&gt;
&lt;li&gt;What is our plan if the community provider becomes incompatible with a new AI SDK release?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The key is to separate protocol compatibility from operational readiness.&lt;/p&gt;

&lt;p&gt;A compatible agent may be easy to connect but still require substantial engineering around security, process management, observability, and user experience.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;ACP is worth evaluating when we want to integrate coding agents into our own applications without building every agent-specific communication layer from scratch.&lt;/p&gt;

&lt;p&gt;The Vercel AI SDK's community provider makes this approach accessible to TypeScript applications by exposing ACP-compatible agents through familiar APIs.&lt;/p&gt;

&lt;p&gt;But ACP should not be treated as a universal replacement for model APIs or as an alternative to MCP. Its value is more specific: it standardizes communication between an agent and the client that uses it, while MCP can provide the tools an agent needs.&lt;/p&gt;

&lt;p&gt;For an AI Chapter Lead, the opportunity is to turn that capability into a reusable engineering practice: define clear architectural boundaries, standardize secure integration patterns, run controlled pilots, and evaluate outcomes using measurable criteria.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;My recommended starting point:&lt;/strong&gt; choose one bounded coding workflow, build a small ACP-based reference implementation, and compare it with the existing approach. Make the production decision based on compatibility, operational effort, security, and measured results—not simply on whether the demo works.&lt;/p&gt;




&lt;h2&gt;
  
  
  References and Further Reading
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;a href="https://ai-sdk.dev/providers/community-providers/acp" rel="noopener noreferrer"&gt;AI SDK — ACP Community Provider&lt;/a&gt; — Installation, configuration, examples, tool usage, and documented limitations.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://agentclientprotocol.com/get-started/introduction" rel="noopener noreferrer"&gt;Agent Client Protocol — Introduction&lt;/a&gt; — ACP's purpose and the interoperability problem it addresses.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://agentclientprotocol.com/get-started/architecture" rel="noopener noreferrer"&gt;Agent Client Protocol — Architecture&lt;/a&gt; — Client-agent communication and MCP integration.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://agentclientprotocol.com/protocol/v1/overview" rel="noopener noreferrer"&gt;Agent Client Protocol — Protocol Overview&lt;/a&gt; — Sessions, messages, progress updates, and permission requests.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://agentclientprotocol.com/protocol/v1/security-considerations" rel="noopener noreferrer"&gt;Agent Client Protocol — Security Considerations&lt;/a&gt; — Security boundaries and integration considerations.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/agentclientprotocol/typescript-sdk" rel="noopener noreferrer"&gt;Agent Client Protocol — TypeScript SDK&lt;/a&gt; — The TypeScript SDK for building ACP clients and agents.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://modelcontextprotocol.io/" rel="noopener noreferrer"&gt;Model Context Protocol Documentation&lt;/a&gt; — MCP concepts, tools, and integration patterns.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/mcpc-tech/mcpc" rel="noopener noreferrer"&gt;MCPC — GitHub Repository&lt;/a&gt; — The project associated with the ACP AI SDK provider.&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>productivity</category>
      <category>beginners</category>
    </item>
  </channel>
</rss>
