<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: mech.app</title>
    <description>The latest articles on DEV Community by mech.app (@mech_app_ai).</description>
    <link>https://dev.to/mech_app_ai</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4089443%2Ffe65488a-e6a2-4e18-b521-f1a296a102be.png</url>
      <title>DEV Community: mech.app</title>
      <link>https://dev.to/mech_app_ai</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mech_app_ai"/>
    <language>en</language>
    <item>
      <title>Sub-Workflow Decomposition in n8n: How Forty-Node Automation Sprawl Forces Tool Boundary Decisions</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Mon, 16 Nov 2026 00:08:09 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/sub-workflow-decomposition-in-n8n-how-forty-node-automation-sprawl-forces-tool-boundary-decisions-1gm</link>
      <guid>https://dev.to/mech_app_ai/sub-workflow-decomposition-in-n8n-how-forty-node-automation-sprawl-forces-tool-boundary-decisions-1gm</guid>
      <description>&lt;p&gt;Automation workflows start small. Three months later you have forty nodes, the same phone-number-cleaning logic copied to four places, and nobody wants to touch it. The solution mirrors programming: extract functions. In n8n, these functions are sub-workflows.&lt;/p&gt;

&lt;p&gt;The real question is not whether to decompose, but where to draw the boundary. Every split creates an interface contract. Get it wrong and you trade spaghetti nodes for spaghetti state-passing. Get it right and you have reusable components that double as agent tools.&lt;/p&gt;

&lt;h2&gt;
  
  
  When to Split: Signals and Anti-Patterns
&lt;/h2&gt;

&lt;p&gt;Split when you see:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The same node sequence in multiple workflows&lt;/li&gt;
&lt;li&gt;A logical unit you can name in one sentence ("upsert contact in CRM")&lt;/li&gt;
&lt;li&gt;A step you need to test independently&lt;/li&gt;
&lt;li&gt;An action an AI agent should invoke on demand&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not split:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Two-node sequences used once (pure overhead)&lt;/li&gt;
&lt;li&gt;Tightly coupled state that requires the parent's full context&lt;/li&gt;
&lt;li&gt;Logic that changes every time the parent changes&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The boundary test: can you write a clear input schema and output contract? If not, the decomposition is premature.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sub-Workflow Architecture in n8n
&lt;/h2&gt;

&lt;p&gt;n8n sub-workflows are not function calls. They are separate workflow executions with their own runtime context. The parent invokes them via the "Execute Workflow" node. The child starts with an "Execute Workflow Trigger" node.&lt;/p&gt;

&lt;h3&gt;
  
  
  State Passing Model
&lt;/h3&gt;

&lt;p&gt;The parent sends a JSON payload. The child receives it as trigger data. No shared memory, no implicit context. This forces explicit contracts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Parent workflow (Execute Workflow node):&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"John Doe"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"phone"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"+1-555-0123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"john@example.com"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Child workflow (Execute Workflow Trigger):&lt;/strong&gt;&lt;br&gt;
Define expected fields in the trigger configuration:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;name&lt;/code&gt; (string)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;phone&lt;/code&gt; (string)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;email&lt;/code&gt; (string)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Declaring fields serves two purposes: the parent sees what to send, and an AI agent consuming this as a tool gets a typed interface.&lt;/p&gt;
&lt;h3&gt;
  
  
  Return Value Contract
&lt;/h3&gt;

&lt;p&gt;The child's final node output becomes the parent's input. Use a Set node or Code node to shape the response explicitly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;contactId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"12345"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;normalized&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;phone&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"+15550123"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"john@example.com"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;created&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The parent receives this as &lt;code&gt;$json&lt;/code&gt; from the Execute Workflow node. No magic, no hidden state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Error Propagation and Failure Modes
&lt;/h2&gt;

&lt;p&gt;Sub-workflow errors do not automatically fail the parent. You must configure error handling at the Execute Workflow node.&lt;/p&gt;

&lt;h3&gt;
  
  
  Three Error Strategies
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Strategy&lt;/th&gt;
&lt;th&gt;Behavior&lt;/th&gt;
&lt;th&gt;Use When&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Continue on Fail&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Parent proceeds, child error in &lt;code&gt;$json.error&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Optional enrichment (e.g., geocoding)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Stop and Error&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Parent halts, execution marked failed&lt;/td&gt;
&lt;td&gt;Critical path (e.g., payment processing)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Retry with Backoff&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Re-execute child up to N times&lt;/td&gt;
&lt;td&gt;Transient API failures&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The default is "Stop and Error." For agent tools, this is often wrong. An agent calling a sub-workflow should receive a structured error response, not a stack trace.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pattern for agent-safe errors:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;In the child workflow, wrap risky nodes in an Error Trigger branch. Return a well-formed error object:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;success&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"INVALID_PHONE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Phone number format not recognized"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;input&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;$input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;phone&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The parent (or agent) can now handle errors programmatically.&lt;/p&gt;

&lt;h2&gt;
  
  
  Exposing Sub-Workflows as Agent Tools
&lt;/h2&gt;

&lt;p&gt;n8n sub-workflows can become agent tools via webhook triggers. Replace the "Execute Workflow Trigger" with a "Webhook" trigger. The sub-workflow now has an HTTP endpoint.&lt;/p&gt;

&lt;h3&gt;
  
  
  Tool Interface Pattern
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Webhook trigger&lt;/strong&gt; receives POST with JSON body&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validate input&lt;/strong&gt; (Code node or IF node)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Execute logic&lt;/strong&gt; (your existing nodes)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Return structured JSON&lt;/strong&gt; (Respond to Webhook node)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The webhook URL becomes the tool endpoint. The input schema becomes the tool's parameter definition.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example tool descriptor for an AI agent:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"normalize_contact"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Clean and validate contact information"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"object"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"properties"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"phone"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"email"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"required"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"endpoint"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://n8n.example.com/webhook/normalize-contact"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The agent calls the endpoint. n8n executes the sub-workflow. The response is the tool result.&lt;/p&gt;

&lt;h2&gt;
  
  
  Authentication and Secret Management
&lt;/h2&gt;

&lt;p&gt;Sub-workflows invoked by agents need authentication. Webhook triggers in n8n support:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Header-based auth&lt;/strong&gt; (API key in &lt;code&gt;X-API-Key&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Basic auth&lt;/strong&gt; (username/password)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No auth&lt;/strong&gt; (dangerous, only for internal networks)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For agent tools, use header-based auth. Store the API key in the agent's credential vault. The sub-workflow validates the key before executing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Validation pattern (Code node):&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;providedKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'x-api-key'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;validKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;WEBHOOK_API_KEY&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;providedKey&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="n"&gt;validKey&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;success&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"UNAUTHORIZED"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Invalid API key"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nv"&gt;$input&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Secrets (CRM credentials, API tokens) should live in n8n's credential store, not hardcoded in nodes. Sub-workflows inherit credentials from their execution context.&lt;/p&gt;

&lt;h2&gt;
  
  
  Versioning and Testing Strategy
&lt;/h2&gt;

&lt;p&gt;Sub-workflows are live code. Changing one affects every caller. Version them explicitly.&lt;/p&gt;

&lt;h3&gt;
  
  
  Versioning Approaches
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;URL versioning&lt;/strong&gt;: &lt;code&gt;/webhook/normalize-contact/v1&lt;/code&gt;, &lt;code&gt;/webhook/normalize-contact/v2&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Input versioning&lt;/strong&gt;: Accept a &lt;code&gt;version&lt;/code&gt; parameter, branch logic internally&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Duplicate workflows&lt;/strong&gt;: Copy the sub-workflow, rename it, update callers gradually&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;URL versioning is cleanest for agent tools. The agent specifies which version to call. Old versions stay live until all callers migrate.&lt;/p&gt;

&lt;h3&gt;
  
  
  Testing Sub-Workflows
&lt;/h3&gt;

&lt;p&gt;n8n has no built-in unit test framework. Test manually:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Direct execution&lt;/strong&gt;: Click "Execute Workflow" in the editor with sample input&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Webhook testing&lt;/strong&gt;: &lt;code&gt;curl&lt;/code&gt; the endpoint with test payloads&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Parent workflow test mode&lt;/strong&gt;: Run the parent with known inputs, inspect sub-workflow execution logs&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For production sub-workflows, maintain a test parent workflow that exercises all code paths. Run it after every change.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decomposition Decision Table
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Decompose?&lt;/th&gt;
&lt;th&gt;Rationale&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Phone normalization logic used in 4 workflows&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;High reuse, clear boundary&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CRM upsert with 12 nodes, used once&lt;/td&gt;
&lt;td&gt;Maybe&lt;/td&gt;
&lt;td&gt;Clear boundary, but no reuse yet&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data transformation tightly coupled to parent state&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Forced state-passing overhead&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent needs to trigger a 5-step process&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Agent tool interface requirement&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Error handling that varies by caller&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Contract too context-dependent&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Observability and Debugging
&lt;/h2&gt;

&lt;p&gt;Sub-workflow executions appear as separate entries in n8n's execution log. You cannot see the parent-child relationship in the UI. This makes debugging harder.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Debugging patterns:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Correlation IDs&lt;/strong&gt;: Pass a unique ID from parent to child, log it in both&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Execution linking&lt;/strong&gt;: Store the parent execution ID in the child's output&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Structured logging&lt;/strong&gt;: Use Code nodes to log key state transitions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Example correlation pattern:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Parent workflow&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;correlationId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$execution&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s1"&gt;'-'&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;correlationId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;inputData&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// Child workflow (first node)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;correlationId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;correlationId&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`[${correlationId}] Starting normalization`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Grep logs by correlation ID to trace execution across workflows.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use sub-workflows when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You have clear, stable interfaces (input schema, output contract)&lt;/li&gt;
&lt;li&gt;The logic is reused across workflows or exposed to agents&lt;/li&gt;
&lt;li&gt;You need independent testing or versioning&lt;/li&gt;
&lt;li&gt;The decomposition reduces cognitive load more than it adds coordination overhead&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid sub-workflows when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The boundary is artificial (splitting for the sake of splitting)&lt;/li&gt;
&lt;li&gt;State-passing becomes more complex than the original monolith&lt;/li&gt;
&lt;li&gt;The child needs deep access to parent context&lt;/li&gt;
&lt;li&gt;You are optimizing for execution speed (sub-workflows add latency)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The real cost is not the extra node. It is the interface contract you must now maintain. If that contract clarifies your system, decompose. If it obscures it, inline the logic.&lt;/p&gt;




&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/fluxintelligent/tutoriel-n8n-decouper-un-gros-workflow-en-sous-workflows-reutilisables-et-les-donner-comme-2me1"&gt;Original tutorial on DEV.to&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>architecture</category>
      <category>automation</category>
      <category>systemdesign</category>
    </item>
    <item>
      <title>ChatGPT Wallet: How OpenAI Built Agent Payment Rails with Card Network Primitives</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Sun, 15 Nov 2026 20:08:01 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/chatgpt-wallet-how-openai-built-agent-payment-rails-with-card-network-primitives-42ol</link>
      <guid>https://dev.to/mech_app_ai/chatgpt-wallet-how-openai-built-agent-payment-rails-with-card-network-primitives-42ol</guid>
      <description>&lt;p&gt;OpenAI shipped ChatGPT Wallet on November 13, 2026. It's the first production payment primitive for agents from a major LLM provider, live in ChatGPT Work, Codex, and dots. The stated design is straightforward: agents can complete purchases with user approval and existing card protections.&lt;/p&gt;

&lt;p&gt;This is a boundary problem. Agents can reason about products, compare prices, and construct purchase intent, but they cannot execute financial mutations without explicit user authorization. The infrastructure question is how to enforce that boundary while keeping agents useful.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Approval Gate Pattern
&lt;/h2&gt;

&lt;p&gt;ChatGPT Wallet implements a synchronous approval gate between agent reasoning and payment execution. The agent proposes a transaction, the system surfaces it to the user, and the charge only proceeds after confirmation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Control flow:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Agent uses tools to search products and construct purchase intent (no financial mutation)&lt;/li&gt;
&lt;li&gt;System generates approval request with amount, merchant, and context&lt;/li&gt;
&lt;li&gt;User explicitly approves or declines&lt;/li&gt;
&lt;li&gt;Approved transactions flow through standard card network rails (Visa/Mastercard tokenization and settlement)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The approval gate is not advisory. The agent's tool call returns a pending state until the user acts. The agent cannot pre-approve future charges, cannot modify amounts after confirmation, and cannot bypass the gate through retry logic or prompt engineering.&lt;/p&gt;

&lt;p&gt;This enforcement happens at the orchestration layer, not the agent layer. The agent sees the approval gate as a hard constraint in its tool schema, similar to how it sees API rate limits or permission boundaries.&lt;/p&gt;

&lt;h2&gt;
  
  
  Card Network Integration Points
&lt;/h2&gt;

&lt;p&gt;OpenAI's announcement emphasizes "existing card protections," which means the system sits on top of traditional payment rails rather than inventing new protocols. The agent-initiated transaction looks like an API purchase to the card network.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Inherited protections:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tokenization&lt;/strong&gt;: Payment credentials are tokenized. The agent never sees raw card numbers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fraud detection&lt;/strong&gt;: Card networks apply standard fraud models to agent-initiated charges.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chargeback rights&lt;/strong&gt;: Users can dispute charges through normal card network dispute flows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;3D Secure&lt;/strong&gt;: When required by merchants, users complete authentication challenges during the approval flow.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The key architectural choice is to treat agent purchases as a new interface to existing payment infrastructure, not as a new payment type. This means ChatGPT Wallet inherits decades of card network security and compliance work (PCI DSS, fraud detection, dispute resolution) without reimplementing it.&lt;/p&gt;

&lt;h2&gt;
  
  
  State Management Across Approval Boundaries
&lt;/h2&gt;

&lt;p&gt;The approval gate introduces asynchronous state that the agent must handle. When the agent proposes a purchase, it enters a waiting state. The conversation continues, but the purchase remains pending until the user acts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;State transitions:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Pending&lt;/strong&gt;: Agent proposed transaction, waiting for user approval&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Approved&lt;/strong&gt;: User confirmed, charge initiated through card network&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Declined by user&lt;/strong&gt;: User rejected the proposal, agent must handle rejection&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Declined by network&lt;/strong&gt;: Card network rejected the charge (insufficient funds, fraud block, expired card)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Partial fulfillment&lt;/strong&gt;: Merchant completed part of the order (requires agent reasoning about next steps)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The agent must reason about these states explicitly. If a card is declined, the agent receives an error signal and can prompt the user to update payment details or suggest alternatives. If the user rejects a proposal, the agent must either propose a different option or abandon the task.&lt;/p&gt;

&lt;p&gt;This state visibility is intentional. The agent's retry logic appears in the conversation. If the agent proposes a second purchase after a decline, the user sees the reasoning and can intervene.&lt;/p&gt;

&lt;h2&gt;
  
  
  Design Trade-offs and Known Constraints
&lt;/h2&gt;

&lt;p&gt;ChatGPT Wallet optimizes for user control over convenience. This creates predictable limitations.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Constraint&lt;/th&gt;
&lt;th&gt;Impact&lt;/th&gt;
&lt;th&gt;Rationale&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Per-transaction approval&lt;/td&gt;
&lt;td&gt;High friction for frequent purchases&lt;/td&gt;
&lt;td&gt;Prevents agents from spending without oversight&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No recurring billing&lt;/td&gt;
&lt;td&gt;Agents can't set up subscriptions&lt;/td&gt;
&lt;td&gt;Each charge requires explicit approval&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Synchronous approval flow&lt;/td&gt;
&lt;td&gt;Adds human latency to agent workflows&lt;/td&gt;
&lt;td&gt;User must be present to approve charges&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Card network dependency&lt;/td&gt;
&lt;td&gt;Limited to Visa/Mastercard rails&lt;/td&gt;
&lt;td&gt;Inherits network downtime and processing delays&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No price negotiation&lt;/td&gt;
&lt;td&gt;Agents accept merchant prices as-is&lt;/td&gt;
&lt;td&gt;Approval gate assumes fixed transaction amounts&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The recurring billing constraint is the most significant. An agent cannot subscribe you to a service without asking every time. If you want an agent to handle monthly bills, you need a different primitive (likely a user-configured standing order outside the agent's control).&lt;/p&gt;

&lt;p&gt;The synchronous approval requirement means agents cannot complete purchases in background workflows. The user must be available to confirm. This works for conversational agents but breaks for fully autonomous task automation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Across OpenAI Products
&lt;/h2&gt;

&lt;p&gt;ChatGPT Wallet is available in three products with slightly different integration patterns:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ChatGPT Work&lt;/strong&gt;: Agents propose purchases during conversations. The approval flow interrupts the chat with a modal or inline confirmation. Organizations can set spending policies and merchant restrictions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Codex&lt;/strong&gt;: Developer agents can call payment tools in code generation workflows. The approval gate still applies, but the context is a code execution environment. The agent must handle the pending state in generated code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;dots&lt;/strong&gt;: Task automation agents can include purchases in multi-step workflows. The workflow pauses at the approval gate and resumes after user action. This creates a human-in-the-loop step in otherwise automated processes.&lt;/p&gt;

&lt;p&gt;The common thread is the approval gate. No matter where the agent runs, the user must confirm the charge. The integration point is always a synchronous approval request, not an async callback or webhook.&lt;/p&gt;

&lt;h2&gt;
  
  
  Security Boundaries and Observability
&lt;/h2&gt;

&lt;p&gt;The approval gate is the primary security boundary, but OpenAI's announcement mentions "existing card protections," which implies additional layers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Expected security controls:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Per-transaction approval&lt;/strong&gt;: Hard requirement, no exceptions&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Audit logs&lt;/strong&gt;: Transaction history visible to users and admins&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Conversation context&lt;/strong&gt;: Agent reasoning (why this product, this merchant, this price) appears in chat history&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Receipt generation&lt;/strong&gt;: Charge confirmations include transaction details&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Observability is built into the conversation interface. The agent's reasoning is visible. The approval request shows full transaction details. The charge confirmation includes a receipt. This transparency helps users catch errors (wrong product, wrong amount, wrong merchant) before approving.&lt;/p&gt;

&lt;p&gt;The audit trail also supports dispute resolution. If a user initiates a chargeback, the conversation logs provide context about the agent's decision-making process.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use ChatGPT Wallet when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You need agents to complete purchases in conversational workflows where users are present&lt;/li&gt;
&lt;li&gt;You want to leverage existing card network protections without building custom payment infrastructure&lt;/li&gt;
&lt;li&gt;You can tolerate per-transaction approval friction&lt;/li&gt;
&lt;li&gt;You need audit trails for compliance or cost management&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid ChatGPT Wallet when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You need agents to handle recurring billing or subscriptions&lt;/li&gt;
&lt;li&gt;You require sub-second transaction latency (approval gates add human delay)&lt;/li&gt;
&lt;li&gt;You're building fully autonomous agents that run without user supervision&lt;/li&gt;
&lt;li&gt;You need agents to negotiate prices or handle complex payment terms&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;ChatGPT Wallet is a first primitive for agent payments, not a complete solution. It handles the common case (agent finds product, user approves, charge executes) and defers the hard cases (subscriptions, partial payments, multi-party transactions). The approval gate pattern is a reasonable starting point for transactional agent tools, but it assumes users will stay engaged with every purchase decision.&lt;/p&gt;




&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://openai.com/index/introducing-chatgpt-wallet" rel="noopener noreferrer"&gt;OpenAI: Introducing ChatGPT Wallet&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>chatgpt</category>
      <category>openai</category>
    </item>
    <item>
      <title>A2UI Protocol: How Agents Stream Layout Data Instead of HTML to Native UI Components</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Sun, 15 Nov 2026 10:07:50 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/a2ui-protocol-how-agents-stream-layout-data-instead-of-html-to-native-ui-components-1e0g</link>
      <guid>https://dev.to/mech_app_ai/a2ui-protocol-how-agents-stream-layout-data-instead-of-html-to-native-ui-components-1e0g</guid>
      <description>&lt;p&gt;Most agent UIs stream markdown or generate HTML strings. A2UI streams layout data that native framework adapters translate into platform components. The agent emits structured events describing buttons, cards, and forms. The client renders them using React, Flutter, SwiftUI, or Jetpack Compose without a WebView or eval().&lt;/p&gt;

&lt;p&gt;This separation creates strict capability boundaries. The agent cannot inject arbitrary code. It composes from a fixed catalog of components defined by the application. The protocol is event-based and optimized for low-latency streaming, making it viable for real-time agent interactions across web and mobile platforms.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Layout Data Instead of Code
&lt;/h2&gt;

&lt;p&gt;Generative UI typically means an LLM writes JSX or HTML, then the client renders it. This works for demos but creates three production problems:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Security boundary&lt;/strong&gt;: Executing agent-generated code requires sandboxing or trusted execution contexts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cross-platform cost&lt;/strong&gt;: An agent that generates React code cannot target Flutter or SwiftUI without retraining or prompt engineering.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Visual consistency&lt;/strong&gt;: Generated code may not respect design systems, accessibility rules, or platform conventions.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A2UI solves this by treating UI as data. The agent outputs a schema-validated stream of layout primitives. The client maps those primitives to native components. The agent never sees platform-specific rendering logic. The client never executes agent-generated code.&lt;/p&gt;

&lt;p&gt;This is the same separation used in server-driven UI frameworks like Litho or Epoxy, but extended to agent-generated content.&lt;/p&gt;

&lt;h2&gt;
  
  
  Protocol Shape
&lt;/h2&gt;

&lt;p&gt;A2UI uses an event-based protocol. The agent emits a stream of JSON events. Each event describes a UI operation: adding a component, updating state, or signaling completion.&lt;/p&gt;

&lt;h3&gt;
  
  
  Core Event Types
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event Type&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;th&gt;Payload&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;component_add&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Append a component to the layout tree&lt;/td&gt;
&lt;td&gt;Component type, props, parent ID&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;component_update&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Modify existing component props&lt;/td&gt;
&lt;td&gt;Component ID, updated props&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;state_sync&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Synchronize application state&lt;/td&gt;
&lt;td&gt;State key, value&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;stream_end&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Signal completion of the UI stream&lt;/td&gt;
&lt;td&gt;Optional metadata&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The agent sends these events over HTTP streaming, WebSocket, or server-sent events. The client maintains a layout tree and applies events incrementally.&lt;/p&gt;

&lt;h3&gt;
  
  
  Example Wire Format
&lt;/h3&gt;

&lt;p&gt;JSONL format (one JSON object per line):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{"type": "component_add", "id": "card-1", "component": "Card", "props": {"title": "Flight Options", "variant": "elevated"}, "parent": "root"}
{"type": "component_add", "id": "button-1", "component": "Button", "props": {"label": "Book Now", "action": "submit_booking"}, "parent": "card-1"}
{"type": "stream_end"}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The client receives these events, validates them against the component catalog schema, and renders them using the native framework adapter.&lt;/p&gt;

&lt;h2&gt;
  
  
  Component Catalog and Capability Boundaries
&lt;/h2&gt;

&lt;p&gt;The application defines a component catalog. This is a schema that lists available components and their props. The agent can only compose from this catalog. If the agent references an undefined component or passes invalid props, the client rejects the event.&lt;/p&gt;

&lt;p&gt;This enforces capability boundaries at the protocol level. The agent cannot:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Render arbitrary HTML or execute JavaScript&lt;/li&gt;
&lt;li&gt;Access components outside the catalog&lt;/li&gt;
&lt;li&gt;Bypass validation rules defined in the schema&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The catalog is versioned. Agents and clients negotiate protocol versions during handshake. This allows safe evolution of the component library without breaking existing agents.&lt;/p&gt;

&lt;h3&gt;
  
  
  Sample Catalog Schema
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;catalog&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;components&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Card&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;props&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'string'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;variant&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'enum'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'flat'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'elevated'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nc"&gt;Button&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;props&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'string'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'string'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;disabled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'boolean'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The agent SDK validates outgoing events against this schema before transmission. The client validates incoming events before rendering. Both sides reject invalid data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Framework Adapters
&lt;/h2&gt;

&lt;p&gt;A2UI provides native adapters for React, Angular, Lit, Flutter, SwiftUI, and Jetpack Compose. Each adapter translates the protocol's layout data into platform-specific components.&lt;/p&gt;

&lt;h3&gt;
  
  
  React Adapter Flow
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Event ingestion&lt;/strong&gt;: The adapter subscribes to the event stream.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tree construction&lt;/strong&gt;: It builds a virtual component tree from &lt;code&gt;component_add&lt;/code&gt; and &lt;code&gt;component_update&lt;/code&gt; events.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reconciliation&lt;/strong&gt;: React's reconciler diffs the tree and updates the DOM.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Event handling&lt;/strong&gt;: User interactions trigger callbacks that send events back to the agent.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The adapter maintains a mapping from A2UI component names to React components. When the agent emits a &lt;code&gt;Button&lt;/code&gt; component, the adapter renders the application's &lt;code&gt;&amp;lt;Button&amp;gt;&lt;/code&gt; component with the provided props.&lt;/p&gt;

&lt;h3&gt;
  
  
  Flutter Adapter Flow
&lt;/h3&gt;

&lt;p&gt;Flutter's adapter works similarly but uses a &lt;code&gt;StreamBuilder&lt;/code&gt; to rebuild the widget tree on each event. The adapter maintains a &lt;code&gt;WidgetTree&lt;/code&gt; model that mirrors the A2UI layout tree. Each event triggers a rebuild of the affected subtree.&lt;/p&gt;

&lt;h3&gt;
  
  
  Type Safety
&lt;/h3&gt;

&lt;p&gt;TypeScript adapters use generated types from the catalog schema. This provides compile-time validation of component props. If the catalog defines &lt;code&gt;Button.label&lt;/code&gt; as a string, the TypeScript adapter enforces that type when rendering.&lt;/p&gt;

&lt;p&gt;Flutter and Swift adapters use code generation to produce strongly typed widget builders from the catalog schema.&lt;/p&gt;

&lt;h2&gt;
  
  
  Streaming and Latency
&lt;/h2&gt;

&lt;p&gt;The protocol is optimized for low-latency streaming. Agents can emit events as soon as they decide on a component, without waiting for the full layout to be computed. This enables progressive rendering.&lt;/p&gt;

&lt;p&gt;For example, an agent booking a flight might:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Emit a loading spinner immediately&lt;/li&gt;
&lt;li&gt;Stream flight options as they are retrieved from the search API&lt;/li&gt;
&lt;li&gt;Emit a booking form once the user selects a flight&lt;/li&gt;
&lt;li&gt;Update the form with validation errors in real time&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each event is independent. The client applies them incrementally. The user sees the UI build up in real time, rather than waiting for a full page load.&lt;/p&gt;

&lt;h3&gt;
  
  
  Backpressure Handling
&lt;/h3&gt;

&lt;p&gt;The protocol supports backpressure. If the client cannot keep up with the event stream, it signals the agent to slow down. This prevents memory exhaustion on mobile devices or slow networks.&lt;/p&gt;

&lt;p&gt;The agent SDK exposes a &lt;code&gt;canSend()&lt;/code&gt; method that checks the client's readiness before emitting the next event.&lt;/p&gt;

&lt;h2&gt;
  
  
  State Management
&lt;/h2&gt;

&lt;p&gt;A2UI separates layout state from application state. Layout state is the component tree. Application state is the data the agent and client share (user selections, form inputs, API responses).&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;state_sync&lt;/code&gt; event synchronizes application state. When the user selects a flight, the client sends a &lt;code&gt;state_sync&lt;/code&gt; event to the agent. The agent updates its internal state and may emit new layout events in response.&lt;/p&gt;

&lt;p&gt;This separation allows the agent to reason about state changes without re-rendering the entire UI. Only the affected components update.&lt;/p&gt;

&lt;h2&gt;
  
  
  Integration with LangGraph and MCP
&lt;/h2&gt;

&lt;p&gt;A2UI integrates with LangGraph for multi-step agent workflows. A LangGraph agent can emit A2UI events at each workflow step, building up the UI as the workflow progresses.&lt;/p&gt;

&lt;p&gt;For example, a customer support agent might:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Emit a search form (initial state)&lt;/li&gt;
&lt;li&gt;Stream search results (after user submits query)&lt;/li&gt;
&lt;li&gt;Emit a detail view (after user selects a result)&lt;/li&gt;
&lt;li&gt;Stream resolution options (after analyzing the issue)&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each step emits A2UI events. The client renders them incrementally. The user sees a cohesive flow without page transitions.&lt;/p&gt;

&lt;p&gt;MCP (Model Context Protocol) integration allows A2UI agents to call external tools and render the results. An agent might call a weather API via MCP, then emit a &lt;code&gt;WeatherCard&lt;/code&gt; component with the response data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability and Debugging
&lt;/h2&gt;

&lt;p&gt;The protocol is self-describing. Each event includes metadata: timestamp, agent ID, session ID. Clients can log the event stream for replay and debugging.&lt;/p&gt;

&lt;p&gt;The A2UI Composer tool (&lt;a href="https://a2ui-project.github.io/composer/" rel="noopener noreferrer"&gt;https://a2ui-project.github.io/composer/&lt;/a&gt;) allows developers to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Manually construct event streams&lt;/li&gt;
&lt;li&gt;Preview how different frameworks render the same events&lt;/li&gt;
&lt;li&gt;Validate catalog schemas&lt;/li&gt;
&lt;li&gt;Test error handling and backpressure&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is critical for debugging agent behavior. Instead of inspecting generated code, you inspect the event stream. You can replay a session, modify events, and see how the UI changes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;Cause&lt;/th&gt;
&lt;th&gt;Mitigation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Invalid component reference&lt;/td&gt;
&lt;td&gt;Agent emits unknown component name&lt;/td&gt;
&lt;td&gt;Client rejects event, logs error, agent retries with fallback&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schema validation failure&lt;/td&gt;
&lt;td&gt;Agent sends props that violate catalog schema&lt;/td&gt;
&lt;td&gt;Client rejects event, agent receives error response, adjusts output&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stream interruption&lt;/td&gt;
&lt;td&gt;Network failure or agent crash&lt;/td&gt;
&lt;td&gt;Client displays last known good state, retries connection&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backpressure overflow&lt;/td&gt;
&lt;td&gt;Agent emits events faster than client can render&lt;/td&gt;
&lt;td&gt;Client buffers events, signals backpressure, agent slows down&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Version mismatch&lt;/td&gt;
&lt;td&gt;Agent uses newer catalog than client supports&lt;/td&gt;
&lt;td&gt;Handshake fails, client requests compatible version or rejects connection&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The protocol is designed for graceful degradation. If the agent emits an invalid event, the client rejects it but continues processing subsequent events. The UI remains functional even if parts of the layout fail to render.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Considerations
&lt;/h2&gt;

&lt;p&gt;A2UI agents typically run server-side. The client connects via HTTP streaming or WebSocket. This allows:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Centralized agent logic&lt;/strong&gt;: No need to deploy agent models to mobile devices.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dynamic catalog updates&lt;/strong&gt;: The server can update the component catalog without client updates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Audit and compliance&lt;/strong&gt;: All agent interactions are logged server-side.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For offline scenarios, the agent can run on-device using a lightweight model. The catalog is bundled with the app. The protocol remains the same, but the transport is in-process rather than over the network.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scaling
&lt;/h3&gt;

&lt;p&gt;Each client session maintains a persistent connection to the agent. This requires connection pooling and load balancing. WebSocket connections are stateful, so sticky sessions or session affinity is required.&lt;/p&gt;

&lt;p&gt;For high-scale deployments, consider:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Connection multiplexing&lt;/strong&gt;: Use HTTP/2 or HTTP/3 to reduce connection overhead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Event batching&lt;/strong&gt;: Group multiple events into a single transmission to reduce round trips.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Edge deployment&lt;/strong&gt;: Run agents at the edge to reduce latency for geographically distributed users.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;Use A2UI when you need strict capability boundaries for agent-generated UIs, or when the same agent must target multiple platforms (web, iOS, Android). The protocol's separation of layout data from rendering logic makes it safer than code generation and more portable than framework-specific solutions.&lt;/p&gt;

&lt;p&gt;Avoid A2UI if your UI is static or if you already have a working server-driven UI framework. The protocol adds complexity: you must define and version a component catalog, implement framework adapters, and manage event streaming. For simple chat interfaces or markdown-based agents, the overhead is not justified.&lt;/p&gt;

&lt;p&gt;A2UI shines when agents compose complex, multi-step UIs that must render natively across platforms. The 17,000+ stars and production adapters for six frameworks suggest real adoption beyond experimentation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/a2ui-project/a2ui" rel="noopener noreferrer"&gt;A2UI GitHub Repository&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://a2ui.org/specification/v1.0-a2ui/" rel="noopener noreferrer"&gt;A2UI Protocol Specification&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://a2ui-project.github.io/composer/" rel="noopener noreferrer"&gt;A2UI Composer (Interactive Demo)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://a2ui.org/reference/renderers/" rel="noopener noreferrer"&gt;Supported Renderers&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>architecture</category>
      <category>llm</category>
      <category>mobile</category>
    </item>
    <item>
      <title>Metal Sandbox: Isolated Execution and Command Scheduling for GPU Coding Agents on Apple Silicon</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Sun, 15 Nov 2026 10:07:49 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/metal-sandbox-isolated-execution-and-command-scheduling-for-gpu-coding-agents-on-apple-silicon-22mh</link>
      <guid>https://dev.to/mech_app_ai/metal-sandbox-isolated-execution-and-command-scheduling-for-gpu-coding-agents-on-apple-silicon-22mh</guid>
      <description>&lt;p&gt;Agents that generate GPU kernels need to compile, test, and iterate on Metal code without monopolizing the host GPU or crashing when concurrent sessions compete for command queues. Metal Sandbox addresses this by isolating file workspaces, scheduling command execution across a pool of Apple Silicon nodes, and migrating eligible commands to idle machines when local capacity is saturated.&lt;/p&gt;

&lt;p&gt;The paper (arXiv 2611.17294v1) reports 1.68× to 2.05× command throughput gains on four Mac mini M4 nodes under concentrated demand, and 22.3% reductions in batch elapsed time when dynamic scheduling replaces static assignment.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Problem: GPU Resource Contention in Agent Loops
&lt;/h2&gt;

&lt;p&gt;A typical GPU coding agent alternates between three phases:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Model inference&lt;/strong&gt; to generate candidate kernel code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Metal compilation&lt;/strong&gt; via &lt;code&gt;xcrun metal&lt;/code&gt; or similar toolchain commands.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Kernel execution&lt;/strong&gt; to measure performance or correctness.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Each phase touches the GPU differently. Inference saturates compute units. Compilation spawns CPU-bound toolchain processes but may trigger shader validation on the GPU. Execution submits command buffers that block until the GPU finishes.&lt;/p&gt;

&lt;p&gt;When multiple agents share one Mac, naive scheduling leads to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Queue head-of-line blocking&lt;/strong&gt;: a long compilation stalls all other agents.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GPU state pollution&lt;/strong&gt;: a crashed kernel can leave the Metal driver in an undefined state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Workspace collisions&lt;/strong&gt;: agents overwrite each other's intermediate &lt;code&gt;.metallib&lt;/code&gt; files or test harnesses.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Architecture: Per-Instance Isolation and Cross-Node Scheduling
&lt;/h2&gt;

&lt;p&gt;Metal Sandbox provides three layers:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Workspace Isolation
&lt;/h3&gt;

&lt;p&gt;Each agent session gets a private directory tree:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/sandbox/agent-&amp;lt;uuid&amp;gt;/
  src/          # generated .metal files
  build/        # compiled .metallib artifacts
  test/         # execution harnesses
  logs/         # stdout/stderr from commands
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;File access is restricted via macOS sandbox profiles. An agent cannot read or write outside its assigned prefix. This prevents one session from corrupting another's build artifacts or leaking intermediate results.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Command-Level Scheduling
&lt;/h3&gt;

&lt;p&gt;Commands fall into two categories:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Parallel&lt;/strong&gt;: compilation, static analysis, or any CPU-bound task that does not require exclusive GPU access.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Serial&lt;/strong&gt;: kernel execution that submits Metal command buffers and waits for GPU completion.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The scheduler maintains one serial execution slot per node. Parallel commands run concurrently up to a configurable limit (typically matching CPU core count). When a serial command arrives and the local slot is occupied, the scheduler can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Queue it locally and wait.&lt;/li&gt;
&lt;li&gt;Migrate it to another node with an idle serial slot.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Migration copies the agent's workspace snapshot to the remote node, executes the command, and returns results (exit code, stdout, stderr, modified files) to the originating agent.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Harbor Integration
&lt;/h3&gt;

&lt;p&gt;Harbor is the workflow orchestration layer. It:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Accepts agent requests (tool calls) and routes them to the scheduler.&lt;/li&gt;
&lt;li&gt;Tracks command dependencies (e.g., execution depends on successful compilation).&lt;/li&gt;
&lt;li&gt;Collects rollout data: command durations, GPU utilization, error rates.&lt;/li&gt;
&lt;li&gt;Triggers automatic verification (unit tests, performance benchmarks) after each kernel generation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Harbor exposes a REST API. Agents submit commands as JSON payloads:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"agent_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"uuid"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"xcrun metal -o kernel.metallib kernel.metal"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"parallel"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"workspace"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"/sandbox/agent-uuid"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The scheduler returns a handle. The agent polls or subscribes to a webhook for completion.&lt;/p&gt;

&lt;h2&gt;
  
  
  Scheduling Primitives and Migration Logic
&lt;/h2&gt;

&lt;p&gt;The scheduler uses a two-level queue:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Queue Type&lt;/th&gt;
&lt;th&gt;Scope&lt;/th&gt;
&lt;th&gt;Policy&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Local&lt;/td&gt;
&lt;td&gt;Per-node&lt;/td&gt;
&lt;td&gt;FIFO for serial, concurrent for parallel&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Global&lt;/td&gt;
&lt;td&gt;Cross-node&lt;/td&gt;
&lt;td&gt;Work-stealing for eligible commands&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Eligibility for migration&lt;/strong&gt;: a command can migrate if:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It has no local file dependencies beyond its workspace snapshot.&lt;/li&gt;
&lt;li&gt;The workspace size is below a threshold (default 100 MB).&lt;/li&gt;
&lt;li&gt;The target node has an idle serial slot or lower parallel load.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Migration cost model&lt;/strong&gt;: the scheduler estimates:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;migration_cost = snapshot_size / network_bandwidth + remote_queue_wait
local_cost = local_queue_wait
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;migration_cost &amp;lt; local_cost&lt;/code&gt;, the command migrates.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Failure handling&lt;/strong&gt;: if a remote execution crashes (GPU hang, kernel panic), the scheduler:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Marks the node as degraded.&lt;/li&gt;
&lt;li&gt;Resubmits the command to another node or the local queue.&lt;/li&gt;
&lt;li&gt;Logs the failure for observability.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The agent sees a retry, not a crash. The host process remains stable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability Hooks
&lt;/h2&gt;

&lt;p&gt;Metal Sandbox exposes metrics via Prometheus endpoints:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;metal_command_duration_seconds{type, node}&lt;/code&gt;: histogram of command execution time.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;metal_gpu_utilization{node}&lt;/code&gt;: sampled GPU active time from &lt;code&gt;ioreg&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;metal_compilation_failures{node}&lt;/code&gt;: counter of failed &lt;code&gt;xcrun metal&lt;/code&gt; invocations.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;metal_migration_count{from, to}&lt;/code&gt;: cross-node command transfers.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Agents can query these metrics to adjust their generation strategy. For example, if compilation failures spike, the agent might reduce code complexity or switch to a simpler kernel template.&lt;/p&gt;

&lt;h2&gt;
  
  
  Experimental Results and Throughput Gains
&lt;/h2&gt;

&lt;p&gt;The paper tests two scenarios:&lt;/p&gt;

&lt;h3&gt;
  
  
  Concentrated Demand (Four Agents, Four Nodes)
&lt;/h3&gt;

&lt;p&gt;Each agent submits a burst of commands. Without scheduling, agents queue locally and wait. With Metal Sandbox:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;1.68× throughput&lt;/strong&gt; for workloads dominated by parallel compilation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;2.05× throughput&lt;/strong&gt; for mixed workloads with frequent serial execution.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The gain comes from parallel commands spreading across nodes and serial commands migrating to idle slots instead of waiting behind local queues.&lt;/p&gt;

&lt;h3&gt;
  
  
  Sustained Load (Sixteen Static Traces)
&lt;/h3&gt;

&lt;p&gt;Replaying recorded agent sessions with four sessions per node (versus one) yields an estimated &lt;strong&gt;2.14× sustained throughput&lt;/strong&gt;. Dynamic scheduling further reduces:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mean tool waiting time by 18.8%&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Batch elapsed time by 22.3%&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These gains assume phase durations remain constant. In practice, remote execution may add network latency, but the paper's controlled LAN setup (1 Gbps Ethernet) keeps migration overhead below 200 ms for typical workspaces.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes and Mitigation
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;Impact&lt;/th&gt;
&lt;th&gt;Mitigation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;GPU hang during kernel execution&lt;/td&gt;
&lt;td&gt;Serial slot blocked until watchdog timeout&lt;/td&gt;
&lt;td&gt;Per-command timeout (default 30s), node marked degraded&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compilation crash (invalid Metal syntax)&lt;/td&gt;
&lt;td&gt;Command fails, agent retries&lt;/td&gt;
&lt;td&gt;Sandbox isolation prevents host crash, logs captured for debugging&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network partition during migration&lt;/td&gt;
&lt;td&gt;Command lost, agent sees timeout&lt;/td&gt;
&lt;td&gt;Resubmit to local queue, mark remote node unavailable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Workspace snapshot corruption&lt;/td&gt;
&lt;td&gt;Execution fails with missing files&lt;/td&gt;
&lt;td&gt;Checksum validation before migration, retry with fresh snapshot&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The scheduler does not attempt distributed consensus. If a node becomes unreachable, commands in flight are resubmitted. Agents tolerate retries because kernel generation is idempotent (same input code produces same compiled artifact).&lt;/p&gt;

&lt;h2&gt;
  
  
  Code Snippet: Submitting a Serial Command with Migration
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;execute_kernel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;agent_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;workspace&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kernel_path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"agent_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;agent_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"command"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"./run_kernel {kernel_path}"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"serial"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"workspace"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;workspace&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"timeout"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"http://scheduler:8080/submit"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;handle&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"handle"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"http://scheduler:8080/status/{handle}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"state"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"completed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"exit_code"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"stdout"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"state"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"failed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The agent does not know or care whether the command ran locally or migrated. The scheduler handles placement transparently.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Shape
&lt;/h2&gt;

&lt;p&gt;A typical setup:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;4 to 8 Mac mini M4 nodes&lt;/strong&gt; on a local network.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One Harbor instance&lt;/strong&gt; (can run on any node or a separate orchestrator).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One scheduler process per node&lt;/strong&gt; (reports to Harbor, accepts remote execution requests).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Shared NFS or object storage&lt;/strong&gt; for large datasets (optional, not required for workspace snapshots).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each node runs macOS with Metal developer tools installed. The scheduler is a Go binary that uses &lt;code&gt;os/exec&lt;/code&gt; for command execution and &lt;code&gt;net/http&lt;/code&gt; for RPC. Harbor is a Python service with a SQLite database for rollout tracking.&lt;/p&gt;

&lt;p&gt;For cloud deployments, replace Mac minis with EC2 Mac instances. Network latency increases migration cost, so tune the eligibility threshold or disable migration for latency-sensitive workloads.&lt;/p&gt;

&lt;h2&gt;
  
  
  Security Boundaries
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox profiles&lt;/strong&gt; prevent agents from reading &lt;code&gt;/etc/passwd&lt;/code&gt; or writing to system directories.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No sudo or root access&lt;/strong&gt; for agent commands.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Network isolation&lt;/strong&gt;: agents cannot open outbound connections (optional policy).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Workspace encryption at rest&lt;/strong&gt; (optional, via FileVault or APFS encryption).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The scheduler does not authenticate agents. In a multi-tenant environment, add API keys or mTLS between agents and Harbor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use Metal Sandbox when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You train or evaluate models that generate Apple Metal kernels and need high iteration velocity.&lt;/li&gt;
&lt;li&gt;You have multiple Mac nodes and want to amortize GPU idle time across concurrent agent sessions.&lt;/li&gt;
&lt;li&gt;You need isolation to prevent one agent's bad kernel from crashing the host or corrupting shared state.&lt;/li&gt;
&lt;li&gt;You can tolerate 100-200 ms migration overhead for commands that would otherwise wait seconds in a local queue.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid it when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You have a single Mac and no network of nodes (the overhead of workspace snapshotting and RPC outweighs the benefit).&lt;/li&gt;
&lt;li&gt;Your workload is purely inference (no compilation or execution), so GPU contention is not a bottleneck.&lt;/li&gt;
&lt;li&gt;You need sub-millisecond command latency (migration and scheduling add observable overhead).&lt;/li&gt;
&lt;li&gt;Your kernels require exclusive GPU access for minutes at a time (migration cannot help if no node ever becomes idle).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The infrastructure is open source and integrates cleanly with existing agent frameworks. If you are building GPU code generation agents on Apple Silicon, Metal Sandbox solves the resource contention problem without requiring a rewrite of your agent loop.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://arxiv.org/abs/2611.17294v1" rel="noopener noreferrer"&gt;ArXiv Paper (2611.17294v1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://arxiv.org/pdf/2611.17294v1.pdf" rel="noopener noreferrer"&gt;PDF&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>computerscience</category>
      <category>performance</category>
    </item>
    <item>
      <title>Shotoku: Local-First Budget Caps and Action Governance for AI Agents</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Sun, 15 Nov 2026 00:07:20 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/shotoku-local-first-budget-caps-and-action-governance-for-ai-agents-5594</link>
      <guid>https://dev.to/mech_app_ai/shotoku-local-first-budget-caps-and-action-governance-for-ai-agents-5594</guid>
      <description>&lt;p&gt;Shotoku is a local-first governance layer that enforces budget caps and approval workflows before agents execute paid actions. It intercepts tool calls, checks policy rules stored in YAML, and writes decisions to a local JSONL audit log. No cloud sync, no custody handoff, no external approval service.&lt;/p&gt;

&lt;p&gt;The core primitive is a synchronous authorization check that runs before any action that costs money, calls a paid API, or mutates external state. If the action exceeds a per-transaction or daily budget, Shotoku denies it or queues it for human approval. The decision is recorded locally, and the agent only proceeds if the status is &lt;code&gt;approved&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;

&lt;p&gt;Shotoku sits between your agent orchestration code and the tool execution layer. The integration point is a single function call:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;authorize&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"@shotoku/core"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;decision&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;authorize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"shopping-agent"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"purchase"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"api.openai.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;24.99&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;decision&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"approved"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// execute the tool call&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;decision&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"pending_approval"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// queue for human review&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// denied, log and skip&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;authorize&lt;/code&gt; function reads a local policy file (&lt;code&gt;policy.yaml&lt;/code&gt;) and appends a decision record to &lt;code&gt;decisions.jsonl&lt;/code&gt;. The policy file defines per-resource rules:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;openai.com&lt;/span&gt;
    &lt;span class="na"&gt;actions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;purchase&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;api_call&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;maxAmount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;
    &lt;span class="na"&gt;maxDailyAmount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;
    &lt;span class="na"&gt;verdict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;approved&lt;/span&gt;

  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;stripe.com&lt;/span&gt;
    &lt;span class="na"&gt;actions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;purchase&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;verdict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;pending_approval&lt;/span&gt;

  &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"*"&lt;/span&gt;
    &lt;span class="na"&gt;verdict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;denied&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The decision log is append-only JSONL. Each line is a JSON object with an ID, status, amount, and timestamp. Approval events reference the original decision ID:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{"id":"dec_01","status":"approved","amount":"12.00","timestamp":"2026-11-14T14:32:01Z"}
{"id":"dec_02","status":"pending_approval","amount":"89.99","timestamp":"2026-11-14T14:33:12Z"}
{"id":"apr_01","type":"approval","decision":"dec_02","status":"approved","timestamp":"2026-11-14T14:45:00Z"}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This structure supports offline reconciliation and tamper detection. If a decision record is missing or out of sequence, you know something was deleted or reordered.&lt;/p&gt;

&lt;h2&gt;
  
  
  State Management and Distributed Execution
&lt;/h2&gt;

&lt;p&gt;The local-first design assumes a single process or a shared filesystem. If you run multiple agent processes on different machines, each maintains its own decision log. There is no built-in reconciliation across nodes.&lt;/p&gt;

&lt;p&gt;For distributed deployments, you have three options:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Shared filesystem&lt;/strong&gt;: Mount a network volume so all processes write to the same &lt;code&gt;decisions.jsonl&lt;/code&gt;. File locking prevents concurrent writes from corrupting the log.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Centralized aggregator&lt;/strong&gt;: Each agent writes to a local log, and a separate process tails all logs and merges them into a central audit store. This adds latency but preserves the local-first guarantee during network partitions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Single orchestrator&lt;/strong&gt;: Run one agent process that dispatches work to stateless workers. Only the orchestrator calls &lt;code&gt;authorize&lt;/code&gt;, so budget tracking stays consistent.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The policy file can be version-controlled and deployed to all nodes. Changes take effect on the next &lt;code&gt;authorize&lt;/code&gt; call, so there is no cache invalidation problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Approval Workflow Implementation
&lt;/h2&gt;

&lt;p&gt;When a decision status is &lt;code&gt;pending_approval&lt;/code&gt;, the agent must block or queue the action. Shotoku does not provide a built-in UI or notification system. You implement the approval loop yourself:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Blocking queue&lt;/strong&gt;: The agent polls &lt;code&gt;decisions.jsonl&lt;/code&gt; for approval events. When a human approves the decision (by appending an approval record), the agent retries the tool call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Callback registration&lt;/strong&gt;: The agent registers a webhook or callback function that fires when an approval event is written. This requires a file watcher or event stream.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Human-in-the-loop UI&lt;/strong&gt;: A separate dashboard reads &lt;code&gt;decisions.jsonl&lt;/code&gt;, displays pending decisions, and writes approval records when a human clicks "Approve."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The approval record is just another JSONL line. The agent code checks for it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;waitForApproval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;decisionId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;watcher&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;watch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"decisions.jsonl"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;readFileSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"decisions.jsonl"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"utf8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"\n"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Boolean&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;approval&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;decision&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="n"&gt;decisionId&lt;/span&gt;
      &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;approval&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;watcher&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="nf"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;approval&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is naive but functional. For production, use a proper event stream or message queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  Budget Reconciliation and Rolling Windows
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;maxDailyAmount&lt;/code&gt; rule enforces a rolling 24-hour budget. Shotoku scans the decision log for all approved transactions in the last 24 hours and sums the amounts. If the new request would exceed the limit, it is denied or queued.&lt;/p&gt;

&lt;p&gt;This scan happens on every &lt;code&gt;authorize&lt;/code&gt; call. For high-throughput agents, this becomes a performance bottleneck. You can optimize by caching the rolling sum and invalidating it every minute, or by indexing the decision log with a time-series database.&lt;/p&gt;

&lt;p&gt;The policy file does not support budget rollover or monthly caps. If you need that, you write custom logic that reads the decision log and adjusts the &lt;code&gt;maxDailyAmount&lt;/code&gt; dynamically.&lt;/p&gt;

&lt;h2&gt;
  
  
  Security Boundaries
&lt;/h2&gt;

&lt;p&gt;The policy file and decision log are plaintext. If an attacker gains filesystem access, they can delete decisions, modify the policy, or forge approval records. Shotoku does not sign or encrypt the log.&lt;/p&gt;

&lt;p&gt;To harden the setup:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Immutable log&lt;/strong&gt;: Write &lt;code&gt;decisions.jsonl&lt;/code&gt; to an append-only file or object storage bucket with versioning enabled.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Separate approval process&lt;/strong&gt;: Run the approval UI on a different machine with read-only access to the decision log. Approvals are written to a separate file that the agent polls.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cryptographic signatures&lt;/strong&gt;: Hash each decision record and include the hash of the previous record, creating a Merkle chain. This detects tampering but does not prevent it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The local-first design means there is no authentication layer. Any process that can read the policy file can call &lt;code&gt;authorize&lt;/code&gt;. If you run untrusted code alongside your agent, it can bypass Shotoku by calling the tool directly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability and Failure Modes
&lt;/h2&gt;

&lt;p&gt;The decision log is your primary observability surface. You can tail it in real time, export it to a monitoring tool, or parse it with &lt;code&gt;jq&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cat decisions.jsonl &lt;span class="o"&gt;|&lt;/span&gt; jq &lt;span class="s1"&gt;'select(.status == "denied")'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Common failure modes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Mitigation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Policy file missing&lt;/td&gt;
&lt;td&gt;All requests denied&lt;/td&gt;
&lt;td&gt;Validate policy file exists on startup&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Decision log corrupted&lt;/td&gt;
&lt;td&gt;JSON parse errors&lt;/td&gt;
&lt;td&gt;Write each line atomically, use file locking&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Approval never arrives&lt;/td&gt;
&lt;td&gt;Agent hangs indefinitely&lt;/td&gt;
&lt;td&gt;Add timeout to approval polling loop&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Budget calculation drift&lt;/td&gt;
&lt;td&gt;Incorrect daily totals&lt;/td&gt;
&lt;td&gt;Periodically reconcile log against external ledger&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concurrent writes&lt;/td&gt;
&lt;td&gt;Interleaved JSON lines&lt;/td&gt;
&lt;td&gt;Use advisory file locks or single-writer pattern&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Shotoku does not retry failed &lt;code&gt;authorize&lt;/code&gt; calls. If the policy file is unreadable or the log write fails, the function throws an error. Your agent code must handle this and decide whether to fail open (allow the action) or fail closed (deny it).&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Shape
&lt;/h2&gt;

&lt;p&gt;Shotoku is a library, not a service. You install it via npm and import it into your agent code. The CLI (&lt;code&gt;shotoku-cli&lt;/code&gt;) provides commands to inspect the decision log and validate the policy file, but it does not run a daemon or server.&lt;/p&gt;

&lt;p&gt;For a single-machine deployment:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Install &lt;code&gt;@shotoku/core&lt;/code&gt; in your agent project.&lt;/li&gt;
&lt;li&gt;Write a &lt;code&gt;policy.yaml&lt;/code&gt; file.&lt;/li&gt;
&lt;li&gt;Call &lt;code&gt;authorize&lt;/code&gt; before every paid action.&lt;/li&gt;
&lt;li&gt;Tail &lt;code&gt;decisions.jsonl&lt;/code&gt; for audit and debugging.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For a multi-machine deployment, add a log aggregator or shared filesystem. The policy file can be deployed via configuration management (Ansible, Chef) or read from a Git repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code Example: Intercepting Tool Calls
&lt;/h2&gt;

&lt;p&gt;If you use LangChain or a similar framework, wrap the tool execution function:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;authorize&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"@shotoku/core"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;executeToolWithBudget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cost&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;decision&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;authorize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"my-agent"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;resource&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;endpoint&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cost&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;decision&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="s2"&gt;"approved"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`Tool call blocked: ${decision.reasons.join(", ")}`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This pattern works for any tool that has a known cost. For tools with variable costs (like LLM API calls), you estimate the cost based on token count or use a conservative upper bound.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use Shotoku when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You run agents on your own infrastructure and need budget enforcement without cloud dependencies.&lt;/li&gt;
&lt;li&gt;You want a tamper-evident audit log for compliance or debugging.&lt;/li&gt;
&lt;li&gt;You need human-in-the-loop approval for high-value actions.&lt;/li&gt;
&lt;li&gt;You can tolerate the operational overhead of managing local state and approval workflows.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid Shotoku when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You run agents in a distributed, multi-tenant environment where local state is impractical.&lt;/li&gt;
&lt;li&gt;You need real-time budget tracking across hundreds of agents (the log scan becomes a bottleneck).&lt;/li&gt;
&lt;li&gt;You require cryptographic guarantees or non-repudiation (the log is not signed).&lt;/li&gt;
&lt;li&gt;You want a turnkey approval UI or notification system (you must build it yourself).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Shotoku is a control primitive, not a complete governance platform. It gives you the building blocks to enforce budgets and approvals, but you are responsible for the approval loop, log aggregation, and security hardening. For teams that already run local-first infrastructure and want fine-grained cost control, it is a clean, minimal solution. For teams that need cloud-native governance or multi-region consistency, it is too narrow.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.shotoku.dev/" rel="noopener noreferrer"&gt;Shotoku Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://news.ycombinator.com/item?id=50496705" rel="noopener noreferrer"&gt;Hacker News Discussion&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>architecture</category>
      <category>security</category>
    </item>
    <item>
      <title>Plannotator's Inbox Pattern: Why Background Agent Execution Needs an Email-Like UI</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Sat, 14 Nov 2026 20:06:41 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/plannotators-inbox-pattern-why-background-agent-execution-needs-an-email-like-ui-n0b</link>
      <guid>https://dev.to/mech_app_ai/plannotators-inbox-pattern-why-background-agent-execution-needs-an-email-like-ui-n0b</guid>
      <description>&lt;p&gt;Chat interfaces assume you are present when the agent runs. That assumption breaks when you have three coding agents refactoring a monorepo overnight, a personal assistant drafting emails, and a research agent pulling competitive analysis. You need a review surface that batches decisions, surfaces artifacts, and lets you annotate without blocking execution.&lt;/p&gt;

&lt;p&gt;Plannotator (10,977 stars, trending #1 on GitHub TypeScript) is a local, browser-based inbox for multi-agent work. It treats agent outputs like email: each task lands in a unified surface where you review diffs, annotate plans, approve prototypes, and send feedback back to the agent. The contract is simple: agents write artifacts to a local directory, Plannotator renders them in a scannable UI, and your annotations flow back as structured input for the next run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the Inbox Model Fits Asynchronous Agent Work
&lt;/h2&gt;

&lt;p&gt;Chat UIs force synchronous interaction. You ask, the agent responds, you reply. That loop works for single-turn tasks but collapses when agents run in the background or when you manage multiple agents with overlapping outputs.&lt;/p&gt;

&lt;p&gt;An inbox model decouples execution from review:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Agents write artifacts to disk.&lt;/strong&gt; Plans, diffs, HTML prototypes, markdown specs. Each artifact is a file with metadata (agent ID, timestamp, task context).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The inbox surfaces unreviewed items.&lt;/strong&gt; You see a list of pending decisions, not a chronological chat log.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Annotations are first-class.&lt;/strong&gt; You highlight a section of a plan, add a comment, mark a diff as approved or rejected. These annotations persist as structured data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Feedback loops are explicit.&lt;/strong&gt; When you click "Send Feedback," Plannotator packages your annotations and writes them to a location the agent reads on the next run.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This pattern mirrors how engineering teams handle code review. You do not sit next to the developer while they write code. You review the PR when it is ready, leave comments, and they iterate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture: File-Based State and Browser Rendering
&lt;/h2&gt;

&lt;p&gt;Plannotator runs entirely in the browser. No server, no database. State lives in a local directory that both the agent and the UI can access.&lt;/p&gt;

&lt;h3&gt;
  
  
  Directory Structure
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;~/.plannotator/
├── inbox/
│   ├── task-001-refactor-auth/
│   │   ├── plan.md
│   │   ├── diff.patch
│   │   ├── metadata.json
│   │   └── annotations.json
│   ├── task-002-landing-page/
│   │   ├── prototype.html
│   │   ├── metadata.json
│   │   └── annotations.json
├── archive/
│   └── task-000-initial-setup/
└── config.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each task is a directory. The agent writes artifacts (plan.md, diff.patch, prototype.html) and a metadata.json file that includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Agent ID (Claude Code, Copilot CLI, custom harness)&lt;/li&gt;
&lt;li&gt;Task description&lt;/li&gt;
&lt;li&gt;Timestamp&lt;/li&gt;
&lt;li&gt;Parent task ID (for multi-step workflows)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Plannotator watches this directory. When a new task appears, it renders the artifacts in the inbox. When you annotate, it writes annotations.json to the same directory.&lt;/p&gt;

&lt;h3&gt;
  
  
  Annotation Contract
&lt;/h3&gt;

&lt;p&gt;Annotations are JSON objects with a schema the agent can parse:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"annotations"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"artifact"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"plan.md"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"line_start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"line_end"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"comment"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Use bcrypt instead of SHA-256 for password hashing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"change_requested"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nl"&gt;"artifact"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"diff.patch"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"file"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"src/auth/login.ts"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"line_start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"line_end"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"comment"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Approved"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"approved"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"overall_status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"needs_revision"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"feedback_sent_at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-14T20:15:00Z"&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The agent reads this file, applies changes, and writes a new task directory with updated artifacts. The cycle repeats.&lt;/p&gt;

&lt;h2&gt;
  
  
  Multi-Agent Coordination
&lt;/h2&gt;

&lt;p&gt;Plannotator supports heterogeneous agents. You might have:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Claude Code&lt;/strong&gt; generating a FastAPI backend&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Copilot CLI&lt;/strong&gt; writing frontend components&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A custom research agent&lt;/strong&gt; pulling competitor pricing data&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each agent writes to the same inbox directory but uses a different agent_id in metadata.json. The inbox groups tasks by agent, so you can review all Claude Code outputs in one session and all Copilot CLI outputs in another.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cross-Agent Dependencies
&lt;/h3&gt;

&lt;p&gt;If Task B depends on Task A, the agent writes a parent_task_id field in metadata.json. Plannotator renders a dependency graph in the UI. You can see which tasks are blocked and which are ready for review.&lt;/p&gt;

&lt;p&gt;This is not orchestration. Plannotator does not schedule agents or enforce execution order. It surfaces dependencies so you can decide whether to approve Task A before Task B runs again.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code Review Surface
&lt;/h2&gt;

&lt;p&gt;Plannotator renders diffs with syntax highlighting and inline comments. You can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Approve individual files&lt;/li&gt;
&lt;li&gt;Request changes on specific lines&lt;/li&gt;
&lt;li&gt;Mark entire diffs as "needs revision" or "approved"&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The diff viewer is a React component that parses .patch files and maps line numbers to the original source. When you add a comment, it stores the file path, line range, and comment text in annotations.json.&lt;/p&gt;

&lt;h3&gt;
  
  
  Integration with Git
&lt;/h3&gt;

&lt;p&gt;Plannotator does not replace Git. It sits between the agent and your version control workflow. A typical flow:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Agent writes a diff.patch to the inbox&lt;/li&gt;
&lt;li&gt;You review in Plannotator, add comments&lt;/li&gt;
&lt;li&gt;Agent applies the patch to a feature branch&lt;/li&gt;
&lt;li&gt;You review the PR in GitHub/GitLab as usual&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The inbox is for high-level approval ("yes, refactor the auth module") before the agent commits code. The PR is for line-by-line review before merging.&lt;/p&gt;

&lt;h2&gt;
  
  
  Artifact Types and Rendering
&lt;/h2&gt;

&lt;p&gt;Plannotator handles multiple artifact types:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Artifact Type&lt;/th&gt;
&lt;th&gt;Rendering&lt;/th&gt;
&lt;th&gt;Annotation Support&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Markdown plans&lt;/td&gt;
&lt;td&gt;Rendered HTML with section anchors&lt;/td&gt;
&lt;td&gt;Inline comments on paragraphs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Diffs (.patch)&lt;/td&gt;
&lt;td&gt;Syntax-highlighted side-by-side view&lt;/td&gt;
&lt;td&gt;Line-level comments&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTML prototypes&lt;/td&gt;
&lt;td&gt;Sandboxed iframe with interaction&lt;/td&gt;
&lt;td&gt;Visual annotations (click to comment)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JSON specs&lt;/td&gt;
&lt;td&gt;Collapsible tree view&lt;/td&gt;
&lt;td&gt;Key-level comments&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Plain text&lt;/td&gt;
&lt;td&gt;Monospace with line numbers&lt;/td&gt;
&lt;td&gt;Line-level comments&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;HTML artifacts are sandboxed in an iframe with a restrictive Content Security Policy. You can interact with the prototype (click buttons, fill forms) but cannot execute arbitrary JavaScript. This prevents malicious agents from injecting code that runs in your browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability and Failure Modes
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What You Can See
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Task history: All tasks in the inbox and archive&lt;/li&gt;
&lt;li&gt;Agent activity: Which agent wrote which artifact, when&lt;/li&gt;
&lt;li&gt;Annotation history: Your comments and status changes over time&lt;/li&gt;
&lt;li&gt;Dependency graph: Which tasks block others&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  What You Cannot See
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Agent internal state: Plannotator does not inspect the agent's memory or reasoning process&lt;/li&gt;
&lt;li&gt;Execution logs: If the agent crashes, you see the last artifact it wrote, not the error trace&lt;/li&gt;
&lt;li&gt;Real-time progress: The inbox updates when the agent writes a new task directory, not during execution&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Failure Modes
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Agent writes malformed metadata.json.&lt;/strong&gt; Plannotator skips the task and logs an error in the browser console. You can manually fix the JSON and refresh.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Agent overwrites an existing task directory.&lt;/strong&gt; Plannotator treats this as a new version. The old artifacts are lost unless you have a backup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Annotations.json is corrupted.&lt;/strong&gt; The agent cannot parse your feedback. You must manually edit the file or delete it and re-annotate.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Multiple agents write to the same task directory.&lt;/strong&gt; Last write wins. Plannotator does not lock directories. Use unique task IDs per agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Shape
&lt;/h2&gt;

&lt;p&gt;Plannotator is a static site. You can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Run it locally with &lt;code&gt;npx serve&lt;/code&gt; or any static file server&lt;/li&gt;
&lt;li&gt;Deploy to Vercel/Netlify and point it at a synced directory (Dropbox, Google Drive)&lt;/li&gt;
&lt;li&gt;Embed it in an Electron app for a desktop experience&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The agent must have write access to the inbox directory. If you deploy Plannotator to a remote server, you need a sync mechanism (rsync, cloud storage API) to move artifacts from the agent's local environment to the server.&lt;/p&gt;

&lt;h3&gt;
  
  
  Security Boundaries
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;No network access.&lt;/strong&gt; Plannotator does not send data to external servers. All state is local.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sandboxed artifacts.&lt;/strong&gt; HTML prototypes run in an iframe with &lt;code&gt;sandbox="allow-scripts allow-same-origin"&lt;/code&gt; and a CSP that blocks external resources.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No authentication.&lt;/strong&gt; Plannotator assumes you control the machine. If you deploy it remotely, add HTTP basic auth or a reverse proxy with authentication.&lt;/p&gt;

&lt;h2&gt;
  
  
  When to Use This Pattern
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use Plannotator when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You run multiple agents asynchronously (overnight builds, background research)&lt;/li&gt;
&lt;li&gt;You need to review and annotate before the agent proceeds&lt;/li&gt;
&lt;li&gt;You want a file-based contract between human and agent (no API, no database)&lt;/li&gt;
&lt;li&gt;You prefer local-first tools with no cloud dependency&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid Plannotator when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You need real-time collaboration (multiple humans reviewing the same task)&lt;/li&gt;
&lt;li&gt;You want the agent to execute based on your feedback without human approval&lt;/li&gt;
&lt;li&gt;You need fine-grained access control (who can approve which tasks)&lt;/li&gt;
&lt;li&gt;You require audit logs for compliance (Plannotator does not track who made which annotation)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;Plannotator solves a real coordination problem: how do you stay in the loop when agents work in the background? The inbox model is a good fit for asynchronous, multi-agent workflows where human approval is a gate, not a bottleneck.&lt;/p&gt;

&lt;p&gt;The file-based contract is simple and inspectable. You can debug by looking at the directory structure. You can version control the entire inbox with Git. You can write custom agents that follow the same schema without touching Plannotator's code.&lt;/p&gt;

&lt;p&gt;The tradeoffs are clear. No real-time collaboration. No built-in orchestration. No server-side logic. If you need those, you are building a different system.&lt;/p&gt;

&lt;p&gt;For solo developers or small teams running local agents, this is a clean primitive. It does one thing (surface agent outputs for review) and does it without adding a database, API, or deployment complexity.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/backnotprop/plannotator" rel="noopener noreferrer"&gt;Plannotator GitHub Repository&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://plannotator.ai/" rel="noopener noreferrer"&gt;Official Site&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>opensource</category>
      <category>tooling</category>
    </item>
    <item>
      <title>CapLite: Why Small Language Model Agents Can't Write Their Own Security Policies</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Sat, 14 Nov 2026 10:06:01 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/caplite-why-small-language-model-agents-cant-write-their-own-security-policies-h34</link>
      <guid>https://dev.to/mech_app_ai/caplite-why-small-language-model-agents-cant-write-their-own-security-policies-h34</guid>
      <description>&lt;p&gt;Enforcement-based prompt injection defenses ask the agent's own language model to write a security artifact: a privilege policy, a dependency graph, a constraint list, or an executable program. Code then enforces that artifact. CaMeL, Progent, IPIGuard, and DRIFT all follow this pattern. The assumption is that the model understands the task well enough to generate a correct security boundary.&lt;/p&gt;

&lt;p&gt;CapLite (arXiv:2611.17115v1) tests that assumption across 15 open-weight models, 13 of them between 1.7B and 12B parameters. The result: small language models cannot reliably write their own security policies. Attacks succeed against 14 models using Progent, 13 using IPIGuard, and 11 using DRIFT. CaMeL's generated programs fail to run in up to 68% of episodes.&lt;/p&gt;

&lt;p&gt;The paper introduces a capability-based enforcement approach that does not ask the model to generate security artifacts at all. Instead, CapLite derives its policy from tool schemas, commits write calls before any tool result is read, and admits a write only if every argument that decides whom or what it affects appears in the user's request. This guarantees write integrity: injected text cannot add a write, and no write reaches a recipient, amount, object, or destination the user did not explicitly state.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Model-Written Artifact Problem
&lt;/h2&gt;

&lt;p&gt;Enforcement defenses treat the agent's model as both the task executor and the security policy author. The model generates a structured artifact that describes what the agent is allowed to do, then a separate enforcement layer checks every tool call against that artifact.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Failure modes when the model is too small:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Incomplete policies&lt;/strong&gt;: The model omits critical constraints, leaving gaps that attacks exploit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Contradictory constraints&lt;/strong&gt;: The model writes rules that conflict, and the enforcement layer picks the permissive interpretation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Syntax errors&lt;/strong&gt;: The generated artifact does not parse or execute, disabling enforcement entirely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Semantic drift&lt;/strong&gt;: The model's understanding of the task shifts between artifact generation and tool invocation, creating a mismatch.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The paper tested this across four defenses:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Defense&lt;/th&gt;
&lt;th&gt;Artifact Type&lt;/th&gt;
&lt;th&gt;Attack Success Rate (13 SLMs)&lt;/th&gt;
&lt;th&gt;Failure Mode&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Progent&lt;/td&gt;
&lt;td&gt;Privilege policy&lt;/td&gt;
&lt;td&gt;14/15 models&lt;/td&gt;
&lt;td&gt;Incomplete privilege declarations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IPIGuard&lt;/td&gt;
&lt;td&gt;Dependency graph&lt;/td&gt;
&lt;td&gt;13/15 models&lt;/td&gt;
&lt;td&gt;Missing or incorrect edges&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DRIFT&lt;/td&gt;
&lt;td&gt;Constraint list&lt;/td&gt;
&lt;td&gt;11/15 models&lt;/td&gt;
&lt;td&gt;Contradictory or overly broad rules&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CaMeL&lt;/td&gt;
&lt;td&gt;Executable program&lt;/td&gt;
&lt;td&gt;Up to 68% episode failures&lt;/td&gt;
&lt;td&gt;Syntax errors, non-executable output&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The pattern is consistent: models below a certain capability threshold cannot generate security artifacts that enforcement code can trust.&lt;/p&gt;

&lt;h2&gt;
  
  
  CapLite's Capability-Based Architecture
&lt;/h2&gt;

&lt;p&gt;CapLite removes the model from the security artifact generation loop. The enforcement policy is derived from tool schemas at initialization, not generated per-task.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Core enforcement rules:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Schema-derived policy&lt;/strong&gt;: Each tool's schema declares which arguments affect external state (recipients, amounts, destinations). The policy is built once from these declarations.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Write-before-read&lt;/strong&gt;: All write calls are committed before any tool result is read. This prevents an attacker from injecting a write after observing intermediate state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;User-stated arguments&lt;/strong&gt;: A write is admitted only if every argument that decides whom or what it affects is explicitly stated in the user's request. The enforcement layer parses the user's input and extracts entities, then checks each write call's arguments against that set.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Orchestration flow:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User request → Parse entities → Agent generates tool calls → Enforcement layer checks writes → Commit writes → Execute reads → Return results
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The enforcement layer sits between the agent and the tool execution runtime. It does not ask the model to justify or explain its tool calls. It checks a simple invariant: did the user mention this recipient, amount, or destination?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example enforcement check:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;admit_write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool_call&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;user_entities&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""
    Admit a write only if all decision arguments appear in user_entities.
    """&lt;/span&gt;
    &lt;span class="n"&gt;decision_args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tool_call&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;decision_arguments&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;arg&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;decision_args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;tool_call&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;arg&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;user_entities&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;user_entities&lt;/code&gt; set is extracted once from the user's request using a simple parser (regex or a small NER model). The &lt;code&gt;decision_arguments&lt;/code&gt; list is declared in the tool schema. No model-generated artifact is involved.&lt;/p&gt;

&lt;h2&gt;
  
  
  Evaluation Results
&lt;/h2&gt;

&lt;p&gt;CapLite was tested on AgentDojo, a benchmark with 13,935 attacked episodes. No attack succeeded. The defense guarantees write integrity: injected text cannot add a write, and no write reaches a recipient, amount, object, or destination the user did not state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Utility and token efficiency:&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Defense&lt;/th&gt;
&lt;th&gt;Utility Under Attack&lt;/th&gt;
&lt;th&gt;Tokens per Episode&lt;/th&gt;
&lt;th&gt;Utility per Token&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;CapLite&lt;/td&gt;
&lt;td&gt;29.0%&lt;/td&gt;
&lt;td&gt;17.1k&lt;/td&gt;
&lt;td&gt;1.70e-3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CaMeL&lt;/td&gt;
&lt;td&gt;27.2%&lt;/td&gt;
&lt;td&gt;95.0k&lt;/td&gt;
&lt;td&gt;2.86e-4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Progent&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;IPIGuard&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DRIFT&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Undefended&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;td&gt;20.0k&lt;/td&gt;
&lt;td&gt;(not reported)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;CapLite uses 18% of CaMeL's tokens and achieves higher utility under attack. It also uses fewer tokens than the undefended agent, which suggests the enforcement overhead is minimal.&lt;/p&gt;

&lt;h2&gt;
  
  
  When Model-Written Artifacts Work
&lt;/h2&gt;

&lt;p&gt;The paper does not identify a hard parameter threshold, but the pattern is clear: models above 12B parameters show better artifact quality, though still not perfect. The failure rate drops as model size increases, but even larger models occasionally generate incomplete or contradictory policies.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Factors that improve artifact reliability:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Instruction-tuned models&lt;/strong&gt;: Models fine-tuned for structured output (e.g., function calling, JSON generation) perform better.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Explicit schema enforcement&lt;/strong&gt;: If the artifact format is strictly validated before enforcement begins, syntax errors are caught early.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Redundant checks&lt;/strong&gt;: Generating multiple artifacts and voting or intersecting them reduces the chance of a single bad artifact.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these techniques eliminate the problem for SLMs. The cost of generating multiple artifacts or using a larger model for artifact generation alone often exceeds the cost of CapLite's schema-derived approach.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Constraints
&lt;/h2&gt;

&lt;p&gt;CapLite's approach works when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tool schemas are well-defined and include decision argument annotations.&lt;/li&gt;
&lt;li&gt;The user's request contains explicit entities (names, amounts, destinations).&lt;/li&gt;
&lt;li&gt;The task does not require the agent to infer implicit recipients or destinations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Failure cases:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Implicit references&lt;/strong&gt;: If the user says "send it to my manager" without naming the manager, CapLite cannot extract the entity. The agent must resolve the reference before the write is admitted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dynamic schemas&lt;/strong&gt;: If tools are added or modified at runtime, the schema-derived policy must be regenerated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Complex state dependencies&lt;/strong&gt;: If a write's validity depends on prior reads (e.g., "transfer the balance from account A to account B"), CapLite's write-before-read rule may block legitimate operations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The paper does not address these cases in detail. The trade-off is clear: CapLite sacrifices flexibility for reliability. It works best in domains where the user's intent is explicit and the tool set is stable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Security Boundary Comparison
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;Policy Source&lt;/th&gt;
&lt;th&gt;Enforcement Point&lt;/th&gt;
&lt;th&gt;SLM Compatibility&lt;/th&gt;
&lt;th&gt;Token Overhead&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model-written artifacts&lt;/td&gt;
&lt;td&gt;Agent's own model&lt;/td&gt;
&lt;td&gt;After artifact generation&lt;/td&gt;
&lt;td&gt;Poor (14/15 fail)&lt;/td&gt;
&lt;td&gt;High (95k)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schema-derived (CapLite)&lt;/td&gt;
&lt;td&gt;Tool schemas&lt;/td&gt;
&lt;td&gt;Before tool execution&lt;/td&gt;
&lt;td&gt;Good (0 attacks)&lt;/td&gt;
&lt;td&gt;Low (17k)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Static allowlists&lt;/td&gt;
&lt;td&gt;Developer-defined&lt;/td&gt;
&lt;td&gt;Before tool execution&lt;/td&gt;
&lt;td&gt;Excellent&lt;/td&gt;
&lt;td&gt;Minimal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runtime sandboxing&lt;/td&gt;
&lt;td&gt;OS or container&lt;/td&gt;
&lt;td&gt;During tool execution&lt;/td&gt;
&lt;td&gt;Excellent&lt;/td&gt;
&lt;td&gt;Variable&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Static allowlists and runtime sandboxing do not require the model to generate security artifacts, but they also do not adapt to the task. CapLite sits between these extremes: it adapts to the user's request without asking the model to write a policy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use CapLite when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You are deploying agents on small language models (under 12B parameters).&lt;/li&gt;
&lt;li&gt;Your tool schemas are stable and well-annotated with decision arguments.&lt;/li&gt;
&lt;li&gt;The user's requests contain explicit entities (names, amounts, destinations).&lt;/li&gt;
&lt;li&gt;You need prompt injection defense without the token overhead of model-written artifacts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid CapLite when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The user's requests rely on implicit references or context that the agent must resolve.&lt;/li&gt;
&lt;li&gt;Your tools are added or modified frequently, requiring policy regeneration.&lt;/li&gt;
&lt;li&gt;The task requires writes whose validity depends on prior reads (e.g., conditional transfers).&lt;/li&gt;
&lt;li&gt;You have access to a large model (70B+) that can reliably generate security artifacts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The core insight is that enforcement defenses scale poorly to small models. If your deployment constraints push you toward SLMs, capability-based enforcement is a better fit than asking the model to write its own security policy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://arxiv.org/abs/2611.17115v1" rel="noopener noreferrer"&gt;CapLite: Prompt-Injection Defense for Small Language Model Agents without Model-Written Security Artifacts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://arxiv.org/pdf/2611.17115v1.pdf" rel="noopener noreferrer"&gt;PDF&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>llm</category>
      <category>security</category>
    </item>
    <item>
      <title>Pomerado's Self-Healing APIs: How Deterministic Web Scraping Turns Legacy Banking Sites Into Agent-Ready MCP Servers</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Sat, 14 Nov 2026 00:08:37 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/pomerados-self-healing-apis-how-deterministic-web-scraping-turns-legacy-banking-sites-into-6ja</link>
      <guid>https://dev.to/mech_app_ai/pomerados-self-healing-apis-how-deterministic-web-scraping-turns-legacy-banking-sites-into-6ja</guid>
      <description>&lt;p&gt;Agents need to interact with banks, airlines, and legacy portals that will never ship official APIs. Pomerado (YC F26) wraps these sites as MCP servers by recording a single manual workflow, then generating deterministic HTTP-based scrapers that self-heal when the DOM changes. The pitch is simple: describe a task in natural language, answer a few questions, and get a working MCP in under an hour. Future calls skip the browser and return results in 5-30 seconds.&lt;/p&gt;

&lt;p&gt;This matters because computer use (Anthropic's term for pixel-level browser automation) is too slow and brittle for repeated financial workflows. Most banks also ban headless browsers in their terms of service. Pomerado's approach sidesteps both problems by converting the site into a structured API that agents can call like any other tool.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Determinism Problem
&lt;/h2&gt;

&lt;p&gt;Traditional web scraping breaks when a bank changes a CSS class or reorders form fields. Pomerado claims "deterministic" scrapers, which means the same input always produces the same output, even after site updates. Here's how that works in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Initial recording&lt;/strong&gt;: An agent-controlled browser performs the task once (login, search flights, check balance). Pomerado captures HTTP requests, response payloads, and DOM structure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Selector generation&lt;/strong&gt;: Instead of brittle XPath or CSS selectors, the system generates multiple fallback strategies (element text, ARIA labels, position relative to stable landmarks).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;State fingerprinting&lt;/strong&gt;: Each page state gets a hash of its structure. When a future request hits a page with a different hash, the self-healing logic kicks in.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The key insight is that financial sites change their presentation layer frequently but rarely change their underlying form schemas or API contracts. Pomerado bets that HTTP-level scraping (not Selenium clicks) can remain stable if you track structural invariants instead of surface selectors.&lt;/p&gt;

&lt;h2&gt;
  
  
  Self-Healing in Practice
&lt;/h2&gt;

&lt;p&gt;When a bank's login page changes, Pomerado's MCP server detects the mismatch and triggers a repair workflow:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Diff detection&lt;/strong&gt;: Compare the current DOM hash to the stored fingerprint.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Selector search&lt;/strong&gt;: Try each fallback selector strategy until one matches the expected element type (input field, submit button).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validation&lt;/strong&gt;: Submit a test request and verify the response matches the expected schema (account balance is a number, flight results are a list).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Commit&lt;/strong&gt;: If validation passes, update the stored selectors and fingerprint. If it fails, mark the MCP as degraded and alert the developer.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This only works if the site's underlying behavior is stable. If a bank redesigns its entire authentication flow (OAuth to SAML, for example), no amount of selector healing will help. Pomerado's model assumes incremental UI changes, not architectural rewrites.&lt;/p&gt;

&lt;h2&gt;
  
  
  Session Management and Rate Limiting
&lt;/h2&gt;

&lt;p&gt;Financial agents need to maintain authenticated sessions across multiple tool calls. Pomerado's MCP servers handle this by:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cookie jars&lt;/strong&gt;: Each MCP instance stores session cookies in encrypted storage. When an agent calls &lt;code&gt;check_balance()&lt;/code&gt; followed by &lt;code&gt;transfer_funds()&lt;/code&gt;, both requests reuse the same session.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Token refresh&lt;/strong&gt;: If a session expires mid-workflow, the MCP automatically re-authenticates using stored credentials (encrypted at rest, decrypted only in memory).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rate limiting&lt;/strong&gt;: Pomerado tracks request frequency per site and injects delays to avoid triggering anti-bot defenses. For high-volume workflows, it rotates IP addresses or uses residential proxies.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The authentication boundary is critical. Pomerado stores credentials on behalf of the agent, which means the MCP server becomes a privileged component. If an attacker compromises the MCP, they gain access to the underlying bank account. This is why Pomerado positions itself as infrastructure for internal agents (employee assistants, back-office automation) rather than consumer-facing bots.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes and Error Surfaces
&lt;/h2&gt;

&lt;p&gt;When an agent tries to execute a trade through a scraped brokerage interface, several things can go wrong:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure Type&lt;/th&gt;
&lt;th&gt;Detection Method&lt;/th&gt;
&lt;th&gt;Agent-Visible Error&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;DOM structure changed&lt;/td&gt;
&lt;td&gt;Hash mismatch on page load&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ToolUnavailable: Site structure changed, healing in progress&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Session expired&lt;/td&gt;
&lt;td&gt;401 or redirect to login&lt;/td&gt;
&lt;td&gt;&lt;code&gt;AuthenticationRequired: Re-run with fresh credentials&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rate limit hit&lt;/td&gt;
&lt;td&gt;429 or CAPTCHA challenge&lt;/td&gt;
&lt;td&gt;&lt;code&gt;RateLimitExceeded: Retry after 60s&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Transaction rejected&lt;/td&gt;
&lt;td&gt;Error message in response body&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;TransactionFailed: Insufficient funds&lt;/code&gt; (parsed from site error)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network timeout&lt;/td&gt;
&lt;td&gt;No response after 30s&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Timeout: Site unresponsive, retry or escalate&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The worst failure mode is silent success. If Pomerado's parser misreads a confirmation page and reports a trade as executed when it actually failed, the agent will proceed with incorrect state. To mitigate this, Pomerado validates every response against an expected schema. If the schema match confidence is below 90%, the MCP returns &lt;code&gt;Uncertain&lt;/code&gt; instead of a false positive.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture: From Browser Recording to MCP
&lt;/h2&gt;

&lt;p&gt;Here's the end-to-end flow for creating a new MCP:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Developer provides task description&lt;/strong&gt;: "Log into Chase, navigate to checking account, return current balance."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pomerado's agent performs task once&lt;/strong&gt;: Opens a headless browser, follows the steps, records all HTTP traffic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code generation&lt;/strong&gt;: Converts the recorded session into a Python function that replays the HTTP requests without a browser.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Schema extraction&lt;/strong&gt;: Infers the output schema (balance is a float, account number is a string) from the response.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MCP packaging&lt;/strong&gt;: Wraps the function in an MCP server with tool metadata (name, description, parameters, return type).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deployment&lt;/strong&gt;: Hosts the MCP on Pomerado's infrastructure or exports it as a Docker container for self-hosting.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The generated code looks like this (simplified):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;httpx&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;mcp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;MCPServer&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ChaseBalanceTool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;session_store&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;session_store&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;httpx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cookies&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cookies&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get_balance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;account_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# Step 1: Navigate to accounts page&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="s2"&gt;"https://secure.chase.com/web/auth/dashboard"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"User-Agent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Mozilla/5.0..."&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;401&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AuthenticationRequired&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="c1"&gt;# Step 2: Parse balance from response&lt;/span&gt;
        &lt;span class="n"&gt;balance_elem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_find_element&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"selector"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"span.balance-amount"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"text_contains"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Available Balance"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"aria_label"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"account balance"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;balance_elem&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ToolUnavailable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Balance element not found, site may have changed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="s2"&gt;"account_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;account_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"balance"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;float&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;balance_elem&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"$"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;","&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"currency"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"USD"&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_find_element&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strategies&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;list&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Optional&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;Element&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;strategy&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;strategies&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;elem&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_try_selector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;strategy&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;elem&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_validate_element&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;elem&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;elem&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;

&lt;span class="c1"&gt;# MCP server registration&lt;/span&gt;
&lt;span class="n"&gt;server&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;MCPServer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;server&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;register_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"chase_get_balance"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;ChaseBalanceTool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get_balance&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;_find_element&lt;/code&gt; method is where self-healing happens. Each strategy is tried in order until one succeeds. If all fail, the tool returns an error and logs the failure for manual review.&lt;/p&gt;

&lt;h2&gt;
  
  
  CI/CD for Scraped APIs
&lt;/h2&gt;

&lt;p&gt;Testing an MCP built from a live bank site is tricky. You can't mock the responses because the whole point is to interact with the real site. Pomerado's testing strategy:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Smoke tests&lt;/strong&gt;: Run each MCP tool once per day against the live site. If it returns a valid response, mark it as healthy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Schema validation&lt;/strong&gt;: Compare the response schema to the stored schema. If fields are missing or types changed, trigger an alert.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Canary deployments&lt;/strong&gt;: When self-healing updates a selector, deploy the new version to 10% of requests first. If the error rate increases, roll back.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Manual review queue&lt;/strong&gt;: When a tool fails smoke tests three times in a row, flag it for human review.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The CI/CD pipeline runs in a sandboxed environment with access to test credentials for each site. For production deployments, Pomerado recommends running MCP servers in a separate VPC with strict egress rules (only allow connections to the target site, no lateral movement).&lt;/p&gt;

&lt;h2&gt;
  
  
  Security Boundaries
&lt;/h2&gt;

&lt;p&gt;Pomerado's architecture introduces several trust boundaries:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Agent to MCP&lt;/strong&gt;: The agent calls the MCP server with parameters (account ID, transfer amount). The MCP validates inputs but trusts the agent to make correct decisions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MCP to site&lt;/strong&gt;: The MCP server authenticates to the bank using stored credentials. The bank trusts the MCP because it looks like a normal user session.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Developer to Pomerado&lt;/strong&gt;: The developer provides credentials to Pomerado's platform. Pomerado encrypts them at rest and only decrypts them when the MCP runs.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The weakest link is credential storage. If Pomerado's key management is compromised, an attacker gains access to all stored credentials. Pomerado mitigates this by:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Using hardware security modules (HSMs) for key storage.&lt;/li&gt;
&lt;li&gt;Rotating encryption keys every 90 days.&lt;/li&gt;
&lt;li&gt;Requiring multi-factor authentication for credential uploads.&lt;/li&gt;
&lt;li&gt;Logging every credential access with tamper-proof audit trails.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For high-security deployments, Pomerado supports self-hosted MCP servers where credentials never leave the customer's infrastructure.&lt;/p&gt;

&lt;h2&gt;
  
  
  When to Use This
&lt;/h2&gt;

&lt;p&gt;Pomerado makes sense when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You need agents to interact with sites that will never ship APIs (government portals, regional banks, legacy insurance systems).&lt;/li&gt;
&lt;li&gt;The site's underlying behavior is stable, even if the UI changes frequently.&lt;/li&gt;
&lt;li&gt;You control the agent's execution environment and can enforce strict security policies.&lt;/li&gt;
&lt;li&gt;The cost of manual data entry or screen scraping exceeds the cost of maintaining scraped APIs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Avoid this approach when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The site has an official API (even if it's poorly documented). Scraping should be a last resort.&lt;/li&gt;
&lt;li&gt;The site uses aggressive anti-bot defenses (CAPTCHAs on every page, device fingerprinting, behavioral analysis).&lt;/li&gt;
&lt;li&gt;The workflow involves high-stakes transactions where silent failures could cause financial loss.&lt;/li&gt;
&lt;li&gt;You need real-time data (scraping adds 5-30 seconds of latency per request).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;Pomerado solves a real problem: agents need structured access to legacy systems, and computer use is too slow for repeated workflows. The deterministic scraping approach is clever because it sidesteps the brittleness of traditional selectors while avoiding the latency of browser automation.&lt;/p&gt;

&lt;p&gt;The self-healing mechanism is the key innovation. If it works as advertised (automatic recovery from 80% of site changes), this could be the difference between a tool that breaks every week and one that runs for months without intervention.&lt;/p&gt;

&lt;p&gt;The biggest risk is credential management. Storing bank passwords in a third-party service is a hard sell for enterprises. Self-hosted deployments mitigate this, but they also eliminate Pomerado's ability to run centralized smoke tests and share healing updates across customers.&lt;/p&gt;

&lt;p&gt;For internal agents (employee assistants, back-office automation), this is a pragmatic solution to the legacy integration problem. For consumer-facing agents, the security and compliance risks are too high. Use this when you own both the agent and the credentials, and when the alternative is paying humans to copy data between systems.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://news.ycombinator.com/item?id=50455811" rel="noopener noreferrer"&gt;Launch HN: Pomerado (YC F26)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pomerado.ai/" rel="noopener noreferrer"&gt;Pomerado Website&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://youtu.be/Bi4jci7rgHo" rel="noopener noreferrer"&gt;Product Walkthrough Video&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>automation</category>
      <category>mcp</category>
      <category>webscraping</category>
    </item>
    <item>
      <title>Harness Tuner: Eval-Driven Loops Turn Agent Tuning from Guesswork into Measurement</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Sat, 14 Nov 2026 00:08:36 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into-measurement-8b0</link>
      <guid>https://dev.to/mech_app_ai/harness-tuner-eval-driven-loops-turn-agent-tuning-from-guesswork-into-measurement-8b0</guid>
      <description>&lt;p&gt;Most agent tuning happens in a loop: tweak the prompt, run it manually, check the output, repeat. Harness Tuner replaces that manual cycle with an automated optimization loop that plans experiments, measures performance on a training set, validates on a held-out set, and never touches the test set until you explicitly promote a candidate.&lt;/p&gt;

&lt;p&gt;The tool targets three tuning dimensions: agent tools (which functions the agent can call), prompts (system and user instructions), and model selection (which LLM backend to use). Instead of guessing which change will improve performance, you define evals and let the loop measure.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the Tuning Loop Works
&lt;/h2&gt;

&lt;p&gt;Harness Tuner is an agentic optimizer. It reads task definitions from Kiln (a structured eval format), plans experiments, runs them, and tracks results. The loop follows a train/val/test discipline borrowed from ML pipelines:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Train set&lt;/strong&gt;: The optimizer runs experiments here, measuring performance and iterating.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validation set&lt;/strong&gt;: Promising candidates are tested here to check for overfitting.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test set&lt;/strong&gt;: Locked until you explicitly promote a configuration for final evaluation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The optimizer is itself an agent. It decides which dimension to tune next based on current performance metrics. If tool selection is the bottleneck, it experiments with different tool combinations. If the prompt is unclear, it generates variations and measures which one improves task completion.&lt;/p&gt;

&lt;h3&gt;
  
  
  Experiment Planning
&lt;/h3&gt;

&lt;p&gt;The optimizer maintains a state machine:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Baseline measurement&lt;/strong&gt;: Run the current configuration on the train set to establish a performance floor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hypothesis generation&lt;/strong&gt;: The agent proposes experiments (e.g., "Try adding a web search tool" or "Simplify the system prompt").&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Execution&lt;/strong&gt;: Run each experiment on the train set and log metrics.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validation&lt;/strong&gt;: If train performance improves, test on the validation set to confirm the gain is real.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Promotion&lt;/strong&gt;: If validation passes, mark the configuration as a candidate for production.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The loop never reads the test set during optimization. This prevents the classic ML mistake of tuning to your test data.&lt;/p&gt;

&lt;h2&gt;
  
  
  Eval Harness Format
&lt;/h2&gt;

&lt;p&gt;Harness Tuner consumes Kiln task definitions. A task is a JSON structure that includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Input&lt;/strong&gt;: The data the agent receives.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Expected output&lt;/strong&gt;: The correct answer or a scoring function.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tools&lt;/strong&gt;: Available functions the agent can call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt template&lt;/strong&gt;: The system and user instructions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here's a simplified example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"task_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"summarize_article"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"input"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"https://example.com/article"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"expected_output"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"summary"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"A concise summary of the article"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"key_points"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"point1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"point2"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"tools"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"fetch_url"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"extract_text"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"prompt"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"You are a summarization agent. Fetch the article and extract key points."&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The optimizer runs this task with different configurations and compares outputs to the expected result. Scoring can be exact match, semantic similarity, or a custom function.&lt;/p&gt;

&lt;h3&gt;
  
  
  Handling Non-Determinism
&lt;/h3&gt;

&lt;p&gt;LLM outputs are stochastic. Harness Tuner runs each experiment multiple times (configurable, default is 3) and aggregates results. It tracks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mean score&lt;/strong&gt;: Average performance across runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Variance&lt;/strong&gt;: How much the output fluctuates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pass rate&lt;/strong&gt;: Percentage of runs that meet the success threshold.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;High variance signals that a configuration is unstable. The optimizer penalizes configurations with high variance even if mean performance is good.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tuning Dimensions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Tool Selection
&lt;/h3&gt;

&lt;p&gt;The optimizer can add, remove, or swap tools. If an agent has access to &lt;code&gt;fetch_url&lt;/code&gt;, &lt;code&gt;search_web&lt;/code&gt;, and &lt;code&gt;extract_text&lt;/code&gt;, the optimizer might test:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Removing &lt;code&gt;search_web&lt;/code&gt; to see if it's a distraction.&lt;/li&gt;
&lt;li&gt;Adding &lt;code&gt;summarize_text&lt;/code&gt; to improve output quality.&lt;/li&gt;
&lt;li&gt;Swapping &lt;code&gt;fetch_url&lt;/code&gt; for a cached version to reduce latency.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each change is measured on the train set. If removing a tool improves performance (e.g., the agent was misusing it), the optimizer keeps the change.&lt;/p&gt;

&lt;h3&gt;
  
  
  Prompt Optimization
&lt;/h3&gt;

&lt;p&gt;The optimizer generates prompt variations using an LLM. It might:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Simplify verbose instructions.&lt;/li&gt;
&lt;li&gt;Add examples to clarify expected behavior.&lt;/li&gt;
&lt;li&gt;Reorder instructions to emphasize critical steps.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Prompt changes are risky because they can degrade performance in unexpected ways. The optimizer runs multiple variations in parallel and picks the one with the best train score, then validates it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Model Selection
&lt;/h3&gt;

&lt;p&gt;The optimizer can swap the underlying LLM. If you start with GPT-4, it might test:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Claude 3.5 Sonnet for tasks that need long context.&lt;/li&gt;
&lt;li&gt;GPT-4o-mini for tasks where speed matters more than quality.&lt;/li&gt;
&lt;li&gt;A local model for tasks with sensitive data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Model selection is measured on cost, latency, and accuracy. The optimizer balances these metrics based on weights you provide.&lt;/p&gt;

&lt;h2&gt;
  
  
  Overfitting Guardrails
&lt;/h2&gt;

&lt;p&gt;Automated tuning can overfit to the train set. Harness Tuner includes several guardrails:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Validation gate&lt;/strong&gt;: No configuration is promoted without passing validation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Diversity penalty&lt;/strong&gt;: The optimizer avoids making too many similar changes in a row.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rollback on regression&lt;/strong&gt;: If a change hurts validation performance, it's automatically reverted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test set lockout&lt;/strong&gt;: The test set is never used during optimization. You manually trigger test evaluation when you're ready to ship.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The optimizer also logs every experiment. You can inspect the history to see which changes were tried and why they were accepted or rejected.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Shape
&lt;/h2&gt;

&lt;p&gt;Harness Tuner runs as a standalone process. You point it at a Kiln task directory, configure the optimization budget (how many experiments to run), and let it work.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;harness-tuner optimize \
  --task-dir ./tasks \
  --budget 50 \
  --train-split 0.7 \
  --val-split 0.2 \
  --test-split 0.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The optimizer writes results to a SQLite database. You can query it to see experiment history, performance trends, and promoted configurations.&lt;/p&gt;

&lt;p&gt;For production, you export the winning configuration as a JSON file and load it into your agent runtime. Harness Tuner doesn't deploy agents; it just finds the best configuration.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability
&lt;/h2&gt;

&lt;p&gt;The optimizer logs every experiment to a structured format. Each log entry includes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Experiment ID&lt;/strong&gt;: Unique identifier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Configuration&lt;/strong&gt;: Tools, prompt, and model used.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Metrics&lt;/strong&gt;: Score, latency, cost, and variance.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dataset split&lt;/strong&gt;: Train, val, or test.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Timestamp&lt;/strong&gt;: When the experiment ran.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can export logs to your observability stack (e.g., Datadog, Grafana) or query them locally. The optimizer also exposes a simple web UI that shows:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Current best configuration.&lt;/li&gt;
&lt;li&gt;Performance over time.&lt;/li&gt;
&lt;li&gt;Experiment history.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Failure Modes
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure Mode&lt;/th&gt;
&lt;th&gt;Cause&lt;/th&gt;
&lt;th&gt;Mitigation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Overfitting to train set&lt;/td&gt;
&lt;td&gt;Optimizer runs too many experiments on the same data&lt;/td&gt;
&lt;td&gt;Use a large train set and strict validation gate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unstable configurations&lt;/td&gt;
&lt;td&gt;High variance in LLM outputs&lt;/td&gt;
&lt;td&gt;Increase number of runs per experiment and penalize variance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Expensive experiments&lt;/td&gt;
&lt;td&gt;Testing large models or slow tools&lt;/td&gt;
&lt;td&gt;Set cost and latency budgets; optimizer will skip expensive configs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Prompt degradation&lt;/td&gt;
&lt;td&gt;Optimizer generates a worse prompt that still scores well on train&lt;/td&gt;
&lt;td&gt;Manual review of promoted prompts before production&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool misuse&lt;/td&gt;
&lt;td&gt;Optimizer adds a tool that the agent uses incorrectly&lt;/td&gt;
&lt;td&gt;Monitor tool call logs and add negative examples to the eval set&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  When to Use Harness Tuner
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use it when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You have a structured eval set and want to systematically improve agent performance.&lt;/li&gt;
&lt;li&gt;You're tuning multiple dimensions (tools, prompts, models) and manual iteration is too slow.&lt;/li&gt;
&lt;li&gt;You need to prevent overfitting and want a train/val/test discipline.&lt;/li&gt;
&lt;li&gt;You want experiment tracking and reproducibility for agent configurations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid it when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You don't have a good eval set. The optimizer is only as good as your evals.&lt;/li&gt;
&lt;li&gt;Your agent tasks are too diverse. The optimizer works best on a narrow task domain.&lt;/li&gt;
&lt;li&gt;You need real-time tuning. The optimization loop is batch-oriented, not online.&lt;/li&gt;
&lt;li&gt;Your agent is already good enough. Don't optimize for the sake of optimizing.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;Harness Tuner is a closed-loop optimizer for agent configurations. It automates the tedious work of testing tool combinations, prompt variations, and model choices. The train/val/test split prevents overfitting, and the experiment tracking gives you a clear history of what worked and why.&lt;/p&gt;

&lt;p&gt;The main risk is garbage-in-garbage-out. If your eval set is bad, the optimizer will tune to the wrong target. Spend time building good evals before you turn on automated tuning.&lt;/p&gt;

&lt;p&gt;For teams running production agents, this is a useful primitive. It turns agent tuning from a manual art into a measured process. Just don't expect it to fix a fundamentally broken agent design.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/Kiln-AI/harness-tuner" rel="noopener noreferrer"&gt;Harness Tuner GitHub Repository&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://news.ycombinator.com/item?id=50484443" rel="noopener noreferrer"&gt;Show HN Discussion&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>automation</category>
      <category>llm</category>
    </item>
    <item>
      <title>CoReMem: Cost-Internalizing Memory for Successive Coding Tasks with Prospective Valuation</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Fri, 13 Nov 2026 20:06:50 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/coremem-cost-internalizing-memory-for-successive-coding-tasks-with-prospective-valuation-1pdg</link>
      <guid>https://dev.to/mech_app_ai/coremem-cost-internalizing-memory-for-successive-coding-tasks-with-prospective-valuation-1pdg</guid>
      <description>&lt;p&gt;Coding agents that work across successive tasks on an evolving repository face a memory problem: they must decide what to remember before they know what the next task will ask. Every memory segment costs tokens to write, tokens to retrieve, and context window space to store. The agent needs to predict which fragments will pay back those costs when the next task arrives.&lt;/p&gt;

&lt;p&gt;CoReMem (Coding-Repository Memory) treats memory selection as a cost-benefit problem. It measures the prospective value of each trajectory segment by running the next task twice: once with the segment included, once without. The difference in resolution rate and step count becomes the segment's measured value. The agent then builds a compact memory around anchors (repository locations and executable checks) that showed positive value, and learns when to act on that memory versus waiting for more context.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Memory Selection Problem
&lt;/h2&gt;

&lt;p&gt;Most coding agents either dump everything into memory or use heuristic filters (error messages, file edits, test results). Neither approach internalizes cost. Dumping everything bloats context and slows retrieval. Heuristic filters discard segments that might be useful and keep segments that won't be.&lt;/p&gt;

&lt;p&gt;CoReMem frames the problem explicitly:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Write cost&lt;/strong&gt;: tokens spent serializing the segment&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read cost&lt;/strong&gt;: tokens spent retrieving and re-parsing it&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Storage cost&lt;/strong&gt;: context window space occupied until the segment is used or pruned&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Future utility&lt;/strong&gt;: expected improvement in resolution rate and step reduction on unknown next tasks&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The agent selects memory when expected utility exceeds combined costs. This requires estimating utility before the next task is revealed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prospective Segment Valuation
&lt;/h2&gt;

&lt;p&gt;CoReMem runs a two-pass evaluation on each candidate segment from the previous task's trajectory:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Baseline run&lt;/strong&gt;: execute the next task starting from the inherited repository state, without the candidate segment in memory&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory run&lt;/strong&gt;: execute the same next task from the same state, with the candidate segment included&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The segment's value is the difference in outcome (resolved vs. unresolved) and efficiency (step count, token usage). Segments that improve resolution or reduce steps get positive value. Segments that add no benefit or slow the agent down get zero or negative value.&lt;/p&gt;

&lt;p&gt;This approach requires access to the next task for measurement, so it works as a training signal rather than a runtime filter. The agent learns which segment types (error traces, file diffs, test outputs, dependency graphs) tend to have high prospective value across task distributions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Anchor-Based Memory Structure
&lt;/h2&gt;

&lt;p&gt;CoReMem organizes memory around two anchor types:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Location anchors&lt;/strong&gt;: file paths, function names, class definitions, variable scopes&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Executable anchors&lt;/strong&gt;: test commands, linter invocations, build steps, runtime checks&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each anchor links to the trajectory segment that produced it. When the agent encounters a location or needs to run a check, it retrieves the linked segment. This structure keeps memory compact and makes retrieval deterministic.&lt;/p&gt;

&lt;p&gt;The agent prunes anchors when their linked segments show low prospective value across multiple tasks. Pruning happens between tasks, not mid-task, to avoid destabilizing the current rollout.&lt;/p&gt;

&lt;h2&gt;
  
  
  Actor Adaptation with Provenance-Gated Credit
&lt;/h2&gt;

&lt;p&gt;The agent learns when to act on memory versus when to wait for more information. CoReMem uses a reward function that prioritizes resolution and penalizes step count in successful runs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;reward&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resolution_bonus&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;step_count&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;step_penalty&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The agent runs rollout groups with and without memory access. The difference in on-policy reward estimates the memory's value. Provenance-gated credit assignment splits this value:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Outcome credit&lt;/strong&gt;: every step in a successful run gets the resolution bonus&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory credit&lt;/strong&gt;: only steps that used memory-supplied anchors get the additional value term from the memory-vs-no-memory comparison&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This prevents the agent from over-attributing success to memory when the task would have resolved without it. It also prevents under-attribution when memory enables a shortcut that saves steps.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture and Flow
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;┌─────────────────┐
│  Task N (done)  │
│  - trajectory   │
│  - repo state   │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────┐
│  Prospective Valuation          │
│  - run Task N+1 without segment │
│  - run Task N+1 with segment    │
│  - measure Δ(resolution, steps) │
└────────┬────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│  Anchor Selection               │
│  - keep high-value segments     │
│  - prune low-value segments     │
│  - serialize anchors + links    │
└────────┬────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│  Task N+1 (new)                 │
│  - actor retrieves anchors      │
│  - provenance tracks usage      │
│  - reward split: outcome + mem  │
└─────────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cycle repeats for each successive task. The agent's memory grows when new anchors prove valuable and shrinks when old anchors stop paying back their cost.&lt;/p&gt;

&lt;h2&gt;
  
  
  Implementation Considerations
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Trade-off&lt;/th&gt;
&lt;th&gt;Mitigation&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Prospective valuation&lt;/td&gt;
&lt;td&gt;Requires running next task twice (expensive)&lt;/td&gt;
&lt;td&gt;Amortize over batch of segments; use cheaper proxy model for initial filtering&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Anchor pruning&lt;/td&gt;
&lt;td&gt;Aggressive pruning loses useful context; conservative pruning bloats memory&lt;/td&gt;
&lt;td&gt;Track anchor usage frequency and recency; prune when both are low&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Provenance tracking&lt;/td&gt;
&lt;td&gt;Adds instrumentation overhead to every step&lt;/td&gt;
&lt;td&gt;Use lightweight span IDs; batch provenance writes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reward shaping&lt;/td&gt;
&lt;td&gt;Step penalty may discourage thorough exploration&lt;/td&gt;
&lt;td&gt;Tune penalty coefficient per task distribution; use separate exploration budget&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The paper reports results on ChainSWE, a benchmark of successive coding tasks on evolving repositories. CoReMem raised the 27B actor's resolve rate from 21.0% to 25.7%, with 17% fewer steps and 11% fewer total cache-aware tokens (including memory module calls) relative to the untrained actor without memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  When Memory Value Goes Negative
&lt;/h2&gt;

&lt;p&gt;Memory can hurt performance in three scenarios:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Stale anchors&lt;/strong&gt;: the repository evolved and the anchor points to outdated code&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Misleading context&lt;/strong&gt;: the segment describes a problem that no longer exists or a solution that no longer applies&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retrieval overhead&lt;/strong&gt;: the cost of fetching and parsing the segment exceeds the value it provides&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;CoReMem handles staleness by re-validating location anchors against the current repository state. If a file path or function name no longer exists, the anchor is pruned immediately. Executable anchors are re-run to confirm they still produce useful signals.&lt;/p&gt;

&lt;p&gt;Misleading context is harder to detect. The agent relies on prospective valuation to identify segments that consistently reduce performance. If a segment type shows negative value across multiple tasks, the agent stops selecting it.&lt;/p&gt;

&lt;p&gt;Retrieval overhead is managed by tracking token counts for each anchor retrieval. If retrieval cost exceeds the segment's measured value, the anchor is marked for pruning.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Shape
&lt;/h2&gt;

&lt;p&gt;CoReMem requires three components:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Memory module&lt;/strong&gt;: stores anchors, handles retrieval, tracks provenance&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Valuation pipeline&lt;/strong&gt;: runs paired task executions, computes segment value&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Actor training loop&lt;/strong&gt;: collects rollouts, assigns credit, updates policy&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The memory module can be a separate service with a vector store for anchor embeddings and a key-value store for segment content. The valuation pipeline runs offline between task batches. The actor training loop runs online during task execution.&lt;/p&gt;

&lt;p&gt;For production deployment, the valuation pipeline can be replaced with a learned value estimator that predicts segment value without running the next task. This trades measurement accuracy for speed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Value estimation drift&lt;/strong&gt;: if the task distribution shifts, historical value measurements become unreliable&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Anchor collision&lt;/strong&gt;: multiple segments link to the same anchor, causing retrieval ambiguity&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Credit leakage&lt;/strong&gt;: provenance tracking misattributes memory usage when the agent uses memory indirectly (e.g., memory informs a search query that finds the answer)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pruning thrash&lt;/strong&gt;: the agent prunes an anchor, then re-discovers it in the next task, wasting write cost&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The paper does not address multi-agent scenarios where memory must be shared or synchronized across actors working on the same repository.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use CoReMem when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your agent works on successive tasks where repository state carries forward&lt;/li&gt;
&lt;li&gt;Memory bloat is a bottleneck (context window exhaustion, slow retrieval)&lt;/li&gt;
&lt;li&gt;You can afford offline valuation runs between task batches&lt;/li&gt;
&lt;li&gt;You need explainable memory selection (anchors provide clear provenance)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid CoReMem when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Tasks are independent (no repository evolution)&lt;/li&gt;
&lt;li&gt;You need real-time memory selection without offline training&lt;/li&gt;
&lt;li&gt;Your task distribution shifts frequently (value estimates go stale)&lt;/li&gt;
&lt;li&gt;You cannot instrument the actor for provenance tracking&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The framework is most valuable for long-running coding agents that maintain a repository over weeks or months. For short-lived agents or one-shot tasks, the valuation overhead exceeds the memory savings.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://arxiv.org/abs/2611.17316v1" rel="noopener noreferrer"&gt;CoReMem paper on ArXiv&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://arxiv.org/pdf/2611.17316v1.pdf" rel="noopener noreferrer"&gt;PDF&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Kapso's Iterative Learning Loop: Enterprise Agents That Build Knowledge Graphs Without Retraining</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Fri, 13 Nov 2026 10:07:09 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/kapsos-iterative-learning-loop-enterprise-agents-that-build-knowledge-graphs-without-retraining-12k2</link>
      <guid>https://dev.to/mech_app_ai/kapsos-iterative-learning-loop-enterprise-agents-that-build-knowledge-graphs-without-retraining-12k2</guid>
      <description>&lt;p&gt;Most enterprise AI agents treat knowledge as static. You embed documents, index them, and hope retrieval finds the right context. Kapso takes a different approach: it runs experiments against enterprise data, learns from the results, and updates a knowledge graph incrementally. No retraining cycles. No upfront ontology design. Just a feedback loop that compounds understanding across investigations.&lt;/p&gt;

&lt;p&gt;The architecture matters because enterprise data environments are messy. One of Kapso's reference projects involved migrating 30+ source systems, thousands of tables, and hundreds of BI reports to a lakehouse. No single person or agent understood how the estate connected. The only path forward was iterative: investigate repos and dbt models, propose hypotheses about data lineage, run experiments, observe failures, and update the knowledge graph with what worked.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Iterative Learning Loop
&lt;/h2&gt;

&lt;p&gt;Kapso's core is a three-stage cycle:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Investigation&lt;/strong&gt;: The agent queries existing knowledge graph nodes, retrieves relevant documents, and identifies gaps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Experimentation&lt;/strong&gt;: It proposes hypotheses (e.g., "Table X feeds Report Y via transformation Z"), runs tests in a sandbox, and collects outcomes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Learning&lt;/strong&gt;: Successful experiments update the knowledge graph. Failed experiments store failure modes as negative examples.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This differs from RAG-only systems in two ways. First, retrieval is not the end state. It feeds into hypothesis generation. Second, the knowledge graph is mutable. Each experiment can add nodes (new entities), edges (relationships), or attributes (constraints learned from failures).&lt;/p&gt;

&lt;h3&gt;
  
  
  State Management Across Sessions
&lt;/h3&gt;

&lt;p&gt;Kapso maintains three state layers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Session state&lt;/strong&gt;: Tracks the current investigation's context, including active hypotheses and pending experiments.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Graph state&lt;/strong&gt;: Persists the knowledge graph itself, with versioning for rollback when experiments invalidate prior assumptions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Experiment log&lt;/strong&gt;: Stores every test run, including inputs, outputs, and whether the hypothesis was confirmed or rejected.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The session state is ephemeral. When an agent completes an investigation, the session closes, but confirmed knowledge moves into the graph. This separation prevents partial or speculative updates from polluting the shared knowledge base.&lt;/p&gt;

&lt;h3&gt;
  
  
  When to Query vs. When to Mutate
&lt;/h3&gt;

&lt;p&gt;The decision logic is straightforward:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Query&lt;/strong&gt;: If the agent's question maps to existing graph nodes and edges, retrieve and return.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mutate&lt;/strong&gt;: If retrieval returns incomplete or conflicting information, or if an experiment confirms a new relationship, update the graph.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The mutation path includes a conflict resolution step. If two agents propose contradictory edges (e.g., "Table A feeds Report B" vs. "Table A does not feed Report B"), the system prioritizes the hypothesis with more recent experimental evidence. Older edges are marked as deprecated, not deleted, so you can trace how understanding evolved.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture: Sandbox, Orchestrator, Graph Store
&lt;/h2&gt;

&lt;p&gt;Here's the deployment shape:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Role&lt;/th&gt;
&lt;th&gt;Failure Mode&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Orchestrator&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Manages agent lifecycle, schedules experiments, routes queries&lt;/td&gt;
&lt;td&gt;Deadlock if circular dependencies in hypotheses&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Sandbox&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Isolated environment for running experiments (SQL, dbt, Python)&lt;/td&gt;
&lt;td&gt;Resource exhaustion if experiments don't timeout&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Graph Store&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Persists knowledge graph with ACID guarantees (Neo4j, TigerGraph)&lt;/td&gt;
&lt;td&gt;Stale reads if replication lag exceeds session TTL&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Experiment Log&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Append-only log of all test runs (S3, Postgres with JSONB)&lt;/td&gt;
&lt;td&gt;Unbounded growth without retention policy&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Retrieval Layer&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Embeds documents and queries for investigation phase (Pinecone, Qdrant)&lt;/td&gt;
&lt;td&gt;Embedding drift if documents update without reindex&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The orchestrator is stateless. It reads session state from the graph store, dispatches experiments to the sandbox, and writes results back. This makes horizontal scaling simple: spin up more orchestrator instances and point them at the same graph store.&lt;/p&gt;

&lt;p&gt;The sandbox is where risk lives. Experiments run arbitrary code (SQL queries, dbt transformations, Python scripts). Kapso isolates each experiment in a containerized environment with resource limits (CPU, memory, wall-clock time). If an experiment hangs or consumes too much memory, the sandbox kills it and logs the failure. The agent learns that the hypothesis was invalid or too expensive.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code: Experiment Execution Flow
&lt;/h2&gt;

&lt;p&gt;Here's a simplified version of how Kapso runs an experiment and updates the graph:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_experiment&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;graph_store&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;experiment_log&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""
    Execute a hypothesis in the sandbox and update the knowledge graph.
    """&lt;/span&gt;
    &lt;span class="n"&gt;session_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session_id&lt;/span&gt;
    &lt;span class="n"&gt;experiment_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;generate_id&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Step 1: Prepare sandbox environment&lt;/span&gt;
    &lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load_context&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;required_tables&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Step 2: Execute hypothesis test&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sandbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;test_code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;timeout_seconds&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;max_memory_mb&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2048&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;success&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exit_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;assertions_passed&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;SandboxTimeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;success&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"timeout"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;SandboxOOM&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;success&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;False&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"out_of_memory"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="c1"&gt;# Step 3: Log experiment&lt;/span&gt;
    &lt;span class="n"&gt;experiment_log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="s2"&gt;"experiment_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;experiment_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"session_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;session_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"hypothesis"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"success"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;success&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"result"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s2"&gt;"timestamp"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Step 4: Update graph if successful&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;success&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;graph_store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_edge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;source_entity&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target_entity&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;relationship&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relationship_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="s2"&gt;"experiment_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;experiment_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="s2"&gt;"confidence"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;confidence_score&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# Store negative example&lt;/span&gt;
        &lt;span class="n"&gt;graph_store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add_negative_edge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;source_entity&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;target_entity&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;relationship&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hypothesis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relationship_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;success&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key detail: negative edges. When an experiment fails, Kapso doesn't just discard the hypothesis. It stores the failure in the graph so future investigations don't repeat the same mistake. This is how the system compounds learning.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability: What Triggered Graph Mutations
&lt;/h2&gt;

&lt;p&gt;You need visibility into which documents or experiments caused graph changes. Kapso exposes three observability hooks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Mutation log&lt;/strong&gt;: Every graph update includes a reference to the experiment ID and the documents that informed the hypothesis.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lineage trace&lt;/strong&gt;: For any graph edge, you can query backward to see the chain of experiments and retrievals that led to it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Conflict dashboard&lt;/strong&gt;: Shows deprecated edges, the experiments that invalidated them, and the new edges that replaced them.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The mutation log is append-only. You can replay it to reconstruct the graph at any point in time. This is useful for debugging when an agent makes a bad decision based on outdated knowledge.&lt;/p&gt;

&lt;h2&gt;
  
  
  Multi-Agent Conflict Resolution
&lt;/h2&gt;

&lt;p&gt;When multiple agents iterate on overlapping data sources, conflicts are inevitable. Kapso handles this with optimistic locking at the graph level:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Each agent reads the graph with a version timestamp.&lt;/li&gt;
&lt;li&gt;When an agent proposes a mutation, the graph store checks if the relevant nodes or edges have changed since the read.&lt;/li&gt;
&lt;li&gt;If they have, the mutation is rejected, and the agent retries with fresh data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This prevents lost updates but introduces retry overhead. In practice, conflicts are rare because agents work on different subgraphs (different source systems or BI reports). When conflicts do occur, the retry loop usually resolves them within one or two attempts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deployment Considerations
&lt;/h2&gt;

&lt;p&gt;Kapso is not a SaaS. It's an engine you deploy in your own infrastructure. The reference architecture assumes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Graph store&lt;/strong&gt;: Managed Neo4j or TigerGraph instance with automated backups.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox&lt;/strong&gt;: Kubernetes cluster with pod-level resource limits and network isolation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Orchestrator&lt;/strong&gt;: Stateless service deployed as a Kubernetes Deployment with horizontal pod autoscaling.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Experiment log&lt;/strong&gt;: S3 or Postgres with a retention policy (e.g., keep experiments for 90 days, then archive to cold storage).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The sandbox is the bottleneck. Each experiment needs its own isolated environment, which means spinning up containers. If you're running hundreds of experiments per hour, you need a cluster with enough capacity to handle the burst. Kapso includes a queueing layer (backed by Redis or RabbitMQ) to smooth out spikes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Security Boundaries
&lt;/h2&gt;

&lt;p&gt;Enterprise data is sensitive. Kapso enforces three security layers:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox isolation&lt;/strong&gt;: Experiments run in ephemeral containers with no network access except to the graph store and experiment log.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Graph-level access control&lt;/strong&gt;: Nodes and edges can have ACLs. An agent can only query or mutate parts of the graph it has permissions for.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Audit trail&lt;/strong&gt;: Every graph mutation is logged with the agent ID, session ID, and experiment ID. You can trace any change back to the agent and investigation that triggered it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The sandbox isolation is critical. If an experiment runs malicious code (intentionally or via prompt injection), it can't exfiltrate data or pivot to other systems. The container dies when the experiment completes, and all state is discarded.&lt;/p&gt;

&lt;h2&gt;
  
  
  When This Makes Sense
&lt;/h2&gt;

&lt;p&gt;Kapso is overkill for simple retrieval tasks. If you just need to answer questions from a static document corpus, use a vector store and a retrieval-augmented generation pipeline. Kapso is for environments where:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The knowledge base is incomplete and evolving.&lt;/li&gt;
&lt;li&gt;You need to run experiments (SQL queries, transformations, simulations) to validate hypotheses.&lt;/li&gt;
&lt;li&gt;Multiple agents or teams are iterating on the same data, and you want to compound learning across investigations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The reference use case (migrating a data estate) is a good fit because the knowledge graph starts empty. You don't know how tables connect or which transformations feed which reports. The only way to learn is to investigate, experiment, and update the graph as you go.&lt;/p&gt;

&lt;h2&gt;
  
  
  When to Avoid It
&lt;/h2&gt;

&lt;p&gt;Skip Kapso if:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your knowledge base is static or changes infrequently. The overhead of maintaining a mutable graph and experiment log isn't worth it.&lt;/li&gt;
&lt;li&gt;You don't need to run experiments. If retrieval alone solves your problem, a vector store is simpler.&lt;/li&gt;
&lt;li&gt;You can't tolerate the operational complexity of managing a graph store, sandbox cluster, and orchestrator.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The system also assumes you have engineering resources to write experiment code (SQL, dbt, Python). If your agents need to generate and run arbitrary experiments without human oversight, you're in reinforcement learning territory, and Kapso's supervised experiment loop won't help.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;Kapso exposes the plumbing of continuous learning agents: stateful update cycles, conflict resolution in graph mutations, and observability for incremental knowledge extraction. It's a good fit for enterprise data projects where the knowledge base is incomplete and you need to compound learning across investigations. The architecture is sound (stateless orchestrator, isolated sandbox, versioned graph store), but the operational overhead is real. You need Kubernetes, a managed graph database, and enough compute to run experiments in parallel.&lt;/p&gt;

&lt;p&gt;Use it when you're migrating data estates, building lakehouses, or investigating complex systems where no one has the full picture. Avoid it if your knowledge base is static, your agents don't need to run experiments, or you can't justify the infrastructure cost.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://news.ycombinator.com/item?id=50471281" rel="noopener noreferrer"&gt;Hacker News Discussion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/Leeroo-AI/kapso" rel="noopener noreferrer"&gt;Kapso GitHub Repository&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://leeroo.com/" rel="noopener noreferrer"&gt;Leeroo Company Site&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>ai</category>
      <category>architecture</category>
      <category>data</category>
    </item>
    <item>
      <title>RAG Permissions at Retrieval Time: Why \\\\\\\"Don't Show It\\\\\\\" Fails as Agent Access Control</title>
      <dc:creator>mech.app</dc:creator>
      <pubDate>Fri, 13 Nov 2026 10:07:08 +0000</pubDate>
      <link>https://dev.to/mech_app_ai/rag-permissions-at-retrieval-time-why-dont-show-it-fails-as-agent-access-control-348j</link>
      <guid>https://dev.to/mech_app_ai/rag-permissions-at-retrieval-time-why-dont-show-it-fails-as-agent-access-control-348j</guid>
      <description>&lt;p&gt;Tutorial RAG systems embed a folder and return the top five chunks. Production RAG systems embed a company with HR files, board decks, customer contracts, and incident reports where half the pages are restricted to specific teams. The moment multiple users with different permissions query the same index, you have an authorization problem that prompt instructions cannot solve.&lt;/p&gt;

&lt;p&gt;The most common mistake is trying to enforce access control in the prompt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;You may only use documents the user is permitted to see. 
Do not reveal confidential information.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;By the time the model reads that instruction, the confidential chunk is already in its context window. You are asking a probabilistic text generator to un-know something you just handed it. Sometimes it complies. Sometimes a well-phrased follow-up or a prompt injection inside another document extracts it anyway.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where Post-Hoc Filtering Breaks
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Failure 1: Filtering After Retrieval
&lt;/h3&gt;

&lt;p&gt;A pattern that looks reasonable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;vector_store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;similarity_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;allowed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;user_can_read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"doc_id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;context&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;allowed&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This has two problems. First, if the filter has a bug or a missing branch, the fallback is "everything." Second, top-k is computed over documents the user cannot see. A restricted folder full of near matches can push every allowed result out of the window, leaving the user with an empty or degraded answer.&lt;/p&gt;

&lt;h3&gt;
  
  
  Failure 2: Metadata Leakage in Hybrid Search
&lt;/h3&gt;

&lt;p&gt;Hybrid search systems merge vector similarity results with keyword BM25 results. If the vector store filters by ACL but the keyword index does not, the merge step can leak document titles, folder names, or snippet previews from forbidden documents. The LLM never sees the full text, but it sees enough metadata to infer structure or content.&lt;/p&gt;

&lt;h3&gt;
  
  
  Failure 3: Re-Ranking Without Re-Authorization
&lt;/h3&gt;

&lt;p&gt;Many RAG pipelines retrieve 50 chunks, then use a cross-encoder or LLM to re-rank them down to 5. If the re-ranker runs after the ACL filter, you are fine. If it runs before, the re-ranker sees forbidden documents and may promote them based on relevance. The final filter then removes them, but the ranking signal has already been contaminated.&lt;/p&gt;

&lt;h2&gt;
  
  
  Retrieval-Time Access Control Architecture
&lt;/h2&gt;

&lt;p&gt;Production systems enforce permissions during the similarity search, not after. This requires injecting user context into the query itself.&lt;/p&gt;

&lt;h3&gt;
  
  
  Option 1: Metadata Filters in the Vector Store
&lt;/h3&gt;

&lt;p&gt;Most vector databases support metadata filters that run alongside the similarity search:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;chunks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;vector_store&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;similarity_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;filter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"$or"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"owner_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"shared_with"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"$contains"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"visibility"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"public"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This works when ACLs are simple (owner, shared list, public flag). It breaks down when permissions are computed dynamically (group membership, role hierarchies, time-based grants). The vector store has no way to evaluate complex authorization logic.&lt;/p&gt;

&lt;h3&gt;
  
  
  Option 2: Separate Indexes Per Tenant
&lt;/h3&gt;

&lt;p&gt;For multi-tenant SaaS, some teams maintain one vector index per customer. Retrieval is scoped to the tenant's index by default. This eliminates cross-tenant leakage but creates operational overhead. You need to manage index lifecycle (create, update, delete), handle schema migrations across thousands of indexes, and route queries to the correct index based on authentication context.&lt;/p&gt;

&lt;h3&gt;
  
  
  Option 3: ACL-Aware Embeddings
&lt;/h3&gt;

&lt;p&gt;Embed the document along with its access control metadata. Store &lt;code&gt;(embedding, doc_id, acl_hash)&lt;/code&gt; in the vector store. At query time, compute the user's effective ACL hash and filter results to matching hashes. This works when ACLs are stable and coarse-grained (department-level, project-level). It fails when permissions change frequently or are user-specific, because you would need to re-embed and re-index every affected document.&lt;/p&gt;

&lt;h3&gt;
  
  
  Option 4: Authorization Proxy Layer
&lt;/h3&gt;

&lt;p&gt;Place an authorization service between the RAG pipeline and the vector store. The proxy intercepts every retrieval request, fetches the user's effective permissions from an external policy engine (OPA, Zanzibar, custom RBAC), and rewrites the vector query to include the appropriate filters. This decouples permission logic from the vector store and the LLM orchestration layer, but it adds latency and a new failure mode.&lt;/p&gt;

&lt;h2&gt;
  
  
  Performance Trade-Offs
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;Latency&lt;/th&gt;
&lt;th&gt;Index Size&lt;/th&gt;
&lt;th&gt;Permission Granularity&lt;/th&gt;
&lt;th&gt;Operational Complexity&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Metadata filters in vector store&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;1x&lt;/td&gt;
&lt;td&gt;Coarse (static tags)&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Separate indexes per tenant&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;N×1x&lt;/td&gt;
&lt;td&gt;Perfect (tenant-scoped)&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ACL-aware embeddings&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;1x&lt;/td&gt;
&lt;td&gt;Medium (group-level)&lt;/td&gt;
&lt;td&gt;Medium (re-indexing cost)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Authorization proxy&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;1x&lt;/td&gt;
&lt;td&gt;Fine (user-level, dynamic)&lt;/td&gt;
&lt;td&gt;Medium (new service)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Post-hoc filtering&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;1x&lt;/td&gt;
&lt;td&gt;None (security theater)&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  State Management and Cache Invalidation
&lt;/h2&gt;

&lt;p&gt;When permissions change, the retrieval layer needs to know immediately. If a user is removed from a project, their next query must not return documents from that project. This requires either:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Synchronous permission checks&lt;/strong&gt; on every retrieval (high latency, high load on the authorization service)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Short-lived permission caches&lt;/strong&gt; with aggressive TTLs (5-60 seconds)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Event-driven cache invalidation&lt;/strong&gt; where the authorization service pushes updates to the RAG layer when ACLs change&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most production systems use a hybrid: cache permissions for 30 seconds, but subscribe to a permission-change event stream to invalidate specific user caches immediately.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability Gaps
&lt;/h2&gt;

&lt;p&gt;Standard RAG observability tracks query latency, chunk relevance, and LLM token usage. It does not track:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;How many chunks were filtered out due to permissions&lt;/li&gt;
&lt;li&gt;Whether the user received a degraded answer because all relevant chunks were forbidden&lt;/li&gt;
&lt;li&gt;Whether a query returned zero results due to an ACL misconfiguration versus genuinely missing data&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You need custom metrics:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;record&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rag.chunks.retrieved"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;all_chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;record&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rag.chunks.allowed"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;allowed_chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;record&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rag.chunks.filtered"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;all_chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;allowed_chunks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;chunks.filtered&lt;/code&gt; is consistently high for a user or query pattern, either the ACLs are too restrictive or the index contains too many documents the user cannot see. Both are actionable signals.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Silent degradation.&lt;/strong&gt; The user asks a question. The answer is technically correct but incomplete because the most relevant chunks were filtered out. The LLM has no way to know it is missing context, so it does not tell the user. The user assumes the answer is complete.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ACL drift.&lt;/strong&gt; Permissions are updated in the source system (Google Drive, Notion, Confluence) but not propagated to the vector store metadata. The RAG system continues to enforce stale permissions, either over-restricting (user cannot see documents they should) or under-restricting (user sees documents they should not).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Metadata injection.&lt;/strong&gt; An attacker uploads a document with crafted metadata (&lt;code&gt;{"shared_with": ["*"]}&lt;/code&gt;). If the ingestion pipeline does not validate metadata against the source system's actual ACLs, the document becomes visible to everyone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Technical Verdict
&lt;/h2&gt;

&lt;p&gt;Use retrieval-time filtering (metadata filters or separate indexes) when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You have more than one user with different permissions querying the same corpus&lt;/li&gt;
&lt;li&gt;Your ACLs are stable and can be represented as metadata tags or tenant IDs&lt;/li&gt;
&lt;li&gt;You can tolerate 30-second permission propagation delays&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Avoid post-hoc filtering (prompt instructions or post-retrieval ACL checks) when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You are building a multi-tenant SaaS product&lt;/li&gt;
&lt;li&gt;Your users have different security clearances or data access levels&lt;/li&gt;
&lt;li&gt;A permission violation would violate compliance requirements (HIPAA, SOC 2, GDPR)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Use an authorization proxy when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Permissions are computed dynamically (role hierarchies, attribute-based access control)&lt;/li&gt;
&lt;li&gt;You need sub-second permission propagation&lt;/li&gt;
&lt;li&gt;You already have a centralized policy engine (OPA, Zanzibar)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Do not rely on the LLM to enforce access control. By the time the model sees the instruction, the access decision has already been made.&lt;/p&gt;

&lt;h2&gt;
  
  
  Source Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.to/cyclopt_dimitrisk/dont-reveal-it-is-not-an-access-control-rag-permissions-belong-in-retrieval-3475"&gt;Original article on Dev.to&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>agents</category>
      <category>llm</category>
      <category>rag</category>
      <category>security</category>
    </item>
  </channel>
</rss>
