<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: 4sapi</title>
    <description>The latest articles on DEV Community by 4sapi (@4sapi).</description>
    <link>https://dev.to/4sapi</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4208416%2F9939a1de-e380-41fc-8162-01ed67aacc88.png</url>
      <title>DEV Community: 4sapi</title>
      <link>https://dev.to/4sapi</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/4sapi"/>
    <language>en</language>
    <item>
      <title>Breaking Down Jev: A Decision Model That Takes Over LLM’s Most Expensive Step</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Thu, 19 Nov 2026 09:01:46 +0000</pubDate>
      <link>https://dev.to/4sapi/breaking-down-jev-a-decision-model-that-takes-over-llms-most-expensive-step-a7m</link>
      <guid>https://dev.to/4sapi/breaking-down-jev-a-decision-model-that-takes-over-llms-most-expensive-step-a7m</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;Large language models are widely used for classification, routing, filtering and binary judgment tasks. Developers often default to general-purpose LLMs for these decision workloads, even though text generation is unnecessary. Every token produced in these tasks drives up latency and inference cost, and most of the output tokens add no business value. This is the core pain point that Jev is designed to solve.&lt;/p&gt;

&lt;p&gt;Jev is not a general chat model. It is a specialized decision model built to output probability distributions over candidate options rather than descriptive text. It skips autoregressive token generation and returns structured choice scores in a single forward pass. For workloads that only need judgment instead of written content, Jev cuts both cost and latency drastically. As TypeSage rolled out Jev to production in September 2026, the service replaced a large share of routing and classification traffic that previously ran on general LLMs. Many engineering teams have started to treat Jev as a dedicated decision layer, separating judgment logic from generative workflows.&lt;/p&gt;

&lt;p&gt;This article unpacks the core mechanics of Jev, contrasts it with structured outputs from standard LLMs, explains its calibration training method RLCD, compares benchmark metrics, defines suitable use cases, and provides a practical five-step rollout workflow. All cost and performance benchmarks referenced in the analysis are sourced from public test data and official release reports.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. What Jev Actually Does
&lt;/h2&gt;

&lt;p&gt;The standard LLM pattern for classification tasks follows a slow, token-by-token generation pipeline. The model produces text such as &lt;code&gt;{"category":"billing","confidence":0.8}&lt;/code&gt;. Engineers then parse the string with regular expressions or JSON parsers to extract results. This workflow has three inherent drawbacks.&lt;br&gt;
First, output generation is sequential. The model emits one token at a time, and classification tasks have no requirement for coherent long-form text. Every generated token consumes compute resources.&lt;br&gt;
Second, partial streaming results are unreliable. If the pipeline receives only a fragment of the final JSON object, downstream systems cannot safely consume incomplete judgment data.&lt;br&gt;
Third, latency accumulates from generation overhead. Even simple classification jobs can be delayed waiting for the full JSON payload.&lt;/p&gt;

&lt;p&gt;Jev changes this paradigm. It extracts the judgment logic from the generation loop and computes the full probability distribution in one pass. A typical Jev response contains no natural language sentences. It directly returns a list of choices, each paired with a probability score. The API response only carries structured distribution data, eliminating the need for string parsing and reducing output token overhead.&lt;/p&gt;

&lt;p&gt;The model supports four primary judgment modes that cover most production decision scenarios.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Choice&lt;/strong&gt;: Discrete selection from a fixed set of options, such as routing tickets to billing, support or security teams.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bool&lt;/strong&gt;: Binary true/false classification with an attached confidence value.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Top-K&lt;/strong&gt;: Returns the highest-probability candidates from a longer option list.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nul&lt;/strong&gt;: A null option for cases where the input falls outside the model’s coverage or cannot be judged.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two critical implementation details are often overlooked.&lt;br&gt;
First, Jev runs in a stateless single-pass inference. Unlike autoregressive LLMs, it does not reuse previous token outputs as context. This makes its latency stable regardless of the number of candidate choices, which is a major advantage for multi-class routing.&lt;br&gt;
Second, the confidence value is not a separate prediction field. It is derived mathematically from the raw probability distribution of choices. The confidence formula uses the maximum probability among all candidates, so the score reflects the model’s certainty over the most likely outcome rather than a separate learned signal.&lt;/p&gt;

&lt;h3&gt;
  
  
  Jev vs LLM Structured Outputs
&lt;/h3&gt;

&lt;p&gt;Many developers confuse Jev’s capability with the structured output feature supported by mainstream LLMs. The two approaches differ fundamentally.&lt;br&gt;
General LLM structured outputs rely on the model generating text that adheres to a predefined JSON schema. The model still runs an autoregressive generation loop, and schema compliance adds extra constraints to the prompt. The model may still produce malformed JSON, and longer schemas increase latency.&lt;br&gt;
Jev is trained natively for distribution output. It does not generate text at all. It directly emits probabilities for each candidate choice. There is no schema parsing, no risk of broken JSON, and no extra cost for more options. The LLM returns a single selected answer; Jev returns the full distribution, letting downstream services apply custom thresholds to decide whether to accept the judgment, escalate to human review, or call a fallback model.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Feature&lt;/th&gt;
&lt;th&gt;LLM Structured Output&lt;/th&gt;
&lt;th&gt;Jev&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Generation mode&lt;/td&gt;
&lt;td&gt;Autoregressive text generation&lt;/td&gt;
&lt;td&gt;Single forward pass, no text generation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schema dependency&lt;/td&gt;
&lt;td&gt;Requires JSON schema definition&lt;/td&gt;
&lt;td&gt;No schema required&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failure risk&lt;/td&gt;
&lt;td&gt;Malformed output, syntax error&lt;/td&gt;
&lt;td&gt;No syntax parsing risk&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Return payload&lt;/td&gt;
&lt;td&gt;Single selected result&lt;/td&gt;
&lt;td&gt;Full probability distribution across all choices&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Latency scaling&lt;/td&gt;
&lt;td&gt;Rises with output length&lt;/td&gt;
&lt;td&gt;Stable, independent of candidate count&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  2. Where the Speed and Cost Savings Come From
&lt;/h2&gt;

&lt;p&gt;The biggest performance gain comes from shifting complexity from O(n) autoregressive sampling to O(1) single-pass inference. Standard LLMs generate each token sequentially. If a classification task requires 10 output tokens, the model runs 10 separate inference steps. Jev evaluates all candidate options in one forward pass, so inference time does not scale with the number of choices.&lt;/p&gt;

&lt;p&gt;The official pricing for Jev is listed at $0.042 per million input tokens. This rate is compared with widely used models such as GPT-4o and Astra. The benchmark data shows substantial cost advantages for different input lengths.&lt;br&gt;
For short inputs of 300 tokens, Jev delivers cost savings of roughly 27 times compared with higher-end general models. For medium inputs around 1500 tokens, the cost multiple expands to 256x. For long inputs of 3000 tokens, Jev can reach up to 444x cost reduction against the baseline. It is important to note that this multiplier applies only to pure judgment tasks. If the workflow requires explanation, prose output or complex reasoning, Jev is not the correct tool.&lt;/p&gt;

&lt;p&gt;Latency is another core advantage. Internal benchmarking shows Jev’s median latency sits around 150 ms, while the same classification task on GPT-4o Astra takes 1.89 seconds. This is a 12.6x reduction in end-to-end response time. The gap becomes more meaningful in agent pipelines, where a single workflow may invoke dozens of routing and filtering judgments. Latency saved on each decision step compounds across the whole agent chain.&lt;/p&gt;

&lt;p&gt;API gateway routing adds a small amount of fixed overhead to every model call. Teams running multiple decision and generative models can use 4sapi as an API gateway to unify endpoint management, permission control and traffic shaping for these mixed workloads.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. RLCD: Calibrated Decisions, the Core Differentiator
&lt;/h2&gt;

&lt;p&gt;RLCD stands for Reinforcement Learning for Calibrated Decisions. This is the training framework that separates Jev from ordinary classification models. Most classification models optimize for label accuracy; RLCD optimizes for statistical calibration.&lt;/p&gt;

&lt;p&gt;Calibration means that the model’s stated confidence matches the real-world hit rate. If Jev assigns 0.8 confidence to a group of predictions, roughly 80% of those predictions should be correct. Calibration is a statistical property of batches, not a guarantee for every single individual judgment. A 0.8 confidence label does not mean that this specific prediction has an 80% chance of being right. It means that across thousands of samples with this score, the correct rate converges near 80%.&lt;/p&gt;

&lt;p&gt;The calibration property can degrade under specific conditions, and official test reports document three known failure modes.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Distribution shift&lt;/strong&gt;: When test data diverges from the training corpus, calibration breaks sharply. In one test set with shifted topic distribution, Jev’s accuracy dropped from 86% to 66%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prompt injection and adversarial samples&lt;/strong&gt;: Deliberately manipulated prompts can push the model to overconfident wrong judgments.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Novel unseen categories&lt;/strong&gt;: When inputs belong to topics that never appeared in training data, Jev’s confidence estimates become unreliable.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This trait defines the correct operational pattern for Jev: teams must monitor calibration continuously, not just static accuracy. Production systems should track confidence buckets and measure the actual correct rate for each bucket. If calibration drifts, the model or prompt set needs retuning.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. The Jev Family: Official Releases and Peer Benchmarks
&lt;/h2&gt;

&lt;p&gt;The primary stable versions are Jev 1.3 and Jev 1.4. Jev 1.3 supports longer context windows, while Jev 1.4 improves image input capability and calibration stability. Public benchmark datasets include JevBench and the Artificial Analysis Decision Index v2.1.&lt;/p&gt;

&lt;p&gt;Benchmark metrics track three key dimensions: accuracy, latency, and ECE (Expected Calibration Error). Lower ECE values mean better calibration. The Decision Index ranks Jev 1.4 very competitively among dedicated decision models. It achieves 92 ms latency with an ECE of 0.054, outperforming many alternatives on the balance of speed and calibration quality.&lt;/p&gt;

&lt;p&gt;The benchmark data also clarifies a critical conclusion: Jev is not universally better than all competing models. It wins specifically on the decision-model axis. When comparing general LLMs, many generative models have stronger reasoning ability but far higher latency and cost. When comparing only specialized decision models, Jev sits in the top tier for latency and calibration, though some alternatives may lead on narrow accuracy subsets.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Suitable and Unsuitable Workloads
&lt;/h2&gt;

&lt;p&gt;Jev is built for strict judgment tasks that satisfy four requirements: fixed candidate options, no requirement for written explanation, high throughput, and tolerance for statistical calibration guarantees.&lt;/p&gt;

&lt;h3&gt;
  
  
  Ideal Scenarios
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Ticket routing: Assign support requests to billing, infrastructure, security or customer success queues.&lt;/li&gt;
&lt;li&gt;Content moderation: Binary or multi-label filtering for user-generated content.&lt;/li&gt;
&lt;li&gt;Agent state transition: Judge whether the agent should call a tool, ask follow-up questions or return a final answer.&lt;/li&gt;
&lt;li&gt;Data quality filtering: Detect malformed, irrelevant or out-of-domain records before they enter downstream pipelines.&lt;/li&gt;
&lt;li&gt;Priority triage: Score incident severity from alert text and metadata.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Poor-Fit Scenarios
&lt;/h3&gt;

&lt;p&gt;Jev should not be selected for these workloads.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Mathematical calculation and precise date arithmetic: Jev is not optimized for numerical reasoning.&lt;/li&gt;
&lt;li&gt;Deep legal or medical expert judgment: It lacks the domain grounding of specialized medical and legal models.&lt;/li&gt;
&lt;li&gt;Tasks requiring justification, explanation or free-form output: It cannot generate descriptive text.&lt;/li&gt;
&lt;li&gt;Complex multi-step reasoning chains: It is designed for single-step bounded choice, not recursive deduction.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  6. Five-Step Rollout Workflow
&lt;/h2&gt;

&lt;p&gt;The official recommended path from prototype to production has five clear stages.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Send the first test request
&lt;/h3&gt;

&lt;p&gt;Developers register an account, retrieve API keys, and send a simple classification payload. The request defines the candidate options and attaches the input text. The SDK returns the probability distribution directly. This step validates network connectivity, authentication and basic response parsing.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Select the correct judgment mode
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Use Choice when the set of options is stable and small.&lt;/li&gt;
&lt;li&gt;Use Top-K when you need to surface several plausible candidates for downstream ranking.&lt;/li&gt;
&lt;li&gt;Use Bool for binary classification with confidence tracking.&lt;/li&gt;
&lt;li&gt;Use Nul as an escape hatch for out-of-domain inputs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Step 3: Reframe prompts for decision mode
&lt;/h3&gt;

&lt;p&gt;Prompts for Jev are different from LLM prompts. Do not ask it to explain or write. The prompt should clearly state the candidate options, define the decision boundary, and specify what the null option represents. Ambiguous prompts are the leading cause of poor calibration.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Evaluate with hundreds of real production samples
&lt;/h3&gt;

&lt;p&gt;Do not rely on synthetic test data. Prepare a curated test set of 300–500 samples drawn from real traffic. Measure accuracy, latency and calibration per confidence bucket. This test set becomes the baseline for future version upgrades.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 5: Tune thresholds and roll out gradually
&lt;/h3&gt;

&lt;p&gt;Calibration is not a one-time fix. Engineers set thresholds for each confidence bucket. High-confidence results can auto-approve; mid-confidence results go to human review; low-confidence or null outputs are routed to fallback LLMs. Roll out traffic in percentage increments, continuously monitoring ECE and error rates.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Closing Thoughts
&lt;/h2&gt;

&lt;p&gt;Jev represents a meaningful separation of concerns in LLM system architecture. Generative models are powerful for drafting, reasoning and open-ended creation, but they are economically inefficient for simple bounded judgment. Jev takes over the routing, filtering and triage work that previously forced developers to burn tokens on unnecessary text generation.&lt;/p&gt;

&lt;p&gt;Its core value is not raw accuracy. Many classification models can hit similar accuracy numbers. The unique selling point is calibrated probability distribution output at ultra-low latency and cost. That said, Jev carries clear limitations. Calibration can drift, it cannot generate explanations, and it fails on tasks requiring deep reasoning. It is a specialized component, not a replacement for general LLMs.&lt;/p&gt;

&lt;p&gt;The best production pattern is a hybrid stack: use Jev for high-volume decision gates, and keep general LLMs for reasoning, generation and fallback handling. This architecture reduces overall inference spend while keeping the flexibility of generative AI for complex user requests.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>api</category>
      <category>llm</category>
    </item>
    <item>
      <title>What to Do When the Context Fills Up, and How to Remember the Last Session</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Thu, 19 Nov 2026 08:53:26 +0000</pubDate>
      <link>https://dev.to/4sapi/what-to-do-when-the-context-fills-up-and-how-to-remember-the-last-session-27m3</link>
      <guid>https://dev.to/4sapi/what-to-do-when-the-context-fills-up-and-how-to-remember-the-last-session-27m3</guid>
      <description>&lt;p&gt;Two problems were left open at the end of the previous part: the trajectory only grows and eventually will not fit, and when the program exits it remembers nothing.&lt;/p&gt;

&lt;p&gt;Both describe the same thing — memory. The intuitive assumption is that memory is a feature of the model, something you switch on. That assumption is wrong. The model's parameters are frozen after training, and every request arrives with no prior state. Everything called memory is built on top of the context window.&lt;/p&gt;

&lt;p&gt;That produces two mechanisms, and only two:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The agent remembers earlier turns in a conversation because those turns are still inside the context.&lt;/li&gt;
&lt;li&gt;The agent remembers across sessions because something was written to storage outside the program and read back into the context at startup.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This part implements both, and the toy agent reaches version 0.4.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The Window Is Finite, and It Degrades Before It Fills
&lt;/h2&gt;

&lt;p&gt;A model can attend to a bounded number of tokens per call — the context window. Depending on the model, this ranges from a few thousand to about a million. Exceed it and the request fails outright.&lt;/p&gt;

&lt;p&gt;Staying under the limit is not sufficient. As context volume grows, the model becomes more likely to drop individual items, and content positioned in the middle is the most likely to be lost. This is not a hardware constraint; it is a behavioral one, and it has a name: context rot. The human analogue is familiar — reading a hundred pages of material before answering usually produces worse answers than reading the three relevant pages.&lt;/p&gt;

&lt;p&gt;The practical conclusion: do not wait for the error. Set a budget well below the hard limit, and act as soon as the trajectory crosses it.&lt;/p&gt;

&lt;p&gt;The difference between the two approaches is visible in one picture. Left alone, content grows in a straight line until the request fails. With a budget, the line climbs to a threshold, drops back, climbs again — a sawtooth that never leaves the safe band.&lt;/p&gt;

&lt;p&gt;Three reasons to control context length, not one:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Avoiding hard failures.&lt;/li&gt;
&lt;li&gt;Preserving answer accuracy.&lt;/li&gt;
&lt;li&gt;Controlling cost.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Point three connects to an earlier calculation: the longer the context, the more expensive every turn. Prompt caching helps only when the prefix is stable, and any edit to earlier content invalidates the cached portion from that point forward. This is why compression strategy and caching strategy cannot be designed independently — a decision covered later in this piece.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Three Exits
&lt;/h2&gt;

&lt;p&gt;When the trajectory is too long, three moves are available. They differ in where the content goes, and each carries a different cost.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Exit&lt;/th&gt;
&lt;th&gt;What happens&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Trim&lt;/td&gt;
&lt;td&gt;Return less at the source; elide old tool results&lt;/td&gt;
&lt;td&gt;May cut something needed later&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compress&lt;/td&gt;
&lt;td&gt;Replace older turns with a summary&lt;/td&gt;
&lt;td&gt;One extra model call; detail loss; cache invalidation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Offload&lt;/td&gt;
&lt;td&gt;Write content to a file, leave a pointer in context&lt;/td&gt;
&lt;td&gt;Must re-read when detail is needed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  2.1 Trim: return less to begin with
&lt;/h3&gt;

&lt;p&gt;Most context bloat comes from tool results, so the cheapest fix is at the source: tools should not return so much in the first place.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A note-reading tool returns only the first two thousand characters, with a trailing note that more remains and can be read on request.&lt;/li&gt;
&lt;li&gt;A search tool returns only the matching lines, not the whole file.&lt;/li&gt;
&lt;li&gt;Old tool results already in the trajectory are replaced with a single line: content elided, re-read if needed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The cost is real: trimming may discard something needed later. So every trim must leave a path back — the agent has to know the content can be retrieved again.&lt;/p&gt;

&lt;h3&gt;
  
  
  2.2 Compress: replace old turns with a summary
&lt;/h3&gt;

&lt;p&gt;Hand the earlier portion of the trajectory to the model, ask for a short summary, then substitute the summary for the original text. After compression the trajectory shrinks substantially — and anything the summary omitted is genuinely gone.&lt;/p&gt;

&lt;p&gt;A summary must retain at least four things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The requirements the user stated.&lt;/li&gt;
&lt;li&gt;What has already been done.&lt;/li&gt;
&lt;li&gt;The conclusions reached.&lt;/li&gt;
&lt;li&gt;What remains unfinished.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Drop any one of those four and subsequent turns are likely to drift.&lt;/p&gt;

&lt;p&gt;Compression carries three costs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;An extra model call.&lt;/strong&gt; Every compression round costs a full request.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Detail loss.&lt;/strong&gt; Specifics that seemed incidental at compression time often turn out to be load-bearing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cache invalidation.&lt;/strong&gt; This is the one most often missed. Compression rewrites earlier content, so from the point of the summary onward, the cached prefix is invalid. Every token after that point must be reprocessed at full price.&lt;/p&gt;

&lt;p&gt;The third cost dictates the compression schedule. You cannot compress a little on every turn — that invalidates the cache every turn, and you pay the reprocessing cost continuously. Instead, compress a large chunk once the budget is crossed. That is exactly why the sawtooth in the earlier figure drops in large steps rather than small ones.&lt;/p&gt;

&lt;p&gt;For teams running agents across several model backends, this cost model is worth centralizing rather than rediscovering per project. Prompt cache pricing differs by provider, and the same compaction policy can be cheap on one endpoint and expensive on another. 4sapi is an API gateway that keeps multiple model endpoints behind one routing layer, which makes per-turn cost visible in one place instead of scattered across dashboards — useful when the compaction threshold itself is a cost decision.&lt;/p&gt;

&lt;h3&gt;
  
  
  2.3 Offload: move content outside
&lt;/h3&gt;

&lt;p&gt;Write the content out of the context and leave one pointer sentence behind.&lt;/p&gt;

&lt;p&gt;A concrete case: the agent reads ten notes and produces a consolidated summary. That summary does not need to stay in the trajectory. Write it to &lt;code&gt;summary.md&lt;/code&gt;, and leave one line in context — summary written to &lt;code&gt;summary.md&lt;/code&gt;. When detail is needed later, read the file.&lt;/p&gt;

&lt;p&gt;Files have almost no capacity limit, which the context window does not. This is also why long tasks work better when the agent records progress and conclusions to a file as it goes: even if the trajectory is later compressed, the file survives.&lt;/p&gt;

&lt;p&gt;The three exits are not mutually exclusive. Combined, they look like this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Exit&lt;/th&gt;
&lt;th&gt;Method&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Trim&lt;/td&gt;
&lt;td&gt;Tools return less; old results elided&lt;/td&gt;
&lt;td&gt;May cut something still needed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compress&lt;/td&gt;
&lt;td&gt;Old trajectory replaced by a summary&lt;/td&gt;
&lt;td&gt;Extra call; detail loss; cache invalidation&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Offload&lt;/td&gt;
&lt;td&gt;Content written to file, pointer remains&lt;/td&gt;
&lt;td&gt;Extra read when detail is needed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  3. The Compaction Code
&lt;/h2&gt;

&lt;p&gt;The interesting part of the implementation is how &lt;code&gt;compact&lt;/code&gt; splits the trajectory into "earlier" and "most recent" segments.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;OpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;base_uri&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"LLM_BASE_URL"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"LLM_API_KEY"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="no"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"LLM_MODEL"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="no"&gt;BUDGET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;600&lt;/span&gt;  &lt;span class="c1"&gt;# token budget for the trajectory; small here for demonstration&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;as_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;model_dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exclude_none&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;compact&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""Compress earlier turns into a summary when the trajectory exceeds budget."""&lt;/span&gt;
    &lt;span class="n"&gt;history&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;as_dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;last_user&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;recent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;last_user&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;last_user&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;history&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ensure_ascii&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="no"&gt;BUDGET&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;
    &lt;span class="n"&gt;ask&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Below is an earlier segment of a conversation. Write a short summary. "&lt;/span&gt;
           &lt;span class="s2"&gt;"Keep: what the user asked for, what has been done, conclusions reached, work not yet finished.\n\n"&lt;/span&gt;
           &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;old&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ensure_ascii&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"[compact] folded {len(old)} earlier messages into one summary"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"[summary of earlier conversation]\n"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;recent&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three implementation details carry the design.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The split point is the last &lt;code&gt;user&lt;/code&gt; message.&lt;/strong&gt; Everything before it is "earlier" and gets summarized; everything from it onward is the most recent round and stays verbatim. Recent content is the most likely to be needed immediately, so it is the last thing to touch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Summarization is a separate model call, outside the main loop.&lt;/strong&gt; It has no tools and no agent state. The instruction explicitly names the four items to retain, because an unconstrained "summarize this" reliably drops one of them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The returned list is: system prompt, summary, most recent round.&lt;/strong&gt; The system prompt stays untouched at the front, so the static prefix remains byte-identical and its cache survives. Only content after the summary is reprocessed.&lt;/p&gt;

&lt;p&gt;That last point is the whole reason the structure looks the way it does. A different ordering — summary first, system prompt after — would invalidate the entire prefix on every compression.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Cross-Session Memory: One File
&lt;/h2&gt;

&lt;p&gt;Compression solves the problem of not fitting within a single run. But the trajectory lives only inside that run; exit the program and it is gone. To make the agent remember tomorrow, something must be stored outside the process. The simplest storage is a file.&lt;/p&gt;

&lt;p&gt;The arrangement has two actions and no more:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Write.&lt;/strong&gt; When something worth keeping long-term appears in conversation, the model calls a tool to append it to a file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read.&lt;/strong&gt; At next startup, the program reads the file contents and places them into context.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  4.1 The tool and the startup read
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="no"&gt;MEMORY&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"memory.md"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# cross-session memory lives here&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;remember&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fact&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="s2"&gt;"""Append one long-term fact to memory.md."""&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="no"&gt;MEMORY&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"utf-8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"- {fact}\n"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"Noted."&lt;/span&gt;

&lt;span class="no"&gt;TOOLS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"remember"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Record a fact that will still matter in later conversations, such as "&lt;/span&gt;
                    &lt;span class="s2"&gt;"long-term preferences or stable habits. Do not record things that only "&lt;/span&gt;
                    &lt;span class="s2"&gt;"matter within this conversation."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"object"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"required"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"fact"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="s2"&gt;"properties"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"fact"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"string"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"description"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"One sentence stating what it is"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="c1"&gt;# At every startup, place memory contents after the system prompt&lt;/span&gt;
&lt;span class="n"&gt;memory_text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;MEMORY&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"utf-8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="no"&gt;MEMORY&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"(none yet)"&lt;/span&gt;
&lt;span class="no"&gt;SYSTEM&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"You are Xiaoji, my assistant for organizing study notes. Keep answers short.\n\n"&lt;/span&gt;
          &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"[Long-term memory about me]\n{memory_text}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"system"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;SYSTEM&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three design notes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What to record and what not to record is written in the tool description.&lt;/strong&gt; The model's decision to call &lt;code&gt;remember&lt;/code&gt; rests entirely on that sentence. A vague description produces either over-recording or under-recording, and both are hard to debug after the fact.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Memory is appended after the system prompt, inside the static prefix.&lt;/strong&gt; It does not change during a run, so it does not disturb the cache. Facts recorded during the current run are already in the trajectory and enter the prefix at the next startup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The read is deliberately crude: load the entire file.&lt;/strong&gt; With a handful of memory entries this is the simplest correct approach. It stops being correct as the file grows — a limitation taken up in section 6.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.2 Wiring the loop
&lt;/h3&gt;

&lt;p&gt;The loop is nearly identical to the previous part's, with one added line: before each new question arrives, decide whether compaction is needed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_steps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;compact&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;          &lt;span class="c1"&gt;# check budget before each new question&lt;/span&gt;
    &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;question&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_steps&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tools&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="no"&gt;TOOLS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;
        &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tool_calls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tool_calls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;remember&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;function&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"[step {step}] {call.function.name} {call.function.arguments}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"tool"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"tool_call_id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"Step budget exhausted before finishing."&lt;/span&gt;

&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"me: "&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"exit"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"quit"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"q"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;break&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Xiaoji: "&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Compaction is placed at the moment a new question arrives rather than mid-step. At that point the previous round is complete, so a tool call and its result cannot be split apart by the cut.&lt;/p&gt;

&lt;p&gt;Put the three code blocks in one file and the agent is at v0.4. Two runs of it follow.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Run one.&lt;/strong&gt; State a preference, then converse for several turns.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;me: From now on, format my note titles as "date-topic", e.g. 1118-context.
[step 1] remember {"fact": "user wants note titles formatted as date-topic, e.g. 1118-context"}
Xiaoji: Noted — title format is date-topic, e.g. 1118-context. I'll use it for all notes.
... (three more turns)
[compact] folded the earlier 4 messages into one summary
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Exit, restart, ask again.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;me: Give today's note about memory a title.
Xiaoji: What's today's date? Give me month/day (e.g. 5-6 → 56), and the title will be "56-memory".
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Restarted, the agent still remembers the title format. It also still does not know today's date — the same shortcoming identified in part one.&lt;/p&gt;

&lt;h3&gt;
  
  
  4.3 The memory was wrong
&lt;/h3&gt;

&lt;p&gt;Opening &lt;code&gt;memory.md&lt;/code&gt; revealed something worth dwelling on. The recorded entry was half a sentence longer than what was actually said:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"…date in four digits, without leading zeros."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That instruction was never given. The model inferred it from the example and recorded it as fact. The &lt;code&gt;56&lt;/code&gt; in the second run was produced by following that wrongly recorded memory.&lt;/p&gt;

&lt;p&gt;Memory written by the model will contain errors, and will add statements the user never made. A wrong memory does not fail once; it propagates into every later conversation that reads it. The countermeasure adopted here is simple: store memory as plain text a human can read, open the file periodically, and correct errors by hand. A machine-readable format would be more convenient for the agent and harder for a person to audit — for memory, auditability wins.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. What to Record and What Not To
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Should record&lt;/th&gt;
&lt;th&gt;Should not record&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Long-term preferences: title format is date-topic&lt;/td&gt;
&lt;td&gt;Requirements tied to one turn: answer more briefly this time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stable facts: I am learning about agents; I have finished tool calling&lt;/td&gt;
&lt;td&gt;Transient state: I am currently on the third paragraph&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Repeated corrections: do not abbreviate in English&lt;/td&gt;
&lt;td&gt;Intermediate task results — those belong in an offloaded file&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;Sensitive data such as passwords and ID numbers&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The test is one question: would this still apply tomorrow, on a different task? If yes, record it.&lt;/p&gt;

&lt;p&gt;Three problems accumulate as memory is used. None of them is handled in v0.4; they are noted here so the limitation is explicit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It grows.&lt;/strong&gt; Every startup loads the whole file, and a large file occupies a large share of the context. Periodic consolidation is required.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It goes stale.&lt;/strong&gt; A preference from six months ago may no longer hold. Entries should carry a date, and newer entries should override older ones.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It contradicts itself.&lt;/strong&gt; Append-only storage accumulates both "prefers short" and "prefers detailed". A write should first check for a nearby older entry and amend it rather than adding a second one.&lt;/p&gt;

&lt;p&gt;Memory is not hard to store. It is hard to govern.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Common Misconceptions
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Original assumption&lt;/th&gt;
&lt;th&gt;What actually happens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;The model has a memory feature&lt;/td&gt;
&lt;td&gt;The model itself does not remember. Memory lives outside and is loaded into context when needed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Staying under the limit is fine&lt;/td&gt;
&lt;td&gt;Degradation appears well before the limit; set a budget below it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compression is lossless&lt;/td&gt;
&lt;td&gt;Summaries drop detail, cost an extra call, and invalidate cache&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;More memory is better&lt;/td&gt;
&lt;td&gt;More memory consumes context, goes stale, and contradicts itself&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Memory is self-written, so it is reliable&lt;/td&gt;
&lt;td&gt;It errs and adds statements never made; it needs review and editing&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The model has no memory. All memory is context work: either the content is still in the window, or it lives outside and is loaded back when needed.&lt;/li&gt;
&lt;li&gt;Context degrades before it fills. Set a budget well below the ceiling.&lt;/li&gt;
&lt;li&gt;A long trajectory has three exits: trim, compress, offload. Compression invalidates cache, so accumulate and cut in large steps rather than a little at a time.&lt;/li&gt;
&lt;li&gt;Cross-session memory is a file plus two actions: write and read.&lt;/li&gt;
&lt;li&gt;Record only what still applies tomorrow on a different task. Memory must be readable and correctable by a person.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Version state:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Xiaoji v0.4&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Added since last version&lt;/td&gt;
&lt;td&gt;Automatic compaction when budget is exceeded; one memory file and a &lt;code&gt;remember&lt;/code&gt; tool&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Can now do&lt;/td&gt;
&lt;td&gt;Long conversations without overflowing; preferences survive a restart&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Still broken&lt;/td&gt;
&lt;td&gt;Memory and notes are loaded in full; both degrade once there is much of either&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The next version has to address memory consolidation and selective loading. For now, the honest assessment is that this agent remembers in the way a notebook remembers — accurately enough to be useful, and only as correct as the last person who checked it.&lt;/p&gt;

&lt;p&gt;International access: [&lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;](&lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;)&lt;br&gt;
Domestic access: [&lt;a href="https://4sapi.org" rel="noopener noreferrer"&gt;https://4sapi.org&lt;/a&gt;](&lt;a href="https://4sapi.org" rel="noopener noreferrer"&gt;https://4sapi.org&lt;/a&gt;)&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>api</category>
      <category>llm</category>
    </item>
    <item>
      <title>GPT-6.2 API Cuts Prices by 20%: Why Migrating Old Code Directly Can Increase Your Bill</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Wed, 18 Nov 2026 08:45:33 +0000</pubDate>
      <link>https://dev.to/4sapi/gpt-62-api-cuts-prices-by-20-why-migrating-old-code-directly-can-increase-your-bill-1h0l</link>
      <guid>https://dev.to/4sapi/gpt-62-api-cuts-prices-by-20-why-migrating-old-code-directly-can-increase-your-bill-1h0l</guid>
      <description>&lt;p&gt;On November 17, 2026, OpenAI officially released two new frontier models, GPT-6.2 Astra and GPT-6.2 Sol. On the same day, the API changelog was updated, and the public beta for the background discount window of the Responses API went live. OpenAI’s official positioning describes the two models clearly: GPT-6.2 Astra is built for the most demanding production workloads, while GPT-6.2 Sol targets high-efficiency, cost-sensitive tasks. Both models deliver stronger outputs with fewer output tokens, allowing developers to complete more work within existing usage quotas.&lt;/p&gt;

&lt;p&gt;For developers integrating OpenAI APIs or building tool flows with Codex, this release brings meaningful pricing changes paired with breaking behavioural adjustments. Many teams will notice that simply swapping the model ID in existing code does not guarantee lower costs. In some cases, unchanged legacy integration logic will trigger extra token consumption and erase the 20% price reduction benefit. This article breaks down the pricing structure, critical behaviour shifts, migration rules, and new discount options for the GPT-6.2 family.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pricing Breakdown: 20% Discount for Astra, Long Context Multipliers and Cache Rules
&lt;/h2&gt;

&lt;p&gt;The table below lists per-million-token pricing under the Standard tier for the new models, alongside their predecessor versions for direct comparison.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Input (USD / 1M tokens)&lt;/th&gt;
&lt;th&gt;Output (USD / 1M tokens)&lt;/th&gt;
&lt;th&gt;Cached Read (USD / 1M tokens)&lt;/th&gt;
&lt;th&gt;Cache Write (30 min window)&lt;/th&gt;
&lt;th&gt;Cache Write (12 hour window)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6.2-Astra&lt;/td&gt;
&lt;td&gt;8.00&lt;/td&gt;
&lt;td&gt;40.00&lt;/td&gt;
&lt;td&gt;0.40&lt;/td&gt;
&lt;td&gt;10.00&lt;/td&gt;
&lt;td&gt;20.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6 Astra (previous generation)&lt;/td&gt;
&lt;td&gt;10.00&lt;/td&gt;
&lt;td&gt;50.00&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;td&gt;12.50&lt;/td&gt;
&lt;td&gt;25.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6.2-Sol&lt;/td&gt;
&lt;td&gt;2.00&lt;/td&gt;
&lt;td&gt;10.00&lt;/td&gt;
&lt;td&gt;0.10&lt;/td&gt;
&lt;td&gt;2.50&lt;/td&gt;
&lt;td&gt;5.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-6.1-Sol (previous generation)&lt;/td&gt;
&lt;td&gt;2.00&lt;/td&gt;
&lt;td&gt;10.00&lt;/td&gt;
&lt;td&gt;0.10&lt;/td&gt;
&lt;td&gt;2.50&lt;/td&gt;
&lt;td&gt;5.00&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;As the official announcement confirms, GPT-6.2 Astra input and output token prices are 20% cheaper than the prior GPT-6 Astra release. The cached read rate sees a far deeper cut, dropping 60% from 1.00 to 0.40 USD per million tokens. GPT-6.2 Sol retains the exact token pricing of GPT-6.1 Sol, with no base price adjustment.&lt;/p&gt;

&lt;p&gt;The cache read multiplier is one of the most impactful changes for cost control. Starting from GPT-5.6, most OpenAI models set cached read pricing at 0.1 times the base input price. GPT-6.2 Astra, GPT-6.2 Sol, GPT-6.1 Sol and GPT-6.1 Terra shift this multiplier down to 0.05. For Astra, this change reduces cached read costs from 1.00 to 0.40 per million tokens. Cache write pricing follows a separate rule: writes within a 30-minute window cost 1.25 times the base input price, while writes in a 12-hour window are charged at 2.5 times the base input rate. It is important to note this is a write surcharge, not an adjustment applied to standard input tokens.&lt;/p&gt;

&lt;p&gt;Base pricing is only the starting point. Final billing is modified by long-context multipliers and service tier selection:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Long context surcharge&lt;/strong&gt;: Any request with more than 272K input tokens triggers doubled input and cached read pricing, while output costs are multiplied by 1.5 for the full request. This applies to the entire request, not just the tokens exceeding the threshold.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speed tier multipliers&lt;/strong&gt;: The Standard tier is the baseline. Fast tier delivers the same context window at double the Standard price. Batch tier costs 50% of Standard, and Ultrafast tier runs at 6 times the Standard rate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Region and residency fees&lt;/strong&gt;: The Flex pricing table has not yet been updated for GPT-6.2 models. Astra and Sol currently support Standard, Fast, Batch and Ultrafast deployments only for US and EU data residency. Using regional endpoints adds an extra 10% fee.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Behaviour Change 1: Editing Conversation Prefixes Removes Prior Reasoning Content
&lt;/h2&gt;

&lt;p&gt;This is the highest-risk change for existing multi-turn agents and tool calling workflows. Under the GPT-6.2 Responses API, any edit to the prefix of a conversation history will discard existing reasoning items. Reasoning items are internal model traces that are not returned to the developer in raw form, but they persist across turns to maintain logical continuity in agent loops.&lt;/p&gt;

&lt;p&gt;OpenAI’s migration documentation explicitly states that once a developer edits earlier messages, modifies tool call results, or alters the &lt;code&gt;developer&lt;/code&gt; instruction block mid-conversation, the API will drop stored reasoning items. The model must recompute that reasoning from scratch, which adds extra input tokens and increases latency. In the worst cases, repeated reasoning recomputation can push total token expenditure far above the savings from the 20% base price cut.&lt;/p&gt;

&lt;p&gt;Several distinct mismatch triggers are defined in the API response field &lt;code&gt;input_transformations&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;prefix_mismatch&lt;/code&gt;: This fires when content before an existing reasoning block is modified. Appending new messages after reasoning items or deleting isolated trailing entries does not trigger this rule.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;end_user_mismatch&lt;/code&gt;: The &lt;code&gt;safety_identifier&lt;/code&gt; value differs between sequential requests from the same conversation thread.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;model_mismatch&lt;/code&gt;: The request switches between model families, or downgrades within the same family. Upgrades to stronger models within the same family preserve reasoning items.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;organization_mismatch&lt;/code&gt;: The request switches workspaces or tenant accounts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Many existing agent frameworks regularly rewrite history to save tokens: truncating tool outputs, summarising long tool results, or rebuilding the &lt;code&gt;developer&lt;/code&gt; system prompt on every turn. All these patterns will trigger &lt;code&gt;prefix_mismatch&lt;/code&gt; and force re-reasoning.&lt;/p&gt;

&lt;p&gt;If you need to adjust reasoning strength mid-conversation without invalidating stored reasoning, do not rewrite the prefix. Instead, insert a &lt;code&gt;configuration_update&lt;/code&gt; message in the message sequence:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"configuration_update"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"reasoning"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"effort"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"high"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;OpenAI’s guidance is straightforward: keep the conversation prefix unchanged across turns whenever you want to retain reasoning context and preserve prompt caching eligibility. Append new instructions instead of rewriting earlier history.&lt;/p&gt;

&lt;h2&gt;
  
  
  Behaviour Change 2: Top-level Instructions Move Behind Input, and Are No Longer Cached
&lt;/h2&gt;

&lt;p&gt;The second major change directly impacts prompt caching economics. For GPT-6.2 models, top-level &lt;code&gt;instructions&lt;/code&gt; are appended after the &lt;code&gt;input&lt;/code&gt; array in the Responses API payload. These top-level instructions are &lt;strong&gt;not eligible for prompt caching&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Prior generations like GPT-5.6 allowed stable system prompts to sit at the start of the message sequence and qualify for cache hits. With GPT-6.2, &lt;code&gt;instructions&lt;/code&gt; are treated as per-request ephemeral guidance. If you place long-term system rules in the top-level &lt;code&gt;instructions&lt;/code&gt; field, every request will count those tokens as uncached input, erasing cache savings.&lt;/p&gt;

&lt;p&gt;To retain caching benefits, persistent system prompts must be placed inside a &lt;code&gt;developer&lt;/code&gt; message at the start of the &lt;code&gt;input&lt;/code&gt; array. This placement keeps the static prompt prefix intact and eligible for repeated cache hits. A simple cost calculation illustrates the gap:&lt;br&gt;
A 5,000-token permanent system prompt placed in top-level &lt;code&gt;instructions&lt;/code&gt; for GPT-6.2 Astra costs &lt;code&gt;5000 * 8 / 1,000,000 = 0.04 USD&lt;/code&gt; on every request. If the same prompt is moved to a leading &lt;code&gt;developer&lt;/code&gt; message and successfully cached, the cost drops to &lt;code&gt;5000 * 0.40 / 1,000,000 = 0.002 USD&lt;/code&gt; per cache hit. Over 10 repeated requests, the difference accumulates rapidly.&lt;/p&gt;

&lt;p&gt;Prompt caching also has a minimum threshold: at least 1024 tokens must be present to create a cache breakpoint. Breakpoints are placed at the end of qualifying messages, and top-level &lt;code&gt;instructions&lt;/code&gt; cannot form a breakpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  New Cost-Saving Option: Background Discount Window for Responses API
&lt;/h2&gt;

&lt;p&gt;Alongside the model release, OpenAI rolled out a public beta background completion window for the Responses API. This discount applies to asynchronous, multi-step agent workloads that do not require immediate low-latency replies. The discount tiers are structured around two window sizes: 2-hour and 8-hour completion windows.&lt;/p&gt;

&lt;p&gt;For Astra, the 2-hour window pricing is 5.60 USD input / 28 USD output per million tokens, while the 8-hour window drops to 4.80 USD input / 24 USD output. For Sol, the 2-hour window rate is 1.40 USD input / 7 USD output, and the 8-hour window is 1.20 USD input / 6 USD output. For comparison, the standard Batch tier for Astra is 4 USD input / 20 USD output, and Sol Batch is 1 USD input / 5 USD output.&lt;/p&gt;

&lt;p&gt;To enable the background discount, set &lt;code&gt;background: true&lt;/code&gt; and define &lt;code&gt;background_options.completion_window&lt;/code&gt; as either &lt;code&gt;2h&lt;/code&gt; or &lt;code&gt;8h&lt;/code&gt;. This mode is designed for chained tool calls, document processing, and background analysis workflows. You can reuse a &lt;code&gt;continuation_id&lt;/code&gt; across follow-up requests to keep the same discounted window and priority queue.&lt;/p&gt;

&lt;p&gt;Two hard limits apply to background discount workloads:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Cache sharing: Requests within the same 2-hour or 8-hour completion window share the Batch/background discount rate.&lt;/li&gt;
&lt;li&gt;Cache isolation: Prompt cache entries created by background requests cannot be reused by standard synchronous requests, even when the prompt content is identical.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is where API routing logic becomes valuable. Teams running a mix of synchronous interactive chat and offline background processing need to separate request flows, apply the correct discount parameters, and avoid cross-contamination of cache contexts. An API gateway can enforce routing rules to isolate low-latency foreground traffic and batched background tasks automatically; 4sapi is an API gateway that simplifies model routing and request policy enforcement for teams running mixed OpenAI workloads.&lt;/p&gt;

&lt;h2&gt;
  
  
  GPT-6.2 Migration Checklist
&lt;/h2&gt;

&lt;p&gt;Direct model ID swap is unsafe for most production systems. Use this checklist to validate changes before full rollout:&lt;br&gt;
| Check Item | Migration Action | Reason |&lt;br&gt;
|---|---|---|&lt;br&gt;
| Model ID | Replace identifier with &lt;code&gt;gpt-6.2-astra&lt;/code&gt; or &lt;code&gt;gpt-6.2-sol&lt;/code&gt; | Targets the new model variant |&lt;br&gt;
| Reasoning strength | Migrate parameter from top-level settings to &lt;code&gt;reasoning.effort&lt;/code&gt; | The Responses API rejects &lt;code&gt;reasoning: none&lt;/code&gt; |&lt;br&gt;
| Tool calling | Migrate all tool calling workflows to Responses API | Chat Completions endpoint no longer supports tool calls for these models |&lt;br&gt;
| Sampling parameters | Remove &lt;code&gt;temperature&lt;/code&gt;, &lt;code&gt;top_p&lt;/code&gt;, &lt;code&gt;logprobs&lt;/code&gt; | These parameters are not supported in Responses API for GPT-6.2 |&lt;br&gt;
| Permanent system prompt | Move long-lived rules into a leading &lt;code&gt;developer&lt;/code&gt; message inside &lt;code&gt;input&lt;/code&gt; | Top-level &lt;code&gt;instructions&lt;/code&gt; cannot be cached |&lt;br&gt;
| ZDR zero-data retention | Enable manually in the request payload | Responses API treats ZDR as false by default |&lt;br&gt;
| Prompt wording | Refine instructions for conciseness | GPT-6.2 defaults to tighter, more direct output style |&lt;/p&gt;

&lt;p&gt;OpenAI also notes two behavioural shifts for prompt design: the model will produce more concise writing by default, and it will proactively propose next steps when user instructions are ambiguous. For long autonomous tasks, design prompts to allow reasonable assumptions without waiting for constant confirmation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Benchmark Interpretation and Known Limitations
&lt;/h2&gt;

&lt;p&gt;All benchmark results cited in the official release are generated by OpenAI’s internal testing pipeline. Independent third-party evaluation reports have not yet been widely published. Key headline results from the announcement include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;GPT-6.2 Astra sets a new high score on AgentBench, reaching 52 points.&lt;/li&gt;
&lt;li&gt;On Terminal-Bench-Science, Astra outperforms prior models by 7.9 percentage points.&lt;/li&gt;
&lt;li&gt;GPT-6.2 Sol is within 0.5 points of Astra on Frontier Code 2.0 testing.&lt;/li&gt;
&lt;li&gt;Internal GPT-6.2 V2 tests show improved factual accuracy, with 2% fewer hallucinated output tokens.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two critical limitations are documented:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;File search behaviour changes: The model may skip missing files silently during agent workflows. Developers should add explicit error handling in &lt;code&gt;developer&lt;/code&gt; instructions to report missing resources instead of continuing execution.&lt;/li&gt;
&lt;li&gt;Stricter network security controls: The model blocks more vulnerability scanning and security testing workflows by default, to align with OpenAI’s safety framework.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  ChatGPT and Codex Availability
&lt;/h2&gt;

&lt;p&gt;The new models are rolling out to ChatGPT and Codex subscribers. GPT-6.2 Astra and Sol are available for Pro subscribers on web, mobile and Codex CLI. Ultrafast tier access is also being extended to Pro 200 subscribers starting with GPT-6.2 Sol. OpenAI recommends Astra for high-complexity reasoning, planning and coding tasks, and Sol for high-volume, routine generation work where cost efficiency is the priority.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final Takeaway
&lt;/h2&gt;

&lt;p&gt;The 20% price reduction for GPT-6.2 Astra is a meaningful cost improvement, but it is conditional. The biggest savings come from the 60% discount on cached reads, and that benefit depends entirely on preserving stable, unmodified prompt prefixes. Legacy code that rewrites conversation history, rebuilds system prompts on every turn, or places permanent instructions in top-level &lt;code&gt;instructions&lt;/code&gt; will lose cache eligibility and trigger repeated reasoning recomputation. The net result is a higher bill, even with the lower per-token base price.&lt;/p&gt;

&lt;p&gt;Teams should test migrations in staging, audit token consumption, separate synchronous and background workloads, and adjust prompt structure to fit the new caching rules. When you manage multiple model variants, discount tiers and endpoint policies, a centralised routing layer keeps configuration consistent across clients.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>api</category>
      <category>discuss</category>
    </item>
    <item>
      <title>Agent Memory Is Not Chat History: Anthropic’s Four-Layer Architecture and the Role of Dream</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Wed, 18 Nov 2026 08:15:09 +0000</pubDate>
      <link>https://dev.to/4sapi/agent-memory-is-not-chat-history-anthropics-four-layer-architecture-and-the-role-of-dream-3d6i</link>
      <guid>https://dev.to/4sapi/agent-memory-is-not-chat-history-anthropics-four-layer-architecture-and-the-role-of-dream-3d6i</guid>
      <description>&lt;p&gt;An agent without cross-session memory behaves like an employee on their first day, every day. Project context gets re-explained. User preferences get re-asked. Mistakes already corrected get repeated.&lt;/p&gt;

&lt;p&gt;Long-term memory attempts to extend four capabilities: project experience (architecture decisions, working commands, causes of failure, unfinished items), user preferences (communication style, long-term goals, stable constraints), experience reuse (turning one error into a rule rather than repeating the experiment), and cross-session task continuity (today’s work building on yesterday’s progress).&lt;/p&gt;

&lt;p&gt;But having memory is not automatically the same as being more reliable. A wrong preference can distort every subsequent interaction. An outdated command can keep breaking tasks. Text injected into the top layer by an untrusted source can influence every future session.&lt;/p&gt;

&lt;p&gt;Memory is therefore both a capability problem and a safety problem. What needs designing is not only where to store things, but what is worth remembering, what to trust, how to correct it, and when to load it.&lt;/p&gt;

&lt;p&gt;This article covers Anthropic’s four-layer memory architecture and the reasoning behind Dream. It is a design-inspiration piece; the concrete implementation belongs to a follow-up. All experimental figures come from the source material and have not been independently replicated here, so they should not be read as unconditional performance guarantees.&lt;/p&gt;

&lt;h2&gt;
  
  
  What “Agent Memory” Actually Covers
&lt;/h2&gt;

&lt;p&gt;Anthropic’s definition: knowledge an agent gains from user interactions and task experience, retained across sessions, and later re-readable as context.&lt;/p&gt;

&lt;p&gt;Four boundaries follow from that definition.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not equal to a change in model weights.&lt;/strong&gt; Training or fine-tuning alters parameters. Memory here is knowledge outside the model, re-readable later. Dream does not update weights while the model “sleeps”.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not equal to in-context learning.&lt;/strong&gt; A model adjusting its answer based on earlier turns in one conversation is in-context learning. Only experience that is retained and can enter a later session counts as cross-session memory.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not equal to context compaction.&lt;/strong&gt; Compaction compresses earlier turns of the current conversation so a long task can continue inside a limited window. It serves “keep this conversation going”. Long-term memory serves “the next conversation inherits the experience”.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not equal to chat log archival.&lt;/strong&gt; The raw transcript matters, and it is the evidential base of the whole system. But pushing all chat history into a new session creates three problems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The context does not fit. History grows continuously while attention and token budgets are finite.&lt;/li&gt;
&lt;li&gt;Important information cannot be found. Key rules are buried among large tool outputs and small talk.&lt;/li&gt;
&lt;li&gt;Unprocessed knowledge is unreliable. An agent’s guess, a wrong conclusion and the final correction can all coexist, leaving a future agent unable to tell which sentence to trust.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A memory system therefore has to extract knowledge from raw experience while preserving a path back to the evidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Four Layers
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;What is stored&lt;/th&gt;
&lt;th&gt;Problem it solves&lt;/th&gt;
&lt;th&gt;How it is read&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;L1 session transcripts&lt;/td&gt;
&lt;td&gt;User requests, agent replies, tool calls, tool results and conclusions&lt;/td&gt;
&lt;td&gt;Preserves raw experience and verifiable evidence&lt;/td&gt;
&lt;td&gt;Search on demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;L2 memory notes&lt;/td&gt;
&lt;td&gt;Facts, rules, plans, preferences, lessons&lt;/td&gt;
&lt;td&gt;Turns one experience into reusable knowledge&lt;/td&gt;
&lt;td&gt;Read or edit on demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;L3 memory representations&lt;/td&gt;
&lt;td&gt;Wiki, topic pages and their links&lt;/td&gt;
&lt;td&gt;Organises cross-note, cross-session knowledge to support navigation and synthesis&lt;/td&gt;
&lt;td&gt;Enter via directory, links or search&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;L4 injected context&lt;/td&gt;
&lt;td&gt;Core goals, stable rules, memory entry points&lt;/td&gt;
&lt;td&gt;Lets the agent know what it already knows without searching first&lt;/td&gt;
&lt;td&gt;Auto-loaded every session&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The whole diagram is a closed loop. Future actions generate new experience, and Dream keeps carrying new evidence back into each layer, rather than building a database once and never maintaining it.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Four Layers Are Not a Mandatory Pipeline
&lt;/h3&gt;

&lt;p&gt;This is a description of how knowledge is processed and when it is loaded — not a claim that all information must travel L1 → L2 → L3 → L4 in sequence.&lt;/p&gt;

&lt;p&gt;A raw transcript can be used directly to build a wiki, and it can influence what gets injected at the top layer. Common conclusions can be promoted; low-frequency detail can sink, leaving only a link at the top. The fourth layer is therefore better understood as a context-loading strategy than as another source of facts.&lt;/p&gt;

&lt;p&gt;That also explains why all history cannot sit in the top layer. Every injected item consumes tokens on every call. Wrong content affects every session. A long instruction file can even reduce adherence. The higher the layer, the smaller, more stable and more trustworthy its contents should be.&lt;/p&gt;

&lt;h3&gt;
  
  
  Memory Representations Are Not a Vector Database
&lt;/h3&gt;

&lt;p&gt;The source uses memory representations, with a wiki and its page links as the primary example, and does not define the third layer as embeddings or a vector database.&lt;/p&gt;

&lt;p&gt;Vector retrieval is certainly one possible implementation of another memory system, but it is not the focus of this architecture. What matters here is organising scattered notes into a navigable, linkable, synthesizable knowledge structure. A wiki and search complement each other rather than substitute for each other.&lt;/p&gt;

&lt;p&gt;The third layer is not simply storing more memory. It builds a directory, topic structure, links and hierarchy so the agent can walk to related knowledge and combine information across pages.&lt;/p&gt;

&lt;h3&gt;
  
  
  Two Index Entry Points Should Not Be Conflated
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;_index.md&lt;/code&gt; is the wiki’s knowledge directory. &lt;code&gt;MEMORY.md&lt;/code&gt; is the memory entry point that Claude Code auto-loads. The two can overlap, but they should not be treated as the same thing. What needs to be constrained conceptually is what automatically enters context, not the entire memory store. Product-specific injection mechanics and loading quotas are implementation detail for the follow-up article.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Core Criterion: Incremental Value for the Future
&lt;/h2&gt;

&lt;p&gt;Writing down a rule is a prediction that similar situations will recur. Writing down a verified fact is a prediction that it will be reusable later without re-checking.&lt;/p&gt;

&lt;p&gt;Memory is therefore fundamentally a prediction: which information will still be useful?&lt;/p&gt;

&lt;p&gt;The source uses Pokémon to illustrate the difference between obvious and non-obvious judgments. Current money and character position can usually be re-observed, so recording them repeatedly is redundant and decays quickly. Verified strategies, causes of failure, route planning and game mechanics, by contrast, disappear permanently if not written down.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Good memory is not a photocopy of environment state. It is experience with incremental value for future action.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That also means more memory is not better. As files accumulate, a future agent spends more time searching, discriminating and updating. A single memory can be evaluated with three questions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Can it be re-observed or retrieved in the future? If so, is another copy worth keeping?&lt;/li&gt;
&lt;li&gt;Can it change a future judgment or action? If not, where is the value in keeping it?&lt;/li&gt;
&lt;li&gt;Does it carry source, time and applicable scope, so a future agent knows whether to keep trusting it?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Memory can also grow around specific goals. A personal assistant can organise content around the user’s goals, preferences and long-term requests, while a legal task may require preserving exact contract clauses rather than a paraphrase. What is called “personal alignment” here is more accurately personalisation and goal fit — not alignment in the AI-safety sense.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Dream Is Needed: Memory Is Wrong and Incomplete
&lt;/h2&gt;

&lt;p&gt;An agent inside a conversation sees only a local slice of context, yet has to judge what deserves long-term retention. Two structural problems follow.&lt;/p&gt;

&lt;h3&gt;
  
  
  Memory Can Be Wrong at the Start, or Become Wrong Later
&lt;/h3&gt;

&lt;p&gt;In the Mt. Moon case, Claude recorded a PokéCenter near the west entrance of Mt. Moon as being to the east, then wrote “Mt Moon DONE”. The uncorrected memory sent the agent back the wrong way repeatedly, and it never reached Cerulean City. Dream compared the note against the original transcript, found the location error, rewrote the note, and the agent re-entered the cave and reached the next city.&lt;/p&gt;

&lt;p&gt;The point of that case is not “it summarised again”. It is that derived memory was reverse-verified against raw events. An error that is never corrected becomes prior knowledge for later sessions and keeps influencing action.&lt;/p&gt;

&lt;p&gt;The transcript is not absolute truth either. It records what the agent actually did and saw, and may contain wrong statements, ambiguous sources and malicious tool output. Verification requires evidence-based judgment, not mechanical copying.&lt;/p&gt;

&lt;h3&gt;
  
  
  A Single-Session View Makes Memory Naturally Incomplete
&lt;/h3&gt;

&lt;p&gt;One session can hardly detect cross-session patterns, and it does not know which pages or links the knowledge base is missing.&lt;/p&gt;

&lt;p&gt;When the source built a legal wiki, nine links pointed to a page that did not exist. A second Dream pass redirected them to existing synonym pages and added three new pages, including an index page.&lt;/p&gt;

&lt;p&gt;Dream therefore carries two kinds of work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Correction&lt;/strong&gt;: resolve contradictions, lower confidence in uncertain conclusions, mark outdated information.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Completion&lt;/strong&gt;: fill in detail and cross-session experience, repair missing pages, directories and links.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It can also deduplicate, clear low-value content, and promote or demote information between layers. Dream is memory maintenance, not batch summarisation.&lt;/p&gt;

&lt;p&gt;Online recording and offline Dream are not substitutes. The former captures significant events in time but scatters the agent’s attention on the main task and lacks hindsight. The latter can invest more computation across sessions but cannot guarantee that results are immediately usable. A sound system usually needs both.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Evaluate a Memory System
&lt;/h2&gt;

&lt;p&gt;Three representative results are kept here, with their boundaries stated. The table follows the source’s evaluation metrics. Different experiments are not directly comparable to each other.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Evaluation&lt;/th&gt;
&lt;th&gt;Result reported by Anthropic&lt;/th&gt;
&lt;th&gt;How to read it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;CL-Bench database tasks&lt;/td&gt;
&lt;td&gt;Claude Opus 5 improved from 0.564 without memory to 0.828&lt;/td&gt;
&lt;td&gt;When similar tasks recur, verified rules, answers and pitfalls can form a learning loop; this cannot be extrapolated as equal gains across arbitrary task types&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stainless document corpus (25 questions)&lt;/td&gt;
&lt;td&gt;Standard satisfaction rate rose from 13% to 48%, across 2 → 128 sessions, using only wiki scoring&lt;/td&gt;
&lt;td&gt;Organising progressively builds a knowledge map; this is a single corpus with 25 questions and cannot support a claim of universal linear growth&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wiki + search combined&lt;/td&gt;
&lt;td&gt;54% → 61%; search alone 53%; the combination also used about 20,000 fewer tokens per question&lt;/td&gt;
&lt;td&gt;The wiki provides navigation and pre-synthesis; search still returns to raw material. The wiki is not a substitute for search&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Counter-examples matter equally. In one Harvey LAB task, many scoring points could be retrieved directly from a single document, and the wiki produced no clear gain. In another Harvey LAB task, the wiki reached 1.77x the size of the original corpus, which is close to no abstraction at all.&lt;/p&gt;

&lt;p&gt;Memory systems should therefore not be judged by how many memory files exist. The indicators that matter are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;whether downstream tasks become more accurate and more stable;&lt;/li&gt;
&lt;li&gt;whether retrieval and context costs fall;&lt;/li&gt;
&lt;li&gt;whether memory is evidence-backed, complete and timely;&lt;/li&gt;
&lt;li&gt;whether errors, conflicts and outdated information can be detected;&lt;/li&gt;
&lt;li&gt;whether top-level injection stays lean.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What transfers is the design mechanism, not the percentage from a single experiment. Repetitive tasks, durable experience, cross-document synthesis and reliable raw evidence benefit most; one-off tasks and single-document lookups may not need a complex wiki at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  Questions That Turn Design Into Practice
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Does a small project need all four layers?&lt;/strong&gt; Not necessarily. The four layers are a role model for understanding memory, not a mandatory engineering checklist. A simple project can start with searchable transcripts plus a small number of memory notes. When notes accumulate and cross-topic navigation becomes difficult, introduce a wiki. Only information that is needed and stable enough across every session deserves the auto-injected layer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;If search or RAG already works, why keep notes and a wiki?&lt;/strong&gt; Search and RAG primarily solve “find relevant material and answer from it”. They do not necessarily solve “which experience is worth retaining long-term, and what should be trusted now”. Notes retain distilled rules and experience; the wiki pre-organises page relationships and cross-document knowledge, and can itself become a retrieval target for RAG. The sensible pattern is usually a combination: use notes and the wiki to locate and understand, use search to return to raw evidence. Do not let the wiki replace transcript search.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can an agent have long-term memory without Dream?&lt;/strong&gt; Yes. An agent that writes notes during work and searches them in a later session already has basic long-term memory. Dream addresses the next quality tier: how errors get corrected with hindsight, how cross-session gaps get filled, how duplicates and conflicts get resolved. It is not a precondition for memory, but a maintenance mechanism once memory scale and requirements rise.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Should Dream run after every session?&lt;/strong&gt; There is no uniform frequency. Every run consumes compute, and results do not always need to be immediately effective. Critical errors can be verified quickly; routine tidying can be batched by conversation volume, memory access frequency, information change rate and acceptable cost. The point is not a fixed “once per day”, but letting high-risk or outdated memory enter review faster.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can Dream make errors multiply?&lt;/strong&gt; It can. Transcripts are not absolute truth, and the model may introduce new errors while organising. Source, time and uncertainty therefore need to be retained; unresolvable cases should be marked as conflicting. Sampling review, downstream task evaluation, version baselines and rollback are all required. Dream provides an opportunity to correct, but it cannot replace verification.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What should not enter long-term memory?&lt;/strong&gt; Secrets and credentials; transient state that can be re-observed at any time; guesses without stated uncertainty; and text inside a transcript instructing the agent to ignore instructions or leak information. None of these should be stored as trustworthy fact. User preference is also not permanent authorisation — sensitive preferences need a stated purpose, an applicable scope, and an update or deletion mechanism, rather than only a plan to “remember forever”.&lt;/p&gt;

&lt;h2&gt;
  
  
  Closing: From Storing the Past to Lowering Future Cost
&lt;/h2&gt;

&lt;p&gt;Anthropic’s four-layer architecture tries to solve four problems at once: raw experience stays traceable, experience becomes reusable, knowledge becomes navigable, and the common entry point stays lean. Dream then uses hindsight to keep correcting and completing that system.&lt;/p&gt;

&lt;p&gt;Its goal is not to let the agent gain knowledge out of nowhere while “asleep”, nor to let memory grow without limit. It is to make the next session’s context more trustworthy, more complete and easier to use.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A good memory system does not store more of the past. It reduces the cost of re-understanding the past in the future.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Two practical notes for teams running this at scale. First, every layer in the architecture is a token cost, so injection budget is an engineering constraint, not a philosophical one. Second, when an agent issues model calls across several providers, memory content and prompt assembly should stay in the application layer rather than being duplicated into each client, so a provider change does not silently change what memory the agent sees. Centralising that layer behind a unified gateway is one way to keep the memory pipeline provider-agnostic; 4sapi provides consolidated model access for teams running mixed-provider agent stacks, which keeps memory assembly and request routing separately configurable.&lt;/p&gt;

&lt;p&gt;The follow-up article moves into implementation: how transcripts are exported, how the analyser extracts memory concurrently, and how the organiser completes the global pass.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;References&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Anthropic, &lt;em&gt;The components of agent memory&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;Anthropic, &lt;em&gt;Dreaming with a dynamic workflow&lt;/em&gt;
&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>api</category>
      <category>llm</category>
    </item>
    <item>
      <title>Three Agent Harness Pitfalls I Hit: Why Context Gets Slower as It Grows</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Tue, 17 Nov 2026 08:59:42 +0000</pubDate>
      <link>https://dev.to/4sapi/three-agent-harness-pitfalls-i-hit-why-context-gets-slower-as-it-grows-5jl</link>
      <guid>https://dev.to/4sapi/three-agent-harness-pitfalls-i-hit-why-context-gets-slower-as-it-grows-5jl</guid>
      <description>&lt;p&gt;While building my own agent harness, I ran into a problem that took longer to diagnose than to fix: the longer a session ran, the slower it got. My first assumption was model behavior. The actual cause was unaddressed gaps in the harness layer itself. This article documents three of those gaps with the code changes that closed them, for anyone else running LLM applications in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfall One: Context Bloat — Everything Gets Resent Every Time
&lt;/h2&gt;

&lt;p&gt;The initial design was the simplest possible one. On every model call, push the full conversation history.&lt;/p&gt;

&lt;p&gt;Short tasks tolerated this. After a few dozen rounds, the arithmetic stopped working. Context length grew without bound, latency climbed with it, and token cost roughly doubled per turn at the tail end of a session. The same prompt that responded in under a second at round five took several seconds at round forty.&lt;/p&gt;

&lt;p&gt;There are three distinct problems bundled inside that symptom, and they need three separate fixes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Layer the message history by type.&lt;/strong&gt; System prompts, user input, tool results, and assistant replies have different lifetimes and different volatility. Managing them as one undifferentiated array means every compression or truncation operation applies uniformly, which is wrong. Tool results in particular are large, disposable, and rarely needed again in full.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Summarize beyond a round threshold.&lt;/strong&gt; After N rounds, the older portion of the conversation should be compressed into a summary rather than retained verbatim. The original text of a tool call from twenty rounds ago is almost never load-bearing. Its conclusion is.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Truncate tool output by default.&lt;/strong&gt; Long text returns — file contents, search results, log dumps — should retain only the first N tokens. If the model needs more, it can ask for more. Paying for a full 50,000-token file read on every subsequent turn because it happened once is pure waste.&lt;/p&gt;

&lt;p&gt;Here is the implementation.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;trim_context&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nf"&gt;estimate_tokens&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="c1"&gt;# compress the middle, keep the head and tail&lt;/span&gt;
            &lt;span class="n"&gt;summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;summarize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                        &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"role"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"content"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;break&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The structural invariant matters more than the specific numbers. The first message — the system prompt — is never touched. The last two messages are never touched, because they carry the immediate task state. Everything between them is eligible for compression. The loop exits when the token estimate falls under the ceiling, or when there is nothing left to compress.&lt;/p&gt;

&lt;p&gt;After this change, context length stabilized and latency dropped with it. The key insight is that the ceiling is enforced by the harness rather than negotiated with the model.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Root cause&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Latency grows with round count&lt;/td&gt;
&lt;td&gt;Full history resent each call&lt;/td&gt;
&lt;td&gt;Round-threshold summarization of middle messages&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Token cost doubles late in session&lt;/td&gt;
&lt;td&gt;Unbounded context accumulation&lt;/td&gt;
&lt;td&gt;Hard token ceiling in &lt;code&gt;trim_context&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Single huge tool output dominates&lt;/td&gt;
&lt;td&gt;File reads, search dumps retained wholesale&lt;/td&gt;
&lt;td&gt;First-N-token truncation, refetch on demand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;System prompt degraded by compression&lt;/td&gt;
&lt;td&gt;Uniform treatment of all messages&lt;/td&gt;
&lt;td&gt;Type-layered management; head and tail protected&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Pitfall Two: The Tool Retry Dead Loop
&lt;/h2&gt;

&lt;p&gt;The second issue appeared during a longer agent run. The agent was stuck on a single tool call, retrying it more than ten times in sequence.&lt;/p&gt;

&lt;p&gt;The mechanism was not subtle. The tool returned an error. The harness had no retry limit. The model saw the error message, reasoned that it should try again, and issued an identical call. The tool failed identically. The cycle repeated until the session was killed manually.&lt;/p&gt;

&lt;p&gt;Two layers of protection resolved it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cap retries per tool invocation.&lt;/strong&gt; Set a maximum of three attempts. Beyond that, return an explicit failure message to the model so it switches strategy instead of repeating the same call.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Classify errors before retrying.&lt;/strong&gt; Network timeouts are retryable — the same call may succeed on the next attempt. Parameter errors are not. A malformed argument will fail identically every time it is sent. Non-retryable errors should be returned immediately with an instruction to use a different approach.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;call_tool_with_retry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;call_tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;is_error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;is_non_retryable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;ToolError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{name} failed {max_retries} times consecutively; try another approach"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The distinction between retryable and non-retryable is the part that actually saves tokens. A blind retry loop burns the full cost of the failed call plus the reasoning that produced it, multiplied by the retry count. Three attempts on a timeout is a reasonable cost. Three attempts on a schema violation is three times the price of a guaranteed failure.&lt;/p&gt;

&lt;p&gt;One additional habit is worth adopting at the same time: include the failure reason in the message handed back to the model. A bare "error" gives the model nothing to reason about. "Argument &lt;code&gt;start_date&lt;/code&gt; must be ISO format" gives it an immediate corrective action.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitfall Three: Failure Recovery — No Idea Where It Stopped
&lt;/h2&gt;

&lt;p&gt;The third problem is specific to long-running tasks. If the model crashes or the request times out, the next session starts with no knowledge of where the previous one stopped.&lt;/p&gt;

&lt;p&gt;The naive response is to add a checkpoint. The actual implementation requires three pieces, not one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Write a snapshot after each round.&lt;/strong&gt; The snapshot has to capture the current message list, the tool call record, and intermediate state. Message history alone is insufficient — if the agent had already written two files and was on the third, resuming with only the conversation transcript means re-executing the first two.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Check on startup and offer a choice.&lt;/strong&gt; If a checkpoint exists, present the user with an explicit decision: start over, or resume from the last breakpoint. Automatic resumption is the wrong default. Sometimes the right move is to discard the run and start clean, especially if the failure was caused by a bad instruction.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Refeed the checkpoint context on resume.&lt;/strong&gt; When resuming, the model needs to be told what it already completed. Without that, it will redo work, or worse, contradict a decision it made before the break.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;save_checkpoint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"messages"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"state"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
              &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"checkpoint.json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;load_checkpoint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"checkpoint.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"checkpoint.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The state object is where most implementations cut corners. Recording the message list is straightforward. Recording which tools have executed, which artifacts exist, and which subtasks are complete is the part that makes resumption actually work. A checkpoint without state is a transcript, not a recovery point.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Checkpoint component&lt;/th&gt;
&lt;th&gt;Purpose if omitted&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Message list&lt;/td&gt;
&lt;td&gt;Model loses conversational context and contradicts prior decisions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool call record&lt;/td&gt;
&lt;td&gt;Completed calls are re-executed, duplicating side effects&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Intermediate artifacts&lt;/td&gt;
&lt;td&gt;Files written before the break are orphaned or overwritten&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Subtask completion flags&lt;/td&gt;
&lt;td&gt;Agent repeats finished work and never reaches remaining tasks&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Engineering Tradeoffs: Self-Built vs Mature Frameworks
&lt;/h2&gt;

&lt;p&gt;Three harness approaches are in common use.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Fully self-built&lt;/strong&gt; — starting from the OpenAI SDK and assembling everything manually. Maximum flexibility, maximum pitfall count. This is what the preceding three sections describe.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Codex-style frameworks&lt;/strong&gt; — tool invocation, context management, and retry logic already integrated. Faster to a working agent, less control over edge behavior.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Harness-style frameworks&lt;/strong&gt; — heavier emphasis on state management and multi-agent coordination. Better for orchestration workloads, less suited to single-agent prototypes.&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;Strength&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fully self-built&lt;/td&gt;
&lt;td&gt;Every step is inspectable; behavior is fully under your control&lt;/td&gt;
&lt;td&gt;Every pitfall is yours to discover and fix&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Codex-style&lt;/td&gt;
&lt;td&gt;Retry, context, and tooling already handled&lt;/td&gt;
&lt;td&gt;Must adopt its abstractions; some requirements are unsupported&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Harness-style&lt;/td&gt;
&lt;td&gt;State management and multi-agent coordination built in&lt;/td&gt;
&lt;td&gt;Overhead for simple single-agent workloads&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The recommendation is straightforward. If the goal is learning, build the harness yourself and walk through these pitfalls — the understanding gained is durable. If the goal is a production environment, use a mature framework and avoid rebuilding the same failure modes from scratch.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the Harness Actually Abstracts
&lt;/h2&gt;

&lt;p&gt;A harness is not an assembly of the OpenAI SDK. Wiring SDK calls together produces a script, not a harness.&lt;/p&gt;

&lt;p&gt;The abstraction has four parts: context management, tool invocation, error recovery, and state persistence.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Context management&lt;/strong&gt; decides what the model sees on each call, in what order, and at what cost. It owns summarization thresholds, truncation policies, and the token ceiling.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tool invocation&lt;/strong&gt; owns schema validation, the retry ceiling, error classification, and the failure messages returned to the model.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Error recovery&lt;/strong&gt; owns checkpoint cadence, state capture, and resumption semantics.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;State persistence&lt;/strong&gt; owns the durable record: what completed, what remained, and what artifacts exist on disk.&lt;/p&gt;

&lt;p&gt;Get those four right, and the model behind them becomes a swappable component. GPT, Claude, or an open-source engine can be substituted without rewriting the workflow, because the workflow lives in the harness layer rather than in the prompt.&lt;/p&gt;

&lt;p&gt;That is the layer worth abstracting. A prompt is a one-time instruction. A harness is a machine that survives repeated execution, and repeated execution is where every failure in this article surfaced.&lt;/p&gt;

&lt;p&gt;One more consideration belongs at the same layer: the boundary to the model provider itself. Teams running several models behind one harness usually maintain a routing layer so that endpoint configuration, credentials, and per-model policy live in one place instead of being duplicated per client. 4sapi provides unified model access as an API gateway, which keeps that boundary in a single configuration surface while the harness owns context, tools, recovery, and state. Swapping the underlying model then becomes a change to one setting rather than a refactor of four subsystems.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist
&lt;/h2&gt;

&lt;p&gt;The three pitfalls reduce to a short list that is worth verifying before any agent ships.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A hard token ceiling is enforced by the harness, not hoped for in the prompt.&lt;/li&gt;
&lt;li&gt;Message history is layered by type, with the system prompt and the two most recent messages protected from compression.&lt;/li&gt;
&lt;li&gt;Tool output is truncated at ingest, with full retrieval on explicit request.&lt;/li&gt;
&lt;li&gt;Retry attempts are capped at a specific number per tool call.&lt;/li&gt;
&lt;li&gt;Errors are classified retryable and non-retryable, and non-retryable failures are returned with a corrective instruction.&lt;/li&gt;
&lt;li&gt;Checkpoints capture state, not just transcripts.&lt;/li&gt;
&lt;li&gt;Resumption is a user decision, not an automatic default.&lt;/li&gt;
&lt;li&gt;The model provider is a swappable component behind the harness boundary.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these require a sophisticated architecture. All of them require that the harness, and not the model, own the decision.&lt;/p&gt;

&lt;p&gt;International access: &lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;&lt;br&gt;
Domestic access: [&lt;a href="https://4sapi.org%5D(https://4sapi.org" rel="noopener noreferrer"&gt;https://4sapi.org](https://4sapi.org&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>devops</category>
      <category>api</category>
    </item>
    <item>
      <title>Why Agent Demos Look Impressive and Still Fail in Production</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Tue, 17 Nov 2026 08:58:11 +0000</pubDate>
      <link>https://dev.to/4sapi/why-agent-demos-look-impressive-and-still-fail-in-production-48in</link>
      <guid>https://dev.to/4sapi/why-agent-demos-look-impressive-and-still-fail-in-production-48in</guid>
      <description>&lt;p&gt;Put two numbers side by side and the gap becomes visible. On the LiveBench 2.0 general task board, Claude Fable 5.2 completes 82.9 tasks and ChatGPT-Pro 4.6 completes 75.6. In a simulated enterprise environment built by infrastructure company EON, the strongest performer — Claude Opus 4.8 — reaches a task success rate of 76.8%.&lt;/p&gt;

&lt;p&gt;Those figures are not contradictory. They measure different things. A benchmark score describes the ceiling of a model on a curated task set. A production success rate describes what survives contact with a real operating environment. Teams routinely treat the first as a forecast for the second, and that substitution is where deployment plans break.&lt;/p&gt;

&lt;p&gt;This is not a model quality problem. It is a structural problem the entire agent industry faces. Naming it before building anything is cheaper than discovering it after shipping.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Gap Is a Property of Probabilistic Systems
&lt;/h2&gt;

&lt;p&gt;Demos look good because a demo is a single run, selected for display. An agent is a probabilistic system. Each step carries some probability of failure. Ten steps compound into ten chances to fail. A hundred steps compound into a hundred.&lt;/p&gt;

&lt;p&gt;Production, by contrast, accepts a customer base of one million and query volume of a million requests, each of which must be handled. A distribution of outcomes with a one-percent failure rate becomes ten thousand failures. That is not a model defect. It is an evaluation defect. A demo reports the best single outcome. Production reports the whole distribution.&lt;/p&gt;

&lt;p&gt;Once that distinction is accepted, the question changes. It stops being "how capable is this agent" and becomes "how often does it fail, and how much does each failure cost when it does." Those are two different questions, and the second one is rarely asked.&lt;/p&gt;

&lt;p&gt;Current leaderboards are honest about what they measure: the ceiling. Almost nobody publishes what happens at the floor. Yet the floor is what determines whether a deployment survives its first month.&lt;/p&gt;

&lt;p&gt;Agents also have two cost structures running simultaneously: the compute, tokens, and latency incurred during a run, and the labor spent reviewing outputs and repairing damage. Benchmark tables report the first. Production teams pay both. An agent that scores well but requires human review on every third output is not a productivity gain. It is a workflow with an unmeasured second shift.&lt;/p&gt;

&lt;p&gt;Recognizing this is where genuine agent engineering begins. Every claim about agent capability that does not come with a failure distribution is marketing collateral, not evidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three Failure Modes Seen in Production
&lt;/h2&gt;

&lt;p&gt;TechCrunch published three recurring failure modes on November 11. Each is worth recording separately, because each requires a different fix.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Early termination.&lt;/strong&gt; An agent stops before the task is complete, and treats an intermediate state as a finished product. The reported case involved an agent that mistook a customer's waiting page for the completed deliverable and submitted it. No error was raised, because none of the intermediate steps violated a rule. Each step was locally valid. The task was simply declared finished too soon.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Silent substitution.&lt;/strong&gt; When an intermediate step fails, the agent does not report the failure. Instead it continues execution using the closest available substitute, and returns a result without any accounting of what was replaced. The final output looks normal. The contamination stays invisible. Downstream consumers inherit a result assembled from parts that were never confirmed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Context collapse on long tasks.&lt;/strong&gt; As conversations lengthen and tool calls accumulate, an agent drifts at some point and loses the thread. Output quality degrades from there onward. The failure is not a crash. It is a gradual loss of coherence that produces fluent, incorrect work.&lt;/p&gt;

&lt;p&gt;These three share a property: none of them appear on a standard benchmark. Single-turn capability, isolated tool competence, and per-step accuracy are all preserved while the composite task fails. A benchmark scoring one step at a time cannot detect a failure that only exists across steps.&lt;/p&gt;

&lt;p&gt;The more realistic the scenario, the more reliably all three occur together. A long conversation combines multiple rounds of dialogue with tool constraints that cannot all be satisfied. What looks like "poor capability" is usually a composite of three distinct defects. Production failures are rarely caused by one weak component. They are usually several components that each work in isolation and combine badly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Digital Twins Push the Company Into the Test Environment
&lt;/h2&gt;

&lt;p&gt;Plural recently raised a $13 million seed round with an eight-person team. Its product is a digital twin of the enterprise: a simulation of a company's knowledge base, tools, and interfaces, assembled so that an agent can be evaluated against something that resembles the environment it will actually operate in.&lt;/p&gt;

&lt;p&gt;This is a step past conventional agent evaluation. It no longer asks whether an agent can complete a task. It asks whether the agent can produce an output the customer accepts, inside the customer's own operational constraints. Permission boundaries, data visibility, interface behavior, and response delays all become part of the test.&lt;/p&gt;

&lt;p&gt;The premise is aggressive but defensible. No simulated environment can fully reproduce a real business. The more practical goal is narrower: reproduce the failure modes that actually occur, at a cost the customer is willing to pay. Simulating every edge case is infeasible. Simulating the failure classes that already appear in production logs is tractable.&lt;/p&gt;

&lt;p&gt;This kind of platform does not compete with existing tooling. Claude Code, Codex, Copilot, and Cursor all continue to work. The evaluation layer does not replace the agent; it tests the agent. For engineering teams, that is the relevant framing: the evaluation environment is capital infrastructure, not a feature of the product. No real production deployment should be certified against an unfixed baseline.&lt;/p&gt;

&lt;p&gt;Digital twins add one property that benchmark suites cannot offer: rollback. When production breaks, the state at the moment of failure is replayable in the simulation. The debugging question becomes "which step diverged" rather than "somehow it went wrong again."&lt;/p&gt;

&lt;h2&gt;
  
  
  Autonomous Driving Already Paid This Tuition
&lt;/h2&gt;

&lt;p&gt;Both Plural co-founders, Ilian Kadar and Edd Lovi, came out of the autonomous driving industry. Kadar previously worked at Nexar and Waymo on simulation and training for driving systems. Both describe the move into agents as a change of domain rather than a change of problem.&lt;/p&gt;

&lt;p&gt;Kadar's summary is direct. Autonomous driving looked exactly like the agent industry does now. Making a demo was easy. Delivering something repeatable and shippable was hard. Early self-driving demos ran in parking lots without a single accident. Once the same system moved onto public roads, the failure rate exceeded what any release process could absorb.&lt;/p&gt;

&lt;p&gt;The industry's response was not better demos. It was simulation: build the environment first, then evaluate the system inside it. Real-world data covered a vanishingly small share of the scenarios that mattered; simulation made the remaining cases reproducible. Autonomous driving spent roughly twenty years learning how to validate a probabilistic system before deploying it at scale. The agent industry is at the same template now.&lt;/p&gt;

&lt;p&gt;The forms of evaluation rhyme: simulated environment, composite tasks, upgrades to the model itself. The difference is that agent environments are far more diverse, and far more numerous per unit of work.&lt;/p&gt;

&lt;p&gt;What autonomous driving also demonstrated is that education by example is expensive. Very few people in that industry could explain why a particular metric mattered. Millions of kilometers accumulate into a single number, and the number no longer carries information that changes a decision. The first time anyone asks which metric is meaningful, that metric already has information content. Autonomous driving spent years and a lot of capital learning that. Agent teams can read the lesson or repeat the tuition.&lt;/p&gt;

&lt;p&gt;Choosing not to build an evaluation baseline means choosing not to verify anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Success Rate Number, On Its Own, Explains Nothing
&lt;/h2&gt;

&lt;p&gt;A figure like 76.8% should immediately raise the question of sampling error. One hundred tasks with 76 successes and ten thousand tasks with 7,600 successes are two different claims. Neither is a statement about capability without an interval.&lt;/p&gt;

&lt;p&gt;The standard method for quantifying that uncertainty is a confidence interval. Wilson's interval is the standard tool here. It produces a range instead of a point.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;wilson_interval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1.96&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
    &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
    &lt;span class="n"&gt;center&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;
    &lt;span class="n"&gt;half&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;z&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sqrt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;center&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;half&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;center&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;half&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;lo&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hi&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;wilson_interval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;76&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{lo:.3f}, {hi:.3f}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run this and the 76.8% figure comes back as roughly 0.73 to 0.82. At a 95% confidence level, the interval spans from about 73.0% to 81.9%.&lt;/p&gt;

&lt;p&gt;That interval is more useful than the point estimate. It is also what a 76.8% number conceals when reported alone.&lt;/p&gt;

&lt;p&gt;The practical implication is immediate. If a vendor reports a success rate without sample size and without an interval, the number cannot be used to predict your own deployment. A team running 100 tasks and a team running 10,000 tasks face different failure budgets. A single rate flattens that difference into a false equivalence.&lt;/p&gt;

&lt;p&gt;A useful rule follows: an interval narrows only when the success rate approaches saturation and the sample is large. Anything short of both leaves a wide band, and a wide band means the deployment decision is still open.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Business Behind the Simulated Environment
&lt;/h2&gt;

&lt;p&gt;The commercial logic is visible in Plural's pricing model. Billing is per simulation-environment invocation, and each customer's environment is trained against its own historical data. Testing with NVIDIA and Nebius indicates that these customer-specific models outperform general-purpose baselines.&lt;/p&gt;

&lt;p&gt;The reason is structural, not adversarial. A simulation environment can only be validated by the customer's own data. Once it is, the tuned model is transferable to the customer's live environment. Without that data, only a general foundation is available to transfer.&lt;/p&gt;

&lt;p&gt;Cost sensitivity is high, and it cuts in the platform's favor. Running a task in simulation costs far less than running the same task against live production systems, where a failure has real consequences. The economics of training against a twin are favorable precisely because the alternative is expensive.&lt;/p&gt;

&lt;p&gt;For customers, the model also changes commercially. What is being sold is not the strongest model available. It is a model narrowed to your own failure distribution. Once the evaluation pipeline is validated and the tuned model is in place, the agent economy's cost structure shifts: simulation replaces much of the trial-and-error that would otherwise happen in production.&lt;/p&gt;

&lt;p&gt;The path is still costly for engineers and teams running it. Building a small evaluation environment demands real effort on its own, and the training data must come from the customer's own environment. The sequence — data, evaluation, training, delivery — means the whole cost is not conversion cost but infrastructure cost. Once that infrastructure stands, conversion becomes much cheaper and more repeatable. That is the characteristic shape of simulation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Minimum Viable Agent Health Check
&lt;/h2&gt;

&lt;p&gt;Everything above reduces to three actions for a team that wants to test its own agent.&lt;/p&gt;

&lt;p&gt;First, list the twenty most-used pages or workflows in your actual business. Not the twenty cases in a benchmark suite. The twenty paths your users traverse most often.&lt;/p&gt;

&lt;p&gt;Second, label three hundred examples by hand and compute an interval, not a rate. Three hundred is the minimum sample at which an interval begins to carry information; the code above will do the arithmetic.&lt;/p&gt;

&lt;p&gt;Third, sort the failures into classes and build a repair roadmap. Failure type matters more than frequency. A premature stop and a silent substitution require different fixes even when their rates are identical. Labeling the failure class is the step where remediation actually starts.&lt;/p&gt;

&lt;p&gt;Here is the comparison of the three evaluation tiers.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Evaluation tier&lt;/th&gt;
&lt;th&gt;What it simulates&lt;/th&gt;
&lt;th&gt;What it can detect&lt;/th&gt;
&lt;th&gt;What it cannot detect&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Public leaderboard&lt;/td&gt;
&lt;td&gt;A general-purpose task set&lt;/td&gt;
&lt;td&gt;Ceiling of general capability&lt;/td&gt;
&lt;td&gt;Anything specific to your environment&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Simulated enterprise&lt;/td&gt;
&lt;td&gt;A standard, generic company&lt;/td&gt;
&lt;td&gt;Approximate performance on composite tasks&lt;/td&gt;
&lt;td&gt;Real user behavior, real data distributions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Digital twin&lt;/td&gt;
&lt;td&gt;Your own historical data&lt;/td&gt;
&lt;td&gt;Fidelity of the real operating environment&lt;/td&gt;
&lt;td&gt;Delivery quality, model generalization&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The three tiers are sequential, not alternatives. A leaderboard tells you which model to start with. A simulated enterprise environment tells you roughly how it composes. A digital twin tells you whether it survives your own business. Each depends on the one before it.&lt;/p&gt;

&lt;p&gt;A short-time, low-intensity check is worth doing before the final decision. Once a baseline harness exists, a three-to-five day iteration is enough to establish the first reliable baseline. Without it, three years of accumulated production experience still leaves no answer to "which step failed first." Build the baseline first, then improve the product. That sequence is worth repeating.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to Do About Failure Distributions
&lt;/h2&gt;

&lt;p&gt;The gap between a demo and a production run is not a phase the industry passes through. It is the shape of probabilistic systems operating under repetition. Two consequences follow from accepting that.&lt;/p&gt;

&lt;p&gt;The first is that evaluation shifts from capability to reliability. Capability is a ceiling, measurable once. Reliability is a distribution, measurable only over volume. Teams that instrument their own failure classes accumulate an asset that cannot be copied from a leaderboard.&lt;/p&gt;

&lt;p&gt;The second is that model selection becomes a per-workload decision rather than a single commitment. A 76.8% rate with a [73, 82] interval precedes the question of whether that rate is acceptable for a given workflow. An internal research assistant tolerates a 20% failure rate if every output is reviewed. An action-taking agent that writes to production systems does not. The same number supports opposite decisions depending on where it lands.&lt;/p&gt;

&lt;p&gt;Routing different workloads to different model endpoints is one way to hold that distinction operationally rather than by convention. 4sapi provides centralized model access, so an application can send high-volume, low-consequence tasks to a cheaper model and gate irreversible actions behind a more reliable one, without maintaining a separate integration per provider. The configuration lives in one place rather than scattered across client code.&lt;/p&gt;

&lt;p&gt;Start with twenty workflows, three hundred labels, one interval, and a failure taxonomy. That is a smaller commitment than a full deployment plan, and it produces the only number that survives contact with production.&lt;/p&gt;

&lt;p&gt;International access: &lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;&lt;br&gt;
Domestic access: &lt;a href="https://4sapi.org" rel="noopener noreferrer"&gt;https://4sapi.org&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>api</category>
    </item>
    <item>
      <title>Reflections on the Relationship Between Agent Application Development and Rendering</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Mon, 16 Nov 2026 08:44:02 +0000</pubDate>
      <link>https://dev.to/4sapi/reflections-on-the-relationship-between-agent-application-development-and-rendering-1d7c</link>
      <guid>https://dev.to/4sapi/reflections-on-the-relationship-between-agent-application-development-and-rendering-1d7c</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;When developers transition from traditional frontend and backend development to building AI Agent applications, they frequently encounter confusion around how generated data pairs with custom UI rendering. This article dissects the core mental model for Agent development: leveraging Large Language Models (LLMs) to drive engineering workflows, and clarifies how LLM capabilities, client-server communication protocols, data generation and UI rendering connect with one another.&lt;/p&gt;

&lt;p&gt;The core idea of Agent application development lies in using LLMs to drive engineering processes. The critical challenge becomes building a working rendering pipeline controlled by LLM outputs. This article breaks the problem down into three major sections: understanding LLM capabilities, defining client-server protocol boundaries, and implementing data generation alongside rendering logic. It includes practical code examples and standard patterns that developers can directly adopt in production projects.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Understanding LLM Capabilities and Core Functions
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Comprehension, Reasoning and Generation
&lt;/h3&gt;

&lt;p&gt;At its most basic level, an LLM is capable of three core operations: understanding input content, performing reasoning, and generating natural language text. A simple API call demonstrates this behavior.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;invoke&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Hello"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model returns a natural language reply. Under the hood, this execution includes three sequential steps:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Comprehend the semantics of the user input message.&lt;/li&gt;
&lt;li&gt;Execute reasoning to determine an appropriate response.&lt;/li&gt;
&lt;li&gt;Generate continuous text output as the final reply.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This fundamental capability enables LLMs to handle conversational interaction, summarization and content creation. For Agent systems, the more important extension is tool invocation via Function Call.&lt;/p&gt;

&lt;h3&gt;
  
  
  Function Call Mechanism
&lt;/h3&gt;

&lt;p&gt;Function Call is not direct function execution inside the LLM. The LLM is essentially a text completion engine. Through training, the model learns to produce structured JSON text to express intent for invoking external tools.&lt;/p&gt;

&lt;p&gt;Before Function Call was widely supported, developers relied heavily on prompt engineering. Engineers would craft detailed instructions to force the model to output content in fixed formats, then parse results with regular expressions. This method was fragile. Minor deviations in model output could break parameter parsing entirely. Some early constraint frameworks such as Guidance were used to mitigate this instability.&lt;/p&gt;

&lt;p&gt;Modern LLMs, including DeepSeek and many other mainstream models, natively support Function Call. This capability expands LLM boundaries from plain text processing to structured tool orchestration. The model can describe tools, define parameters and trigger external logic. This is the foundation that makes practical AI Agent workflows possible.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Client and Server Protocol Design
&lt;/h2&gt;

&lt;h3&gt;
  
  
  SSE (Server-Sent Events)
&lt;/h3&gt;

&lt;p&gt;Server-Sent Events, standardized by W3C, is a one-way real-time push protocol built on HTTP. It uses the &lt;code&gt;text/event-stream&lt;/code&gt; content type for long-lived connections. Servers continuously push data chunks to clients over this persistent channel. Common use cases include real-time notifications, progress streaming and streamed AI model output.&lt;/p&gt;

&lt;p&gt;SSE defines a fixed message format for event streaming.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;event: progress
id: 12
retry: 3000
data: {"percent":60}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;event&lt;/code&gt;: Identifies the type of event message.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;id&lt;/code&gt;: Event identifier for resuming interrupted streams.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;retry&lt;/code&gt;: Time interval for reconnection attempts in milliseconds.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;data&lt;/code&gt;: The payload body of the event.
Blank lines separate individual event entries.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Clients must include the &lt;code&gt;Accept: text/event-stream&lt;/code&gt; HTTP header to negotiate this stream mode. Developers can also design custom binary or JSON streaming formats instead of using the standard SSE specification.&lt;/p&gt;

&lt;p&gt;SSE fits Agent applications naturally. Agent workflows run on the server, and LLM inference consumes significant time. Streaming incremental output to the client delivers a responsive user experience.&lt;/p&gt;

&lt;p&gt;The native browser &lt;code&gt;EventSource&lt;/code&gt; API has notable limitations.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Native &lt;code&gt;EventSource&lt;/code&gt; only supports GET requests. It cannot attach request bodies, so developers cannot send message payloads to the server.&lt;/li&gt;
&lt;li&gt;It strictly enforces one-way communication. The standard usage pattern where users submit messages and receive continuous downstream output cannot be implemented directly with native EventSource.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The standard workaround uses POST requests with the &lt;code&gt;Accept: text/event-stream&lt;/code&gt; header. POST requests can carry JSON payloads, files and image content. The server detects the header and responds with streamed SSE content.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/api/chat"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"POST"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"Accept"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"text/event-stream"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Hello"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;// Read bytes incrementally from response.body&lt;/span&gt;
&lt;span class="c1"&gt;// Decode UTF-8, parse SSE events and feed to UI rendering&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Microsoft’s &lt;code&gt;@microsoft/fetch-event-source&lt;/code&gt; open-source library solves these limitations. It uses fetch under the hood and supports POST-based SSE streaming. The following example shows a reusable message sending function with custom rendering callbacks.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;fetchEventSource&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"@microsoft/fetch-event-source"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;sendMessage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;render&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;controller&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;AbortController&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;answer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;completed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetchEventSource&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/api/chat"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"POST"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="s2"&gt;"Content-Type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"application/json"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;openWhenHidden&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="nf"&gt;onopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;contentType&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"content-type"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ok&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;contentType&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"text/event-stream"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`HTTP ${response.status}: stream initialization failed`&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nf"&gt;onmessage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"delta"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;answer&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="nf"&gt;render&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;answer&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"done"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;completed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;controller&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nf"&gt;onclose&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;completed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nb"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Stream terminated prematurely"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nf"&gt;onerror&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;sendMessage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Analyze this report"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;querySelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"#answer"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;textContent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The core idea of SSE can be extended to custom streaming protocols. Instead of standard SSE, developers can implement NDJSON streams. Each line contains one serialized JSON object.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"progress"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"percent"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"completed"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nl"&gt;"result"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s2"&gt;"Analysis finished"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This custom line-delimited JSON approach works well for Agent applications that need richer structured event types beyond simple text chunks.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Data Generation and UI Rendering
&lt;/h2&gt;

&lt;p&gt;For LLMs and backend services to drive frontend components, the system must describe available UI components. Metadata defines component use cases, accepted parameters, supported events and user interaction callbacks.&lt;/p&gt;

&lt;p&gt;A simplified React approval component demonstrates this pattern.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;ApprovalProps&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;requestId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;impact&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="n"&gt;onResolve&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;decision&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"approved"&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="s2"&gt;"rejected"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;void&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nc"&gt;Approval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;impact&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;onResolve&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;ApprovalProps&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;h3&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;h3&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;impact&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;button&lt;/span&gt; &lt;span class="n"&gt;onClick&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;onResolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"approved"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="nc"&gt;Approve&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;button&lt;/span&gt; &lt;span class="n"&gt;onClick&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;onResolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"rejected"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="nc"&gt;Reject&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;button&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;section&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The TypeScript interface defines data structure and callback signatures. However, type definitions alone are insufficient. The LLM and backend cannot understand the scenario this component applies to or what events to trigger. Developers must attach component metadata to describe purpose, parameters and event mapping.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;approvalMeta&lt;/code&gt; object below provides machine-readable metadata for this UI component.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;approvalMeta&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Approval"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Display an operation and its impact, request user approval or rejection."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;usewhen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"The workflow requires user confirmation before continuing."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;onResolve&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"approval.resolve"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;targetArgs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"requestId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;payloadFromArgs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"decision"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This metadata explains that calling &lt;code&gt;onResolve(decision)&lt;/code&gt; emits an &lt;code&gt;approval.resolve&lt;/code&gt; event. It carries the &lt;code&gt;requestId&lt;/code&gt; identifier and decision payload.&lt;/p&gt;

&lt;p&gt;Developers maintain component types, descriptions and event mappings. Build tools combine TypeScript type information and metadata definitions to generate standardized machine-readable component specifications. The LLM reads these specifications and decides when to render which UI component inside the Agent workflow.&lt;/p&gt;

&lt;p&gt;This separation creates a clean boundary:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Backend / LLM: Generate structured data and event instructions.&lt;/li&gt;
&lt;li&gt;Client: Receive structured payloads, render matching UI components, capture user interactions and send events back to server.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The LLM does not directly manipulate DOM elements. It only outputs declarative instructions for UI rendering. The client interprets these instructions and manages native component rendering. This design decouples Agent logic from frontend implementation details.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. System Architecture and Agent Workflow Summary
&lt;/h2&gt;

&lt;p&gt;The full Agent rendering workflow can be summarized in a sequential pipeline:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;User sends a request from frontend client via POST streaming request.&lt;/li&gt;
&lt;li&gt;Server receives input, invokes LLM reasoning.&lt;/li&gt;
&lt;li&gt;LLM decides whether to generate plain text or emit component rendering instructions through Function Call.&lt;/li&gt;
&lt;li&gt;Server pushes structured events to client over SSE or custom stream protocol.&lt;/li&gt;
&lt;li&gt;Client parses incoming events. If the payload describes a UI component, the client instantiates and renders the matching component.&lt;/li&gt;
&lt;li&gt;User interacts with rendered UI, triggering component callbacks.&lt;/li&gt;
&lt;li&gt;Client sends interaction events back to server.&lt;/li&gt;
&lt;li&gt;Server resumes Agent reasoning using the user interaction result.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This architecture keeps rendering logic fully contained on the frontend. The LLM remains responsible only for high-level workflow decisions and data generation. This separation avoids tight coupling between model outputs and UI frameworks.&lt;/p&gt;

&lt;p&gt;In distributed Agent systems, developers often manage multi-model traffic routing through an API gateway. 4sapi provides unified endpoint management that simplifies orchestration when multiple LLMs serve Agent workloads.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Common Pitfalls and Engineering Best Practices
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Protocol Selection Tradeoffs
&lt;/h3&gt;

&lt;p&gt;SSE is ideal for server-to-client streaming, but it cannot handle bidirectional real-time communication natively. If an Agent needs frequent bidirectional message exchange, WebSocket may be more suitable. For most text and UI streaming scenarios, POST-based SSE is simpler to implement and easier to debug.&lt;/p&gt;

&lt;h3&gt;
  
  
  Component Metadata Maintenance
&lt;/h3&gt;

&lt;p&gt;Metadata must stay lightweight and machine parseable. Avoid embedding long natural language descriptions. Every component should explicitly define trigger conditions, input schemas and output event formats. Outdated metadata causes the LLM to select incorrect components, breaking rendering behavior.&lt;/p&gt;

&lt;h3&gt;
  
  
  Separation of Concerns
&lt;/h3&gt;

&lt;p&gt;Do not let LLMs directly output raw HTML or JSX. Direct code generation creates severe security risks and tight coupling. The preferred pattern is declarative rendering instructions. The LLM outputs component identifiers and parameters. The client renders pre-built registered components.&lt;/p&gt;

&lt;h3&gt;
  
  
  Stream Error Handling
&lt;/h3&gt;

&lt;p&gt;Streaming connections can drop unexpectedly. Client code must implement reconnection logic, abort controllers and partial state recovery. Each event should carry an identifier so clients can resume interrupted streams without losing state.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Building Agent applications requires rethinking the traditional frontend-backend data flow. The core principle is a clear separation: LLMs handle reasoning and structured data generation, server protocols deliver streamed payloads, and the client consumes instructions to render predefined UI components.&lt;/p&gt;

&lt;p&gt;The Function Call capability enables LLMs to declare intent for UI components. SSE and custom streaming protocols reliably transmit incremental data from server to browser. Component metadata bridges the gap between LLM reasoning and frontend rendering. This mental model helps developers avoid common mistakes such as trying to make LLMs directly generate UI markup.&lt;/p&gt;

&lt;p&gt;This pattern supports scalable Agent systems. New UI components can be added independently, and the LLM can use them once their metadata is registered. When running multi-model Agent deployments, unified gateway services streamline endpoint management and request routing.&lt;/p&gt;

&lt;p&gt;International access: &lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;&lt;br&gt;
Domestic access: &lt;a href="https://4sapi.cn" rel="noopener noreferrer"&gt;https://4sapi.cn&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>devops</category>
      <category>api</category>
    </item>
    <item>
      <title>Deep Dive into Claude Code’s Three Configuration Systems: settings.json / CLAUDE.md / memory</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Mon, 16 Nov 2026 08:26:41 +0000</pubDate>
      <link>https://dev.to/4sapi/deep-dive-into-claude-codes-three-configuration-systems-settingsjson-claudemd-memory-1lo</link>
      <guid>https://dev.to/4sapi/deep-dive-into-claude-codes-three-configuration-systems-settingsjson-claudemd-memory-1lo</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;Claude Code provides three separate configuration mechanisms, each with distinct scopes, loading logic and maintenance patterns. Many developers treat them interchangeably, which often leads to subtle and costly mistakes. Writing rules into &lt;code&gt;settings.json&lt;/code&gt; will produce no effect. Putting runtime parameters into &lt;code&gt;CLAUDE.md&lt;/code&gt; bloats the conversation context. Storing static facts inside memory may trigger incorrect inference. This article clarifies the core positioning of each component, explains loading sequence, practical syntax, common pitfalls, and how to combine them properly for production-grade agent workflows.&lt;/p&gt;

&lt;p&gt;The three systems operate at different layers and do not replace one another.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;settings.json&lt;/code&gt;: Runtime environment configuration, read by the program.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;CLAUDE.md&lt;/code&gt;: Human-authored project specification, read by the AI.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;memory&lt;/code&gt;: Cross-session knowledge storage, written and indexed automatically by the AI.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The most frequently misunderstood trait is the load-once semantic of &lt;code&gt;settings.json&lt;/code&gt;. Any modification applied after startup will not take effect until a full restart. Editing and reloading the chat window alone cannot activate changes inside this file.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. settings.json – Program Runtime Configuration
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1.1 Core Scope
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;settings.json&lt;/code&gt; defines environment variables, permissions, tool hooks and UI display parameters for Claude Code. It is consumed by the application runtime rather than interpreted by the LLM. The model itself cannot directly read the file content; the parsed configuration injects into the program environment before the agent session starts.&lt;/p&gt;

&lt;h3&gt;
  
  
  1.2 File Override Hierarchy
&lt;/h3&gt;

&lt;p&gt;Configuration cascades across global, project and local instance levels. Later entries override earlier ones.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Global &lt;code&gt;~/.claude/settings.json&lt;/code&gt;: Shared across all workspaces. Stores token endpoints, proxy settings, language preference and model selection.&lt;/li&gt;
&lt;li&gt;Project-level &lt;code&gt;.claude/settings.json&lt;/code&gt;: Committed into Git. Defines team-shared hooks, tool permissions and repository-wide command allowlist.&lt;/li&gt;
&lt;li&gt;Project &lt;code&gt;.claude/settings.local.json&lt;/code&gt;: Personal local override. Excluded from version control, for private environment variables and custom shell commands.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  1.3 Main Configuration Sections
&lt;/h3&gt;

&lt;h4&gt;
  
  
  env: Environment Injection
&lt;/h4&gt;

&lt;p&gt;This block injects environment variables into the shell context used by Claude Code. Values here override UI input selections, which explains why modifying UI proxy settings may fail after restart. Developers can verify injected variables via &lt;code&gt;env | grep -i anthropic&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"env"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"ANTHROPIC_BASE_URL"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"[https://xxx.xxx/anthropic](https://xxx.xxx/anthropic)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"ANTHROPIC_API_KEY"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"xxxx"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"API_TIMEOUT_MS"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"60000"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"API_PROXY"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"[http://127.0.0.1:7890](http://127.0.0.1:7890)"&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h4&gt;
  
  
  permissions: Command Allow / Deny Rules
&lt;/h4&gt;

&lt;p&gt;Permission rules control shell execution behavior. The &lt;code&gt;allow&lt;/code&gt; array contains approved commands that run without repeated confirmation, improving workflow speed. The &lt;code&gt;deny&lt;/code&gt; block has higher priority and blocks risky operations unconditionally.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"permissions"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;"allow"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="s2"&gt;"bash -c ls"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s2"&gt;"git diff"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s2"&gt;"cat *"&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nl"&gt;"deny"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="s2"&gt;"rm -rf"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="s2"&gt;"curl | bash"&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h4&gt;
  
  
  hooks: Tool Lifecycle Triggers
&lt;/h4&gt;

&lt;p&gt;Hooks execute custom scripts before or after tool invocations. Typical use cases include automatic code formatting, lint checks and status reporting after file writes.&lt;/p&gt;

&lt;h4&gt;
  
  
  statusLine: Custom Status Rendering
&lt;/h4&gt;

&lt;p&gt;This section customizes the bottom status bar. It can display active model name, Git branch, context token consumption percentage and other runtime metrics.&lt;/p&gt;

&lt;h3&gt;
  
  
  1.4 Key Features &amp;amp; Common Bugs
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One-time loading&lt;/strong&gt;: The file parses only at startup. Live edits do not apply in an ongoing session. A full restart of Claude Code is required.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Silent failure on malformed JSON&lt;/strong&gt;: Syntax errors will make the entire config ignored without explicit warning. Always validate JSON before saving.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Version control practice&lt;/strong&gt;: Global and project settings files are tracked by Git, while &lt;code&gt;settings.local.json&lt;/code&gt; stays local and uncommitted.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  2. CLAUDE.md – Project Specification for AI
&lt;/h2&gt;

&lt;h3&gt;
  
  
  2.1 Scope
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;CLAUDE.md&lt;/code&gt; is human-written instruction text loaded fully into the LLM context at the beginning of every conversation. It defines coding conventions, architectural constraints, test requirements and repository rules that the agent must obey. The AI reads and follows these rules, while the program runtime does not parse this file.&lt;/p&gt;

&lt;p&gt;A recommended maintenance pattern is splitting large monolithic &lt;code&gt;CLAUDE.md&lt;/code&gt; into rule files under &lt;code&gt;./claude/rules/&lt;/code&gt;. Group specifications by topic: logging standards, tech stack constraints, database rules and API contract rules. Modular files simplify incremental updates instead of maintaining a thousands-line single document.&lt;/p&gt;

&lt;h3&gt;
  
  
  2.2 What to Write
&lt;/h3&gt;

&lt;p&gt;Add enforceable rules that the AI cannot infer merely by reading source code. The test standard: if the AI may produce working code violating team agreements, write the constraint here.&lt;br&gt;
Examples of enforceable constraints:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Layered architecture restrictions: facade, service, dao, mapper layering and cross-layer prohibition.&lt;/li&gt;
&lt;li&gt;Mandatory code style, import ordering and logging field schema.&lt;/li&gt;
&lt;li&gt;Test requirements: every change must supply unit tests.&lt;/li&gt;
&lt;li&gt;Change workflow: OpenAPI spec changes must update corresponding implementation and test files.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  2.3 What Not to Write
&lt;/h3&gt;

&lt;p&gt;Avoid redundant information already visible in codebase. Do not record temporary local environment details such as personal proxy addresses, local ports or one-off setup notes. Those environment settings belong inside &lt;code&gt;settings.json&lt;/code&gt;, not &lt;code&gt;CLAUDE.md&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  2.4 Critical Performance Constraint
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;CLAUDE.md&lt;/code&gt; loads in full for every new chat session. Longer documents consume more context window capacity and reduce available space for task content. Rule documents should remain high-density. Focus on hard constraints rather than descriptive background explanation or optional suggestions. Concise rules carry stronger weight than verbose prose.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. memory – Cross-Conversational Memory
&lt;/h2&gt;

&lt;h3&gt;
  
  
  3.1 Scope
&lt;/h3&gt;

&lt;p&gt;Memory stores observations recorded by the AI automatically. It solves repeated context reintroduction across separate chat sessions.&lt;br&gt;
The difference compared with &lt;code&gt;CLAUDE.md&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;CLAUDE.md&lt;/code&gt;: Static human-authored rules that never change during conversation.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;memory&lt;/code&gt;: AI-written factual observations, learned from task execution history.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3.2 Index and Body Two-Tier Architecture
&lt;/h3&gt;

&lt;p&gt;Memory adopts an index + body design. The index (usually around 4KB) loads fully each session. The full body (often up to 100KB) loads selectively only when relevant hooks trigger. This design reduces token overhead versus loading all records unconditionally like &lt;code&gt;CLAUDE.md&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Each memory entry includes metadata, conclusion, reasoning trace and relevance hooks. Hooks act as conditional triggers to decide whether to pull the full memory body into context. Vague hook descriptions cause false negatives, preventing relevant memories from loading.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.4 Four Memory Entry Types
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;userhook&lt;/code&gt;: User preference, workflow habit and confirmed operating steps. Must include reasoning explaining why this rule applies.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;project&lt;/code&gt;: Project-specific constraints, code patterns and fixed repository conventions.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;feedback&lt;/code&gt;: Past failure summaries and fixes learned from prior task attempts.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;reference&lt;/code&gt;: External documents, URLs, ticket IDs and specification links.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  3.5 Important Traits
&lt;/h3&gt;

&lt;p&gt;Memory records a snapshot at write time. It is not real-time state tracking. Facts stored in memory may become outdated after code refactoring or configuration changes. The agent treats memory entries as historical observations instead of live file inspection results.&lt;/p&gt;

&lt;h3&gt;
  
  
  3.6 What Should Not Be Saved
&lt;/h3&gt;

&lt;p&gt;Do not duplicate information already present in source code, Git history or &lt;code&gt;CLAUDE.md&lt;/code&gt;. Memory should capture lessons that would otherwise require repeated human explanation. A simple filtering test: will this fact save repeated questions three months later? If not, skip recording it.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Comparative Summary &amp;amp; Decision Tree
&lt;/h2&gt;

&lt;h3&gt;
  
  
  4.1 Core Comparison Table
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Item&lt;/th&gt;
&lt;th&gt;settings.json&lt;/th&gt;
&lt;th&gt;CLAUDE.md&lt;/th&gt;
&lt;th&gt;memory&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Reader&lt;/td&gt;
&lt;td&gt;Program runtime&lt;/td&gt;
&lt;td&gt;AI LLM&lt;/td&gt;
&lt;td&gt;AI LLM&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Author&lt;/td&gt;
&lt;td&gt;Human&lt;/td&gt;
&lt;td&gt;Human&lt;/td&gt;
&lt;td&gt;AI (human can request deletion)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Load timing&lt;/td&gt;
&lt;td&gt;Load once on startup; restart required to apply edits&lt;/td&gt;
&lt;td&gt;Full load every new conversation&lt;/td&gt;
&lt;td&gt;Index always loaded; body lazy loaded by hook match&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Version control&lt;/td&gt;
&lt;td&gt;Project file committed; local override ignored&lt;/td&gt;
&lt;td&gt;Tracked in Git&lt;/td&gt;
&lt;td&gt;Not tracked&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Content type&lt;/td&gt;
&lt;td&gt;Environment variables, permissions, hooks&lt;/td&gt;
&lt;td&gt;Project coding rules and architecture specs&lt;/td&gt;
&lt;td&gt;Historical observations and lessons&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  4.2 Decision Logic for Placement
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Environment variables, shell allowlist and runtime parameters → &lt;code&gt;settings.json&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Permanent team coding standards and architecture rules → &lt;code&gt;CLAUDE.md&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Lessons learned, project observations and repeated pitfalls → &lt;code&gt;memory&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  4.3 Boundary Examples
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Proxy configuration: belongs in &lt;code&gt;settings.json&lt;/code&gt; env section. Placing proxy info in &lt;code&gt;CLAUDE.md&lt;/code&gt; wastes context tokens.&lt;/li&gt;
&lt;li&gt;Database connection rule: permanent architecture constraint, write to &lt;code&gt;CLAUDE.md&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Past failure: "This repository cannot use raw MySQL client without connection pooling" → store as memory entry.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  5. Practical Implementation Guidance
&lt;/h2&gt;

&lt;h3&gt;
  
  
  5.1 Best Practices for settings.json
&lt;/h3&gt;

&lt;p&gt;Separate global shared configuration and project-specific rules. Use &lt;code&gt;settings.local.json&lt;/code&gt; exclusively for private local overrides.&lt;br&gt;
Validation workflow after edits:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Create backup before modification.&lt;/li&gt;
&lt;li&gt;Validate JSON syntax.&lt;/li&gt;
&lt;li&gt;Restart Claude Code fully.&lt;/li&gt;
&lt;li&gt;Print environment variables to verify injection.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Permission design principle: deny rules take priority. Start from minimal allowlist and expand command scope gradually. Audit permission logs after execution.&lt;/p&gt;

&lt;h3&gt;
  
  
  5.2 CLAUDE.md Maintenance
&lt;/h3&gt;

&lt;p&gt;Split rule sets into multiple topic files rather than a single giant markdown document. Each rule must define consequences of violation. Remove obsolete rules periodically, as outdated rules increase context cost and confuse the agent. Keep mandatory constraints inside this document.&lt;/p&gt;

&lt;h3&gt;
  
  
  5.3 Memory Maintenance
&lt;/h3&gt;

&lt;p&gt;Memory should be actively curated rather than left to fully automatic recording. Developers can explicitly request memory write or deletion. Write precise hook conditions for selective loading. Periodically inspect indexes, merge duplicate records and archive expired observations.&lt;/p&gt;

&lt;h3&gt;
  
  
  5.4 Combined Workflow
&lt;/h3&gt;

&lt;p&gt;The standard production workflow: define permanent rules in &lt;code&gt;CLAUDE.md&lt;/code&gt;, configure runtime environment via &lt;code&gt;settings.json&lt;/code&gt;, and let memory accumulate task lessons. Periodically review and clean memory records. In agent development workflows, developers may route LLM traffic through 4sapi, an API gateway to manage multi-model endpoints and request routing.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Troubleshooting Common Failures
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Environment proxy not working: Check &lt;code&gt;env&lt;/code&gt; section inside &lt;code&gt;settings.json&lt;/code&gt;, verify injection with env command, restart agent.&lt;/li&gt;
&lt;li&gt;AI ignores architecture rules: Confirm rules exist in &lt;code&gt;CLAUDE.md&lt;/code&gt;, check document length and clarity of constraints.&lt;/li&gt;
&lt;li&gt;Repeated mistakes on solved problems: Validate memory hooks, ensure relevant memory entries load when matching tasks start.&lt;/li&gt;
&lt;li&gt;Changes to config have no effect: The most common root cause is editing &lt;code&gt;settings.json&lt;/code&gt; without restarting the entire Claude Code process.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The three configuration systems of Claude Code divide responsibilities clearly. &lt;code&gt;settings.json&lt;/code&gt; controls the execution environment of the agent program. &lt;code&gt;CLAUDE.md&lt;/code&gt; delivers static project rules to the LLM on every session launch. Memory stores historical observations with lazy loading, reducing repeated context setup. Misplacing configuration content creates performance loss, security risk and unexpected agent behavior. Mastering their loading semantics and boundaries is essential for reliable code agent deployment. When building multi-model agent systems, developers can simplify endpoint management through unified gateway services.&lt;/p&gt;

&lt;p&gt;International access: &lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;&lt;br&gt;
Domestic access: &lt;a href="https://4sapi.cn" rel="noopener noreferrer"&gt;https://4sapi.cn&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>api</category>
    </item>
    <item>
      <title>AI Agent Infinite Loops: Fix the System, Not the Model</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Sat, 07 Nov 2026 09:14:35 +0000</pubDate>
      <link>https://dev.to/4sapi/ai-agent-infinite-loops-fix-the-system-not-the-model-5c4o</link>
      <guid>https://dev.to/4sapi/ai-agent-infinite-loops-fix-the-system-not-the-model-5c4o</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;In production agent deployments, infinite tool-calling loops are one of the most costly and frequently misdiagnosed failure modes. When teams observe an agent stuck retrying the same failing tool call dozens of times, the instinct is often to blame the model’s reasoning ability, rewrite the system prompt, or lower the maximum step limit. In practice, root-cause analysis of real-world incidents shows the opposite: the majority of loops are not caused by weak model intelligence. They are caused by the absence of structured error semantics at the tool execution layer.&lt;/p&gt;

&lt;p&gt;This article presents a four-layer error governance framework implemented in Python 3.11+, compatible with mainstream agent frameworks including LangChain and CrewAI. The architecture systematically addresses retry loops by adding typed error classification, exponential backoff with jitter, circuit breakers and atomic state checkpoints. Controlled benchmark data from 20 side-by-side test runs quantifies the impact of each layer on call volume, task duration and failure recovery.&lt;/p&gt;

&lt;h2&gt;
  
  
  How Infinite Loops Form: Unstructured Error Strings
&lt;/h2&gt;

&lt;p&gt;In one representative production incident, monitoring logs showed a single &lt;code&gt;search_docs&lt;/code&gt; tool invoked 31 consecutive times. Every call returned a plain &lt;code&gt;429 Too Many Requests&lt;/code&gt; error string. The agent read the generic error message, interpreted the failure as a transient glitch, and continued retrying for 40 minutes until it hit the step limit. Across 6 concurrent agent tasks monitored over a two-month period, 4 exhibited similar looping behavior.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Standard Execution Loop
&lt;/h3&gt;

&lt;p&gt;The structural root cause lies in the default agent execution pattern used in most tutorials and framework implementations:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;done&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;step&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;MAX_STEPS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;action&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;decide&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;dispatch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In this pattern, every failure is returned to the model as a plain text string. The model sees only that the call failed, not whether it is safe to retry, permanently impossible, or systemically catastrophic. Without structured error semantics, the model has no way to distinguish a rate-limit error from a permission error from a missing-resource error. It defaults to retrying, and a loop forms.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why Lowering MAX_STEPS Is Not a Fix
&lt;/h3&gt;

&lt;p&gt;A common first attempt at a remedy — reducing &lt;code&gt;MAX_STEPS&lt;/code&gt; from 40 to 15 — does not solve the problem. It only makes the task fail faster. All useful work completed before the failure is discarded, and the root cause remains unaddressed. The agent still burns time and tokens on redundant retries; it just hits the wall sooner.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 1: Error Classification — Retryable vs Non-Retryable
&lt;/h2&gt;

&lt;p&gt;The first and highest-impact layer is a formal error taxonomy. Instead of returning raw error strings to the model, every tool exception is normalized into one of three structured classes. This is the foundational change that makes all higher layers effective.&lt;/p&gt;

&lt;h3&gt;
  
  
  Three-Tier Error Taxonomy
&lt;/h3&gt;

&lt;p&gt;All tool failures are mapped to three categories, each with an explicit retry policy and clear semantic meaning:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;TransientError&lt;/strong&gt;: Temporary failures that are safe to retry, such as rate limiting, network jitter and short timeouts. Marked &lt;code&gt;retryable = True&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PermanentError&lt;/strong&gt;: Failures that will not change on retry, such as missing resources, invalid arguments and insufficient permissions. Marked &lt;code&gt;retryable = False&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;FatalError&lt;/strong&gt;: Catastrophic failures that should abort the entire task, such as expired credentials, exhausted quota and full downstream outages. Marked &lt;code&gt;retryable = False&lt;/code&gt;.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;AgentToolError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;retryable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
    &lt;span class="n"&gt;error_type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;TransientError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;AgentToolError&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;retryable&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;
    &lt;span class="n"&gt;error_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;transient&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;PermanentError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;AgentToolError&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;retryable&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
    &lt;span class="n"&gt;error_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;permanent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;FatalError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;AgentToolError&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;retryable&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
    &lt;span class="n"&gt;error_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fatal&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Normalization and Tool Mapping
&lt;/h3&gt;

&lt;p&gt;A &lt;code&gt;normalize_error&lt;/code&gt; function maps every tool’s native exception types, HTTP status codes and error message patterns into the three-tier taxonomy. This mapping step is the engineering foundation of the entire framework. Building the full mapping for all integrated tools typically takes a few hours of work, and it eliminates the largest share of retry loops.&lt;/p&gt;

&lt;h3&gt;
  
  
  Measured Impact
&lt;/h3&gt;

&lt;p&gt;After adding classification alone, repeated redundant calls for the same tool in the same task dropped from 27 to 3 in representative test runs. The model receives the error category alongside the message, so it knows immediately whether retrying is a rational strategy. This single change eliminates the majority of simple infinite loops without any modification to prompts or model selection.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 2: Exponential Backoff with Jitter
&lt;/h2&gt;

&lt;p&gt;Classification tells the system which errors can be retried. The second layer controls how retries are executed, to avoid amplifying load on downstream services and creating self-reinforcing outages.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why Immediate Retry Is Harmful
&lt;/h3&gt;

&lt;p&gt;Retrying immediately after a transient error sounds reasonable in isolation, but under concurrency it creates resonant retry storms. If five parallel tasks fail at the same time and all retry immediately, they send five simultaneous requests at once. This synchronized wave of retries can overwhelm the downstream service, extend the outage and make the original problem worse.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Retry Algorithm
&lt;/h3&gt;

&lt;p&gt;The implementation uses truncated exponential backoff with uniform random jitter, with three tunable parameters:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;max_attempts = 3&lt;/code&gt;: maximum retries per logical tool call&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;base_delay = 0.8&lt;/code&gt; seconds: initial backoff base duration&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;cap = 8&lt;/code&gt; seconds: upper bound on individual delay length&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;jitter_factor = 0.5&lt;/code&gt;: range of random variation added to each delay&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Delay per attempt follows the formula:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;base_delay&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;random&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;jitter_factor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Adding jitter spreads retries across a time window and eliminates synchronized retry waves. In concurrent test scenarios, peak downstream load drops by roughly 50%.&lt;/p&gt;

&lt;h3&gt;
  
  
  Task-Level vs Tool-Level Retry Accounting
&lt;/h3&gt;

&lt;p&gt;An important design choice is to track retry budgets at the task level, not per-tool. If every tool in a 17-tool workflow got 3 retries of its own, a task could theoretically attempt 51 calls in the worst case. A task-level counter caps total retries for the entire task at 6, keeping total call volume bounded and predictable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 3: Circuit Breaker Pattern
&lt;/h2&gt;

&lt;p&gt;Backoff handles short transient failures. For sustained outages — for example, a downstream service down for half an hour — even backoff-limited retries remain wasteful. The third layer introduces a circuit breaker, following the classic three-state model popularized by Hystrix.&lt;/p&gt;

&lt;h3&gt;
  
  
  Three States of the Circuit Breaker
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Closed&lt;/strong&gt;: normal operation. All calls pass through, and consecutive failures are counted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Open&lt;/strong&gt;: failure threshold reached. All calls fail immediately with a &lt;code&gt;TransientError&lt;/code&gt;, without hitting the downstream service.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Half-open&lt;/strong&gt;: after the cool-down period elapses, one probe call is allowed through. If it succeeds, the circuit closes and counters reset. If it fails, the circuit reopens.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The default configuration trips the circuit after 5 consecutive failures, with a 60-second cool-down window. Failure counters reset on each successful call.&lt;/p&gt;

&lt;h3&gt;
  
  
  Behavioral Impact
&lt;/h3&gt;

&lt;p&gt;In the earlier 429 rate-limit scenario that lasted half an hour, a circuit breaker would have tripped on the 5th failure. From that point on, the agent would receive a clear structured signal that the tool was currently unavailable, and could switch to an alternative tool, adjust its plan, or inform the user — instead of burning another 25 minutes on fruitless retries.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 4: Atomic State Checkpoints and Resumable Execution
&lt;/h2&gt;

&lt;p&gt;The first three layers prevent infinite loops and reduce wasted work. The fourth layer ensures that work already completed is not lost when a task does fail.&lt;/p&gt;

&lt;h3&gt;
  
  
  The Full-Restart Problem
&lt;/h3&gt;

&lt;p&gt;For long multi-step tasks, an abort at step 18 of 30 discards all 17 previously completed sub-tasks. The next run starts from scratch, repeating all the work that already succeeded. For tasks that take tens of minutes, this multiplies both cost and latency.&lt;/p&gt;

&lt;h3&gt;
  
  
  Atomic Checkpoint Persistence
&lt;/h3&gt;

&lt;p&gt;After every successfully completed sub-task, the full task state is serialized and written to disk. The write uses &lt;code&gt;os.replace&lt;/code&gt; for atomicity, so a process kill mid-write can never leave a corrupted checkpoint file.&lt;/p&gt;

&lt;p&gt;The main execution loop is modified to support resumption:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;On task start, check for an existing checkpoint file.&lt;/li&gt;
&lt;li&gt;If one exists, load state and resume from the last completed sub-task.&lt;/li&gt;
&lt;li&gt;If no checkpoint exists, start from the beginning.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Permanent Error Recording
&lt;/h3&gt;

&lt;p&gt;An easily overlooked edge case: permanent errors must also be recorded in the checkpoint. Otherwise, a resumed task will immediately hit the same permanent error again, wasting a call. Recording known-bad paths ensures resumption does not repeat already-failed work.&lt;/p&gt;

&lt;h3&gt;
  
  
  Outcome
&lt;/h3&gt;

&lt;p&gt;With checkpointing, failed tasks no longer deliver zero value. A task that fails at step 18 still preserves 17 completed sub-results. The user gets partial output, and the next run picks up where the last one left off. For long batch tasks, this can reduce total runtime by more than half.&lt;/p&gt;

&lt;h2&gt;
  
  
  Combined Effect: Measured Results from 20 Control Runs
&lt;/h2&gt;

&lt;p&gt;To quantify the total impact of the full four-layer stack, 20 controlled experiments were run on identical tasks with identical prompts, comparing the baseline ungoverned agent against the fully instrumented version.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Baseline, no governance&lt;/th&gt;
&lt;th&gt;Four-layer framework&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Average tool calls per task&lt;/td&gt;
&lt;td&gt;41&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Average task duration&lt;/td&gt;
&lt;td&gt;41 minutes&lt;/td&gt;
&lt;td&gt;6 min 20 sec&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Task failure rate&lt;/td&gt;
&lt;td&gt;27 / 40&lt;/td&gt;
&lt;td&gt;6 / 20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Average duration of failed tasks&lt;/td&gt;
&lt;td&gt;27 minutes&lt;/td&gt;
&lt;td&gt;7.8 minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sub-task results delivered on failure&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;7.8&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Key Observations
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Loops are effectively eliminated&lt;/strong&gt;. Error classification and backoff remove the core conditions that cause sustained retry cycles. Most tasks now run to completion normally instead of spinning until the step limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Failures are fast and deliver partial value&lt;/strong&gt;. Instead of burning 27 minutes and producing nothing, failed tasks exit early and retain the sub-tasks that already succeeded. This changes the failure profile from total loss to partial completion.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Returns are unevenly distributed&lt;/strong&gt;. The first two layers — classification and backoff — deliver the largest share of improvement. Circuit breaking and checkpointing add smaller but increasingly important gains as task length, concurrency and runtime increase. For short simple tasks, the first two layers are often sufficient. For long production batch jobs, all four layers quickly pay for themselves.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Three Practical Implementation Recommendations
&lt;/h2&gt;

&lt;p&gt;Teams adopting this pattern can follow three prioritized recommendations to maximize return on engineering effort.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Build error classification first, everything else second
&lt;/h3&gt;

&lt;p&gt;No retry logic, circuit breaker or checkpoint system works correctly if errors are not properly categorized. Classification is the foundation that every higher layer depends on. It is also the change with the smallest code footprint and the largest immediate impact. Start by defining the three error classes and mapping existing tool exceptions into them. Do not add more complex mechanisms until classification is in place.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Retry at the task level, not the tool level
&lt;/h3&gt;

&lt;p&gt;Do not ask the model to decide how many times to retry or how long to wait. Error handling is an engineering concern, not a reasoning concern. The model should receive only final results, not intermediate retry attempts. This keeps context window usage low, reduces prompt complexity, and produces more predictable behavior.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Use atomic write for checkpoint files
&lt;/h3&gt;

&lt;p&gt;Always use &lt;code&gt;os.replace&lt;/code&gt; or equivalent atomic file operations for checkpoint persistence. Corrupted checkpoint files from interrupted writes are a common source of after-hours debugging work. Atomic write eliminates this entire failure class with one small code change.&lt;/p&gt;

&lt;h2&gt;
  
  
  Enterprise-Scale Architecture Considerations
&lt;/h2&gt;

&lt;p&gt;For individual agents and small deployments, the four in-agent layers are sufficient to eliminate most looping failures. For enterprise-scale agent fleets operating across multiple model providers and dozens of tool endpoints, application-layer error handling delivers the strongest reliability when paired with infrastructure-layer traffic governance.&lt;/p&gt;

&lt;p&gt;An enterprise-grade LLM gateway such as 4sapi provides centralized rate limiting, global circuit breaking and unified error telemetry across all upstream services. This creates a defense-in-depth architecture: in-agent handling manages per-task behavior, while the gateway layer applies global policies that apply consistently across every agent and every client tool. For organizations running dozens of agent instances, this reduces operational overhead and ensures uniform reliability standards across the entire AI estate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;As agent systems move from demonstration projects to core production workloads, their reliability bottleneck is rapidly shifting from model reasoning ability to engineering governance. Infinite loops are not evidence that the model is unintelligent. They are evidence that the system has not been given structured error signals and clear handling rules.&lt;/p&gt;

&lt;p&gt;The four-layer framework presented here is essentially an application of decades of distributed systems reliability wisdom to agent architecture. Error classification, exponential backoff, circuit breaking and checkpoint resumption are standard patterns in traditional service engineering. They work just as well for agent tool invocation chains.&lt;/p&gt;

&lt;p&gt;Over time, these mechanisms will stop being competitive differentiators and become standard production hygiene, just as they did for web services and microservices. Teams that build error governance into their agent architecture early will see lower operational costs, higher task success rates, and fewer late-night incident calls.&lt;/p&gt;

&lt;p&gt;International access: &lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;&lt;br&gt;
Domestic access: &lt;a href="https://4sapi.org" rel="noopener noreferrer"&gt;https://4sapi.org&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>api</category>
      <category>agents</category>
    </item>
    <item>
      <title>When AI Agents Claim "Task Complete": Evidence Chains and Product Gates in ChipCode’s Engineering Design</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Fri, 06 Nov 2026 09:35:09 +0000</pubDate>
      <link>https://dev.to/4sapi/when-ai-agents-claim-task-complete-evidence-chains-and-product-gates-in-chipcodes-engineering-2fmg</link>
      <guid>https://dev.to/4sapi/when-ai-agents-claim-task-complete-evidence-chains-and-product-gates-in-chipcodes-engineering-2fmg</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;As AI code agents move from experimental prototypes into production software workflows, one systemic problem consistently undermines trust and efficiency: the completion illusion. Across nearly all agent implementations, systems regularly report a task as “done”, “verified” or “tested” while leaving bugs, untested paths, and broken logic undetected. For engineering teams, this creates a hidden quality tax: every agent-generated output must be rechecked from scratch, eroding much of the productivity gain.&lt;/p&gt;

&lt;p&gt;ChipCode, an AI-native software engineering platform, addresses this problem not by asking models to be more honest, but by removing the model’s authority to declare completion entirely. Its architecture is built around two core mechanisms: an append-only evidence ledger that records verifiable test outcomes, and a set of product gates that compute task state directly from artifact files. Under this design, an agent’s self-assessment carries no formal weight. Status is determined by executable, reproducible checks, not by natural language claims.&lt;/p&gt;

&lt;p&gt;This article decomposes the root causes of false completion in AI agents, outlines the three-tier credibility model that underpins ChipCode’s design, explains the structure and rationale of its evidence ledger and product gate system, and discusses the practical tradeoffs and principles that guide implementation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Three Forms of False Completion
&lt;/h2&gt;

&lt;p&gt;In practice, “task completed” from an AI agent almost always falls into one of three categories, none of which guarantees a genuinely finished, verified result.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Tool execution return success
&lt;/h3&gt;

&lt;p&gt;The most common form of false completion is when an agent invokes a tool — a shell command, a file write operation, a test runner — and receives a zero exit code or “ok” response. The agent then reports the step as complete based solely on the return value.&lt;/p&gt;

&lt;p&gt;The flaw here is obvious on inspection: a successful return code confirms only that the command ran without crashing. It does not verify that the command produced the intended effect. A script can exit cleanly while writing incorrect output; a file save operation can succeed while containing broken logic. The agent treats execution success as outcome success, conflating process with result.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Test execution without coverage guarantees
&lt;/h3&gt;

&lt;p&gt;A more sophisticated variant appears when agents run test suites. If all executed tests pass, the agent declares the feature fully verified. In reality, the test set may cover only a tiny fraction of the changed code. The agent reports 100% pass rate while omitting that only a small share of the code path was exercised.&lt;/p&gt;

&lt;p&gt;This is not necessarily deliberate deception. It reflects how large language models reason about testing: they anchor on the visible test output and treat it as a complete picture of quality. The gap between “tests passed” and “the change is correct” is systematically understated.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Self-declared completion with no verification
&lt;/h3&gt;

&lt;p&gt;The least reliable form is when an agent simply states that the work is done, with no tool output or test result to back the claim. This is most common in early, less constrained agent implementations, but it appears even in advanced systems during long multi-turn conversations.&lt;/p&gt;

&lt;p&gt;Across all three categories, the core problem is the same: authority over task status is delegated to the same system that performs the work. There is no independent verification layer. For production use, this is an unacceptable design.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Model Self-Reporting Cannot Be Fixed With Prompts
&lt;/h2&gt;

&lt;p&gt;It is tempting to assume that better prompting can solve the problem — instructions like “be honest about verification” or “do not claim completion until fully tested.” In practice, the unreliability of self-reported completion runs deeper than prompt discipline. Three structural biases are at work.&lt;/p&gt;

&lt;h3&gt;
  
  
  Morphological bias
&lt;/h3&gt;

&lt;p&gt;Large language models learn the linguistic form of completion statements from vast corpora of technical writing. They learn what a successful verification report sounds like, and they learn to produce text that matches that form. This is not the same as performing actual verification. The model generates the language of completion because that is the statistically expected next token, not because it has independently confirmed the result.&lt;/p&gt;

&lt;h3&gt;
  
  
  Sampling bias
&lt;/h3&gt;

&lt;p&gt;When asked to produce a final summary, models optimize for a clean, confident conclusion. They tend to downplay or omit unresolved issues, edge cases, and partial failures in order to produce a coherent “success” narrative. The output reads as 100% complete, even when only 80–90% of the work is properly done. This is a sampling effect, not a deliberate lie, but the practical impact is the same: stakeholders are misled about readiness.&lt;/p&gt;

&lt;h3&gt;
  
  
  State amnesia across turns
&lt;/h3&gt;

&lt;p&gt;In long multi-turn conversations, models gradually lose precise memory of earlier verification steps. An agent that properly ran tests on turn three may, by turn ten, describe the task as fully verified even though some conditions were never met. The earlier caveats and open items fade from the context window, and the model’s self-assessment drifts upward over time.&lt;/p&gt;

&lt;p&gt;None of these three issues are reliably fixed by prompt engineering. Adding “be honest” instructions does not change the statistical patterns that produce completion language, sampling bias, or state decay. The only reliable fix is to take status determination out of the model’s hands entirely.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three Tiers of Credibility
&lt;/h2&gt;

&lt;p&gt;ChipCode’s design rests on an explicit model of credibility, divided into three tiers. Each tier has a different source of truth, a different level of reliability, and a different role in the system.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Evidence form&lt;/th&gt;
&lt;th&gt;Determined by&lt;/th&gt;
&lt;th&gt;Trust level&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Linguistic&lt;/td&gt;
&lt;td&gt;Natural language statements&lt;/td&gt;
&lt;td&gt;The model itself&lt;/td&gt;
&lt;td&gt;Lowest — subjective and prone to drift&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tool&lt;/td&gt;
&lt;td&gt;Execution logs and return codes&lt;/td&gt;
&lt;td&gt;Tool runtime&lt;/td&gt;
&lt;td&gt;Medium — confirms execution, not outcome&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Process&lt;/td&gt;
&lt;td&gt;Artifact files and scripted checks&lt;/td&gt;
&lt;td&gt;Independent verification logic&lt;/td&gt;
&lt;td&gt;Highest — reproducible, auditable, tamper-resistant&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The critical insight is that trust increases as you move down the stack. Language is the least trustworthy layer. Tool logs are better but still incomplete. Only process-level artifacts — files written to disk, outputs re-run by independent scripts, checks computed as pure functions of state — provide a stable, verifiable basis for declaring work complete.&lt;/p&gt;

&lt;p&gt;ChipCode’s entire architecture is designed to push all formal status decisions to the process tier. The agent may say whatever it wants in chat, but its words do not affect official task state. Official state is computed from files by code that the agent does not control.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Evidence Ledger: From Claims to Immutable Records
&lt;/h2&gt;

&lt;p&gt;To move verification into the process tier, ChipCode uses an append-only JSON Lines file named &lt;code&gt;evidence.jsonl&lt;/code&gt;. Every verification result is written as a new line in this file. Existing lines are never edited or deleted. The file serves as a tamper-resistant ledger of what has been tested, what passed, what failed, and what remains inconclusive.&lt;/p&gt;

&lt;p&gt;Four design principles define the ledger’s behavior.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Tri-state results
&lt;/h3&gt;

&lt;p&gt;Every verification entry records one of three states: &lt;code&gt;pass&lt;/code&gt;, &lt;code&gt;fail&lt;/code&gt;, or &lt;code&gt;inconclusive&lt;/code&gt;. Most systems use a binary pass/fail model, but binary states force the system to treat untested conditions as either passed or failed — both of which are misleading.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;inconclusive&lt;/code&gt; state is essential. It marks conditions that could not be verified, either because the test environment is unavailable, the setup is incomplete, or the check is not yet automated. This prevents the agent from treating untested paths as implicitly correct. It also forces visibility into coverage gaps.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Explicit coverage tracking
&lt;/h3&gt;

&lt;p&gt;Each entry lists exactly which conditions it covers, using a structured tag format such as &lt;code&gt;c1&lt;/code&gt;, &lt;code&gt;c2&lt;/code&gt;, &lt;code&gt;c3&lt;/code&gt;. The system does not infer coverage from file names or natural language descriptions. It reads explicit tags.&lt;/p&gt;

&lt;p&gt;This design closes a common loophole: agents cannot claim broad verification from narrow tests. If a test covers only conditions c1 and c3, the ledger records exactly that. The system can then compute which required conditions have no passing evidence, and report the gap explicitly.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Dot-note semantics: record only what is proven
&lt;/h3&gt;

&lt;p&gt;Entries describe what was verified, not what was attempted. The rule is simple: if a check ran and passed, write what it proved. If it did not run or failed, do not write a positive claim.&lt;/p&gt;

&lt;p&gt;This follows a minimal-evidence philosophy. The ledger is not a journal of everything the agent tried. It is a record of things that have been independently confirmed. This keeps the signal-to-noise ratio high and makes automated auditing straightforward.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Retraction by addition, not deletion
&lt;/h3&gt;

&lt;p&gt;When a prior finding is overturned — for example, a test that previously passed now fails after a code change — the system does not erase the old entry. It appends a new retraction entry that explicitly invalidates the earlier record.&lt;/p&gt;

&lt;p&gt;This follows the same logic as double-entry accounting. The full history remains intact. You can see when a claim was made, when it was retracted, and why. There is no way to rewrite history by editing old lines.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why JSON Lines is the right format
&lt;/h3&gt;

&lt;p&gt;The ledger uses plain JSONL rather than a database for three practical reasons.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Append-only safety&lt;/strong&gt;: New lines are written at the end of the file. Concurrent writes from multiple agents do not overwrite each other.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No infrastructure dependency&lt;/strong&gt;: The system works with a single file. No database, no service, no extra moving parts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Robustness&lt;/strong&gt;: A partially corrupted JSONL file loses only the broken line. It does not take down the whole ledger.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The tradeoff is that querying and aggregation require a small script, but for most engineering teams, a few hundred lines of Python is a negligible cost for a system that works reliably without external services.&lt;/p&gt;

&lt;h3&gt;
  
  
  Automated validation script
&lt;/h3&gt;

&lt;p&gt;The ledger is not just for humans to read. ChipCode includes a standalone validation script that reads &lt;code&gt;evidence.jsonl&lt;/code&gt;, compares covered conditions against the required set, and reports missing coverage, failed checks, and inconclusive items. This script can be plugged directly into CI pipelines or pre-commit hooks.&lt;/p&gt;

&lt;p&gt;The validation logic follows a strict order: first count failures, then count missing coverage, then count inconclusive items. Reversing the order would produce misleading summaries — for example, reporting “90% coverage” while hiding that one critical check failed entirely.&lt;/p&gt;

&lt;p&gt;For enterprise organizations operating agentic development platforms across dozens of teams, traceability must extend both into the codebase and into the model invocation layer. An enterprise-grade LLM gateway like 4sapi aligns naturally with this design: it maintains immutable, audit-grade logs of every model request and response, enforces fine-grained access policies, and supports private deployment so invocation data never leaves the organization’s domain. This infrastructure-level accountability complements the code-level evidence ledger, creating end-to-end auditability.&lt;/p&gt;

&lt;h2&gt;
  
  
  Product Gates: State Computed From Artifacts, Not Announced by Agents
&lt;/h2&gt;

&lt;p&gt;The evidence ledger records verification results. Product gates sit one level higher: they define what it means for an entire product phase to be complete, and they compute phase state directly from artifact files.&lt;/p&gt;

&lt;p&gt;ChipCode divides product development into five sequential phases: concept, styling, hardware, software, and integration. Each phase produces a defined set of artifact files. The official state of each phase — “in progress”, “blocked”, “completed” — is a pure function of those files. Given the same files, the state calculation always returns the same result.&lt;/p&gt;

&lt;h3&gt;
  
  
  How state calculation works
&lt;/h3&gt;

&lt;p&gt;State is not stored as a mutable field. It is computed on the fly by checking the presence, validity, and consistency of required artifacts.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;If required files are missing, the phase is &lt;strong&gt;in progress&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;If validation errors are present in the files, the phase is &lt;strong&gt;blocked&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;If all required artifacts exist and pass all checks, the phase is &lt;strong&gt;completed&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This design has three powerful consequences.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Agents cannot talk their way past a gate
&lt;/h3&gt;

&lt;p&gt;The agent has no authority to set state. It can only write files. Because state is computed by independent code, the agent cannot claim completion in chat and have the system believe it. The only way to advance state is to produce the required artifacts.&lt;/p&gt;

&lt;p&gt;This eliminates the completion illusion at the system level. No matter how confidently the agent describes its progress, the gate does not move until the files are in place and verified.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. State is fully reproducible
&lt;/h3&gt;

&lt;p&gt;Because state is a pure function of files, anyone on the team can re-run the state calculation and get the same answer. There is no reliance on the agent’s memory, no dependency on chat history, and no room for subjective interpretation.&lt;/p&gt;

&lt;p&gt;This is transformative for collaboration. When a reviewer opens the project, they see the same state the engineer sees. When a project is handed off between teams, state travels with the files. There is no “he said she said” about what was finished.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Reviews have a shared ground truth
&lt;/h3&gt;

&lt;p&gt;Before product gates, reviews often degenerated into arguments about what the agent said it had done. With file-computed state, both sides look at the same set of artifacts and the same set of checks. Discussion focuses on the actual work product, not on interpreting conversational history.&lt;/p&gt;

&lt;h3&gt;
  
  
  A concrete example: 3D preview gate
&lt;/h3&gt;

&lt;p&gt;One illustrative example is the 3D preview gate. For a styling phase to be marked complete, three conditions must all be met: 3D source files (&lt;code&gt;scad&lt;/code&gt;), 3D meshes (&lt;code&gt;stl&lt;/code&gt;), and a 2D render preview. If any one is missing, the phase stays in progress.&lt;/p&gt;

&lt;p&gt;Under earlier designs, the agent would often claim the styling phase was done after producing only the source files. With the product gate in place, incomplete output stays visibly incomplete. Users do not get a false sense of completion, and the agent is guided to produce the full set of deliverables.&lt;/p&gt;

&lt;p&gt;This principle generalizes: every gate has a clear, complete checklist of required artifacts. The gate stays closed until every item on the list is present and valid.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where Human Judgment Remains Necessary
&lt;/h2&gt;

&lt;p&gt;Removing the agent’s authority to declare completion does not eliminate human decision points. In ChipCode’s design, humans retain final authority at three key junctures.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Confirmation of key decisions
&lt;/h3&gt;

&lt;p&gt;Major architectural choices, tradeoff decisions, and design direction changes require explicit human approval. The agent can propose options and outline tradeoffs, but it cannot unilaterally commit to a path that changes the project’s direction.&lt;/p&gt;

&lt;p&gt;This is a deliberate guardrail. Agents excel at execution within a defined frame, but they lack the business context and risk judgment to make high-stakes tradeoffs. Keeping humans in the loop for these decisions prevents the agent from optimizing for local completion while creating larger downstream problems.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Final acceptance of deliverables
&lt;/h3&gt;

&lt;p&gt;At the end of each phase, a human reviewer performs final acceptance. The product gate confirms that all required artifacts exist and pass automated checks, but human judgment is still needed to assess quality, fitness for purpose, and alignment with intent.&lt;/p&gt;

&lt;p&gt;Automated checks catch objective failures. Human review catches subjective gaps, aesthetic issues, and cases where the work meets the letter of the spec but misses the spirit.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Handling of exceptions and edge cases
&lt;/h3&gt;

&lt;p&gt;When verification fails in unexpected ways, or when the standard gate rules do not fit a special case, a human decides how to proceed. The system defaults to blocking progress on uncertainty, but humans can grant exceptions, adjust requirements, or approve partial completion with documented rationale.&lt;/p&gt;

&lt;h3&gt;
  
  
  The value of three-state outputs
&lt;/h3&gt;

&lt;p&gt;The tri-state &lt;code&gt;pass / fail / inconclusive&lt;/code&gt; model applies at the gate level as well. Not every phase is a simple binary of done or not done. Some conditions may be verified, others blocked, and still others pending external dependencies.&lt;/p&gt;

&lt;p&gt;ChipCode therefore supports partial completion states. A phase can be recorded as partially complete with specific open items, rather than being forced into a binary “done” or “not done” box. This matches real engineering reality, where work often proceeds incrementally and some items legitimately depend on external inputs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Practical Costs of This Design
&lt;/h2&gt;

&lt;p&gt;No architecture is free. The evidence-ledger and product-gate approach delivers stronger trust and fewer completion surprises, but it comes with three measurable costs.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Extra file overhead
&lt;/h3&gt;

&lt;p&gt;Every project gains at least one additional file — the evidence ledger — plus gate definition files. For teams used to working with just source code, this adds administrative weight. On a typical 200-file project, adding one &lt;code&gt;evidence.jsonl&lt;/code&gt; is a modest cost, but it is not zero.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. JSONL is not immediately intuitive
&lt;/h3&gt;

&lt;p&gt;Developers accustomed to relational databases or structured dashboards may find a flat JSONL file inelegant. Querying history, generating summaries, and building reports all require small helper scripts. The format trades immediate visual polish for robustness and simplicity.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Team learning curve
&lt;/h3&gt;

&lt;p&gt;The whole team must internalize the rules: append only, do not edit old entries, trust the gate state over chat messages, distinguish between what is proven and what is unproven. This takes training and discipline. In the early adoption phase, teams will occasionally edit ledger entries directly or ignore gate states, undermining the system’s value.&lt;/p&gt;

&lt;p&gt;These costs are real, but they are bounded and predictable. The cost of the completion illusion, by contrast, is unbounded: undetected bugs, rework, slipped deadlines, and quality erosion that accumulates silently over time. For most engineering organizations, the tradeoff is clearly favorable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three Foundational Design Principles
&lt;/h2&gt;

&lt;p&gt;Underlying all of ChipCode’s design choices are three core principles that generalize well beyond this specific system.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Prefer files over dialogue
&lt;/h3&gt;

&lt;p&gt;Whenever status, evidence, or configuration can be represented as a file, use a file. Files are auditable, diffable, version-controllable, and reproducible. Dialogue is ephemeral, subjective, and prone to drift.&lt;/p&gt;

&lt;p&gt;This is the most important principle for building trustworthy agent systems. The more state lives in files, the less you depend on the model’s memory and honesty.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Prefer append over delete
&lt;/h3&gt;

&lt;p&gt;Whenever possible, add new records rather than modifying old ones. Append-only systems preserve history, support audit, and resist accidental data loss. They are also easier to reason about in concurrent environments.&lt;/p&gt;

&lt;p&gt;This principle applies to evidence ledgers, change logs, and state transitions. Erasing history may feel cleaner, but it destroys accountability.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Focus on what is proven, not what is unproven
&lt;/h3&gt;

&lt;p&gt;Build status systems around verified positive evidence. Do not treat absence of failure as proof of success. A task is not done because no one has found a bug yet. It is done when specific, defined checks have passed.&lt;/p&gt;

&lt;p&gt;This principle counters the natural human and model tendency to treat clean output as correct output. It forces the system to earn completion status one verified condition at a time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;The completion illusion is not a minor annoyance. It is a fundamental barrier to production-grade AI agent deployment. As long as agents are allowed to grade their own work, engineering teams cannot fully trust their output, and the productivity gains of automation are eaten up by re-verification.&lt;/p&gt;

&lt;p&gt;ChipCode’s approach — evidence ledgers, product gates, and file-computed state — solves this problem at the architectural level. It does not try to make models more honest. It removes the need to trust them at all for status determination. Verification moves from the linguistic tier, where models are unreliable, to the process tier, where independent code and immutable files provide stable, auditable ground truth.&lt;/p&gt;

&lt;p&gt;For organizations scaling agentic development, the same pattern applies at the infrastructure level. Trustworthy agent systems require both application-layer verification and infrastructure-layer governance. A robust LLM gateway with audit logging, access control, and failover reliability ensures that model invocation itself is as accountable as the code the model produces.&lt;/p&gt;

&lt;p&gt;Over time, this pattern will become standard. Engineering teams will no more accept self-declared completion from AI agents than they accept self-declared correctness from human engineers. Verification, independent checking, and auditable records are not optional extras. They are the foundation on which reliable, scalable AI-augmented software development is built.&lt;/p&gt;

&lt;p&gt;International access: &lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;&lt;br&gt;
Domestic access: &lt;a href="https://4sapi.org" rel="noopener noreferrer"&gt;https://4sapi.org&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>api</category>
    </item>
    <item>
      <title>Kimi Hosted Agents: Renting the Runtime Instead of Building It</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Thu, 05 Nov 2026 09:15:58 +0000</pubDate>
      <link>https://dev.to/4sapi/kimi-hosted-agents-renting-the-runtime-instead-of-building-it-5al4</link>
      <guid>https://dev.to/4sapi/kimi-hosted-agents-renting-the-runtime-instead-of-building-it-5al4</guid>
      <description>&lt;h2&gt;
  
  
  Abstract
&lt;/h2&gt;

&lt;p&gt;On November 5, 2026 at 11:35, the Kimi open platform released Kimi Hosted Agents, a managed agent service that delivers the harness runtime as a cloud service, described as Harness as a Service. Developers no longer build the execution framework themselves. They receive a 7×24 hosted environment supporting long-horizon tasks and multi-agent collaboration, configurable on demand for two deployment classes: connecting into internal enterprise systems for monitoring and incident response, or embedding into external products as in-app research and customer-service assistants. The announcement reads as a one-minute product note. What it actually sells is a change in who owns the operational layer. Below, the release is examined through that lens: the capability inventory, the architecture read as a contract, the economics of build versus rent, the governance terms, and the decision framework a buyer would apply.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. What Changed Is Ownership, Not Capability
&lt;/h2&gt;

&lt;p&gt;Nothing here is a new model. The token generator is unchanged. What moves is the layer between the model and the application.&lt;/p&gt;

&lt;p&gt;An agent is a program that wraps a model in a loop. It assembles context, calls tools, persists state, enforces permissions, and recovers when a step fails. The scaffolding around those functions is the harness. Building it is the costly part of any serious agent deployment, and it is the part most teams underestimate.&lt;/p&gt;

&lt;p&gt;The announcement puts a figure on that cost in operational terms rather than currency: enterprises often spend several months constructing a harness foundation covering code execution environments, context and session management, tool invocation, permission control, and recovery from task interruption. The work does not stop at launch. The environment needs ongoing maintenance and upgrades afterward.&lt;/p&gt;

&lt;p&gt;Kimi's proposition is to absorb that layer. The vendor assumes responsibility for maintenance and upgrades of the underlying environment, and the stated effect is a compressed cycle from prototype validation to application launch.&lt;/p&gt;

&lt;p&gt;Read at the level of the transaction, the buyer is not purchasing inference capacity. The buyer is transferring operational ownership of an execution tier.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. The Capability Inventory
&lt;/h2&gt;

&lt;p&gt;The released product can be inventoried in three blocks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Deployment surfaces.&lt;/strong&gt; Two classes are supported. Internal integration connects hosted agents to enterprise systems and business dashboards for monitoring and incident response. External embedding places agents inside consumer-facing products as research assistants or customer-service assistants.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Runtime properties.&lt;/strong&gt; A 7×24 hosted environment, long-horizon task execution, and multi-agent collaboration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Package contents.&lt;/strong&gt; More than 140 skills and more than 30 plugins, spanning web retrieval, professional data queries, office document processing, and content generation. Eight financial agent templates open in the first batch: investment research modeling, deep research, morning report generation, IPO tracking, and position monitoring. Documentation ships alongside, covering architecture rationale, configuration tutorials, and cost estimation.&lt;/p&gt;

&lt;p&gt;The inventory matters because it defines the scope of what the customer stops owning. Skills and plugins are commodity content. The runtime properties are the actual product.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. The Architecture Read as a Contract
&lt;/h2&gt;

&lt;p&gt;The published diagram is usually read as a feature map. It is more useful read as a statement of what responsibility transfers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Platform services
&lt;/h3&gt;

&lt;p&gt;Five groups sit above the runtime core.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Group&lt;/th&gt;
&lt;th&gt;Components&lt;/th&gt;
&lt;th&gt;Responsibility transferred&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Agent definition and configuration&lt;/td&gt;
&lt;td&gt;Planning instructions, session state, files, naming&lt;/td&gt;
&lt;td&gt;Configuration management&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Environment and automation&lt;/td&gt;
&lt;td&gt;Sandbox, browser, network access, package deployment, file handling, scheduled tasks&lt;/td&gt;
&lt;td&gt;Execution environment provisioning and scheduling&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Project-level permission governance&lt;/td&gt;
&lt;td&gt;Project isolation, resource access, keys and credentials&lt;/td&gt;
&lt;td&gt;Internal access control and secret custody&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Skills and plugins&lt;/td&gt;
&lt;td&gt;Definition, installation, invocation, results&lt;/td&gt;
&lt;td&gt;Tool integration and lifecycle management&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Memory and sandbox&lt;/td&gt;
&lt;td&gt;Session memory, agent memory, context engineering&lt;/td&gt;
&lt;td&gt;State persistence and retrieval&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Durable harness core
&lt;/h3&gt;

&lt;p&gt;Five elements sit below.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Context engineering.&lt;/strong&gt; Long-context assembly and compression, plus retrieval and summarization from history.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Customizable agent loop.&lt;/strong&gt; Objective-to-task decomposition with configurable output format, planning algorithm, and completion verification.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long-horizon execution.&lt;/strong&gt; Cross-session workflows with continuation and recovery.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Multi-agent collaboration.&lt;/strong&gt; Orchestration and planning across agents, concurrent coordination, and context isolation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runtime.&lt;/strong&gt; Task scheduling and execution isolation.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each of these is a subsystem that a self-built harness would have to implement, test, and maintain. The contract is straightforward: these subsystems are now the platform's problem.&lt;/p&gt;

&lt;h3&gt;
  
  
  Claimed value
&lt;/h3&gt;

&lt;p&gt;Four benefits are attached: out-of-the-box operation with no self-built harness and no self-managed operations; security and controllability through environment isolation, policy limits, observable events, and controllable data; self-evolution through event-trajectory logging that feeds a system control loop; and application integration through pre-built components for office work, editing, finance, and gaming.&lt;/p&gt;

&lt;p&gt;Of the four, the self-evolution claim is the most interesting and the least verifiable from outside. Trajectory logging that drives optimization implies a feedback loop between recorded execution and future behavior. Whether that loop is real, and over what timescale it produces measurable improvement, is an empirical question the announcement does not answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. The Economics of Renting
&lt;/h2&gt;

&lt;p&gt;The build-versus-rent calculation is the release's central argument, and it deserves careful framing because the honest answer is conditional.&lt;/p&gt;

&lt;p&gt;Four cost categories shift to the vendor under the hosted model.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Engineering labor.&lt;/strong&gt; Months of harness construction convert into integration work. The customer's engineers configure agents instead of implementing schedulers, sandboxes, and recovery logic.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Maintenance labor.&lt;/strong&gt; The environment requires ongoing upkeep. Upgrades, dependency churn, and compatibility fixes become the platform's responsibility rather than the customer's backlog.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Time to production.&lt;/strong&gt; The stated benefit is a shorter path from prototype to launch. Time is the cost category most teams underprice, because the delay is invisible in a budget line and visible only in a missed window.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Operational expertise.&lt;/strong&gt; Running a durable execution environment well requires skills that most application teams do not have and do not want to hire for. Renting imports that expertise without a headcount request.&lt;/p&gt;

&lt;p&gt;Three cost categories do not shift, and they are the ones that determine fit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Integration.&lt;/strong&gt; Connecting agents to internal systems, mapping business processes onto agent workflows, and validating behavior against real data remain the customer's work. A hosted runtime makes integration easier by removing infrastructure prerequisites, not by performing the integration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Governance design.&lt;/strong&gt; Defining permission boundaries, deciding what an agent may reach, and specifying escalation paths are policy decisions. No platform can make them on the customer's behalf.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Evaluation.&lt;/strong&gt; Verifying that an agent behaves correctly across a large skill surface is harder than building the runtime. A runtime with 140+ skills and 30+ plugins expands the behavioral space that must be checked. That checking capability must be built regardless of where the runtime lives.&lt;/p&gt;

&lt;p&gt;The calculation therefore resolves to a single question: does the workload's shape match the platform's abstraction? When the answer is yes, the engineering savings are real. When the workload needs custom execution semantics, unusual tool protocols, or data handling constraints the platform does not express, the savings shrink and the migration cost grows.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. The Governance Terms
&lt;/h2&gt;

&lt;p&gt;Four operational guarantees are stated explicitly, and each maps to a specific failure mode that a self-built harness typically handles badly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Isolated sandboxes with network access permission control.&lt;/strong&gt; This limits blast radius when a tool call misbehaves, and converts an implicit permission assumption into an explicit policy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Boundary definition with termination authority.&lt;/strong&gt; An operator can set operation boundaries and terminate a session at any time. Termination authority is the human interrupt path, and its existence is what makes autonomous execution acceptable in an enterprise setting.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Retained execution traces with audit export.&lt;/strong&gt; Execution is searchable after the fact, and records can be exported for audit. This turns incident investigation into a query rather than a reconstruction.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Billing rules with usage reports.&lt;/strong&gt; For a continuously running autonomous system, cost visibility is a control signal. It determines whether a workload is worth operating.&lt;/p&gt;

&lt;p&gt;Two of these deserve to be separated, because they govern different perimeters.&lt;/p&gt;

&lt;p&gt;Project-level permission governance is the internal boundary: what an agent may reach within the platform's resource model. Network access permission control is the external boundary: what the agent may send outside the sandbox.&lt;/p&gt;

&lt;p&gt;An agent can hold a valid internal credential and still make an outbound call that policy should block. It can also be permitted at the network layer while using an internal resource it should never have been granted. Both perimeters need coverage, and the external one is usually the less instrumented.&lt;/p&gt;

&lt;p&gt;Outbound traffic is where multi-provider deployments get opaque. Which upstream was called, with which credential, at what rate, and returning what status — these questions need a single answer source. An access gateway such as 4sapi occupies that position: routing outbound calls across upstream APIs, scoping credentials per project, applying rate limits per provider, and logging each call for audit. It complements the platform's internal project isolation rather than duplicating it. The platform governs what the agent can touch inside its project. The gateway governs what leaves it. When agents run continuously and call several external services in sequence, the second record is what makes the first one interpretable.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. The Interceptor Before Inference
&lt;/h2&gt;

&lt;p&gt;The security component is the most consequential technical inclusion.&lt;/p&gt;

&lt;p&gt;Kimi Inference Hook inspects task input and tool-return content before every model inference. Security rules are defined by the enterprise. The hook executes the allow-or-deny decision. Private and sensitive data is intercepted before it reaches inference.&lt;/p&gt;

&lt;p&gt;The placement carries the argument. Interception occurs before inference, not after generation. A post-generation filter examines output; a pre-inference hook examines input, which is where sensitive material actually enters the loop.&lt;/p&gt;

&lt;p&gt;Two consequences follow. Rule ownership stays with the enterprise, so policy is not dictated by the vendor. Enforcement is centralized at one choke point rather than scattered across tool implementations, which makes the policy auditable in one place.&lt;/p&gt;

&lt;p&gt;The scope limit is stated in the announcement: the component is currently open only to enterprise-verified users through targeted access. And a hook that inspects model input and tool returns covers the inbound data path. It does not by itself govern what the agent sends outward through tool calls. That remains an egress concern, which is why the two perimeters described above are distinct.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. Finance as the First Proof Point
&lt;/h2&gt;

&lt;p&gt;Eight financial templates open in the first batch. The vertical choice is not arbitrary.&lt;/p&gt;

&lt;p&gt;Financial research is a workload with long horizons, heavy document consumption, scheduled execution, and strict audit requirements. It exercises the full stack: context engineering over long filings, long-horizon execution across sessions, scheduled tasks for morning reports, position monitoring that runs continuously, and audit export for compliance.&lt;/p&gt;

&lt;p&gt;It is also a workload where error cost is high and traceability is mandatory. A platform that can host a position-monitoring agent with searchable execution traces has demonstrated more than a platform that can host a chat assistant. Choosing a demanding vertical as the first reference case is a credibility signal, and it also sets the evaluation bar for every other vertical that follows.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Lock-In Is the Product
&lt;/h2&gt;

&lt;p&gt;The final line of the announcement frames the shift: agent runtime moves from months of engineering investment to an on-demand cloud service, another step in vendors extending toward productivity deployment.&lt;/p&gt;

&lt;p&gt;That framing is accurate and incomplete. The deeper change is in the nature of what is being sold.&lt;/p&gt;

&lt;p&gt;Inference is a commodity with a published per-token price and near-zero switching cost. A hosted harness is a service with accumulating state: session memory, agent memory, configured skills, permission policies, and audit history all live in the platform. Those artifacts encode institutional knowledge built up over months of operation.&lt;/p&gt;

&lt;p&gt;Migrating a model is a configuration change. Migrating a runtime with accumulated memory and audit history is a project with a timeline and a risk register.&lt;/p&gt;

&lt;p&gt;That is a stickier product than inference, and the governance guarantees are what make the stickiness acceptable. Isolation, termination authority, audit export, and usage reporting are the terms under which a customer agrees to place operational state on someone else's infrastructure. Without them, the engineering savings would not be enough to justify the transfer.&lt;/p&gt;

&lt;h2&gt;
  
  
  9. A Decision Framework
&lt;/h2&gt;

&lt;p&gt;For a team weighing adoption, five questions sort the decision.&lt;/p&gt;

&lt;p&gt;Does the workload's execution model match the platform's five core abstractions? If the agent loop, context strategy, or collaboration model needs semantics the platform cannot express, the fit is poor regardless of savings.&lt;/p&gt;

&lt;p&gt;Is the integration surface internal or external? Internal system integration requires mapping business processes, which no platform performs. The effort estimate should account for it.&lt;/p&gt;

&lt;p&gt;Can the required policies be expressed in the pre-inference hook and the project permission model? Policy expressiveness is the binding constraint on what governance the platform can actually enforce.&lt;/p&gt;

&lt;p&gt;Is external egress instrumented? If agents call multiple upstream services, the observability record must exist somewhere. The platform's internal audit does not automatically cover outbound traffic.&lt;/p&gt;

&lt;p&gt;Can the team evaluate the agent independently? With 140+ skills, behavioral verification is the residual engineering task, and it does not disappear when the runtime is rented.&lt;/p&gt;

&lt;h2&gt;
  
  
  10. What to Verify Before Committing
&lt;/h2&gt;

&lt;p&gt;The claims worth testing are operational rather than architectural.&lt;/p&gt;

&lt;p&gt;Whether the sandbox boundary holds under adversarial tool use. Whether termination is effective against an agent mid-execution rather than only between discrete steps. Whether audit export carries tool-level detail sufficient for incident reconstruction, or only session summaries. Whether enterprise-defined rules in the pre-inference hook can express the policies an organization actually needs. Whether the documentation's cost estimation tracks metered reality across long-horizon workloads, where token consumption is driven by iteration count rather than request count.&lt;/p&gt;

&lt;p&gt;Each property would also have to be demonstrated by a self-built harness. The difference is that with a hosted service, the demonstration is a vendor evaluation conducted before commitment, not an internal build conducted after.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Kimi Hosted Agents is a runtime product wrapped in an agent announcement. It separates platform services from a durable harness core, defines internal project isolation and external network access control as distinct perimeters, and places its security interceptor before inference rather than after generation. The 140+ skills, 30+ plugins, and eight financial templates define the surface. The tenancy of the harness defines the substance.&lt;/p&gt;

&lt;p&gt;The offer is coherent and its economics are real but conditional. Months of infrastructure work convert into integration work, not into zero work. The four governance guarantees make the transfer acceptable, and the accumulated runtime state makes the transfer durable. A buyer's task is to confirm three things before signing: that the abstractions match the workload, that the policy surface can express the required rules, and that external egress is observable on its own terms.&lt;/p&gt;




&lt;p&gt;International access: &lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;&lt;br&gt;
Domestic access: &lt;a href="https://4sapi.org" rel="noopener noreferrer"&gt;https://4sapi.org&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>discuss</category>
      <category>news</category>
    </item>
    <item>
      <title>AI Agent Security Governance: From Principle Advocacy to Engineering Capability</title>
      <dc:creator>4sapi</dc:creator>
      <pubDate>Wed, 04 Nov 2026 10:17:11 +0000</pubDate>
      <link>https://dev.to/4sapi/ai-agent-security-governance-from-principle-advocacy-to-engineering-capability-42gp</link>
      <guid>https://dev.to/4sapi/ai-agent-security-governance-from-principle-advocacy-to-engineering-capability-42gp</guid>
      <description>&lt;h2&gt;
  
  
  Abstract
&lt;/h2&gt;

&lt;p&gt;Over the past two years, the main theme of AI security governance has been rule-setting. Ethics principles, risk tiers, and compliance checklists accumulated into broad consensus. That changed as AI evolved from a passive-response model into an agent that can plan autonomously, call tools, and run for long periods. A sharp question emerged: the industry does not lack principles; it lacks the layer that translates principles into engineering capability. Since 2025, several milestone actions made this clear. China's AI security governance framework iterated to version 3.0 and, for the first time, released an agent risk management framework as an appendix. The US NIST launched an AI agent standard initiative. China's CAICT published an agent deployment risk management guide. The center of governance is moving from advocacy to implementation. This article reconstructs that path, based on the technical note by vipxieliang dated November 4, 2026.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The Signal: Agent Risk Management Gets Its Own Slot
&lt;/h2&gt;

&lt;p&gt;A key change in the AI security governance framework 3.0 is separating agent risk management from general AI risk and issuing it as a dedicated appendix. The form itself carries information.&lt;/p&gt;

&lt;p&gt;The move represents three shifts. From inclusion to separate listing: the agent is no longer treated as one application scenario of a large model but as a technical form with its own risk structure. From principle to framework: the appendix provides decomposable risk dimensions and a checklist of measures, not a value statement. From model side to system side: the focus expands from model output content to the tool chains, data flows, and runtime environment that surround the agent.&lt;/p&gt;

&lt;p&gt;The implication is direct. The method used to govern chatbots cannot govern an agent that acts on its own. Regulators and standard-setters have recognized that distinction.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Three Risk Characteristics of Agents
&lt;/h2&gt;

&lt;p&gt;Designing governance capability requires understanding how agent risk differs from traditional AI risk. Three features stand out.&lt;/p&gt;

&lt;h3&gt;
  
  
  Dynamic evolution
&lt;/h3&gt;

&lt;p&gt;An agent's behavior is not generated once. It unfolds inside a perception-planning-action-observation loop. The same prompt can lead to completely different execution paths depending on tool returns and memory state. Static input-output review cannot cover this. A risk may appear only at step seven, while the first six steps all look normal. Risk accumulates gradually inside the loop rather than exposing itself at the entry point.&lt;/p&gt;

&lt;h3&gt;
  
  
  Long-term accumulation
&lt;/h3&gt;

&lt;p&gt;Agents often have memory and state persistence. A single harmless operation can be written into long-term memory and then repeatedly referenced and amplified in later tasks, forming a risk deposit. Typical scenarios include an incorrect conclusion written into a knowledge base, data obtained beyond authorization being cached, and user preferences being over-generalized. Once memory pollution forms, it spreads along a write-reference-amplify chain.&lt;/p&gt;

&lt;h3&gt;
  
  
  Cross-system propagation
&lt;/h3&gt;

&lt;p&gt;Agents connect to external systems through tool calls: databases, code repositories, email, payment interfaces, and other agents. A single unauthorized call can propagate along the call chain, with impact far beyond a single application boundary. This creates an accountability problem. When agent A calls agent B, and B operates system C, whose responsibility is it when something fails?&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Full-Lifecycle Risk and Seven Categories of Measures
&lt;/h2&gt;

&lt;p&gt;Decomposing risk by lifecycle is the first step toward implementable governance. A common division runs: design and development, deployment and launch, runtime use, and retirement.&lt;/p&gt;

&lt;h3&gt;
  
  
  Stage risks
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Stage&lt;/th&gt;
&lt;th&gt;Typical risks&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Design and development&lt;/td&gt;
&lt;td&gt;Objective-setting deviation, overly broad tool permissions, insufficient evaluation coverage, untrusted supply-chain components&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Deployment and launch&lt;/td&gt;
&lt;td&gt;Configuration errors, improper credential management, missing isolation boundaries, excessive default permissions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Runtime use&lt;/td&gt;
&lt;td&gt;Prompt injection, unauthorized operations, objective drift, memory pollution, cascading failure&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retirement&lt;/td&gt;
&lt;td&gt;Data residue, unreclaimed credentials, uncleaned dependency components&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Seven categories of prevention measures
&lt;/h3&gt;

&lt;p&gt;The measures can be grouped into seven categories, forming a closed loop from governance to technology.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Governance and accountability. Define the responsible entity, approval flow, and accountability path. Establish risk tiering and a launch-review process.&lt;/li&gt;
&lt;li&gt;Least privilege and isolation. Grant the minimum necessary permission per task, use allowlists for tool calls, and force human confirmation for sensitive operations.&lt;/li&gt;
&lt;li&gt;Input-output and behavior control. Validate against prompt injection and tool-return parameters. Set policy-based interception and rate limits for critical actions.&lt;/li&gt;
&lt;li&gt;Observability and audit. Record planning steps, tool calls, and returned results across the full chain so behavior is traceable and reproducible.&lt;/li&gt;
&lt;li&gt;Memory and data governance. Apply write validation, freshness, and provenance markers to long-term memory to prevent pollution spread.&lt;/li&gt;
&lt;li&gt;Evaluation and red-team testing. Run adversarial tests before launch, covering unauthorized access, injection, and objective hijacking, with continuous regression.&lt;/li&gt;
&lt;li&gt;Emergency response and circuit breaking. Set abnormal-behavior detection, automatic degradation, and one-click circuit breaking, with a defined incident-handling flow.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The common property of all seven is that each can be engineered rather than staying at the level of "should strengthen security management." Linked across the lifecycle, they form a complete protection chain.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. The International View: NIST's AI Agent Standard Initiative
&lt;/h2&gt;

&lt;p&gt;On the other side of the Pacific, NIST's AI agent standard initiative sends a similar signal: the standards system needs to be redefined for the agent form. The focus areas converge on several directions.&lt;/p&gt;

&lt;p&gt;Interoperability and interface standards. Communication protocols among agents, and between agents and tools, must be unified. Otherwise, a security boundary cannot be defined. Identity and authorization. How an agent is identified, and how it obtains and delegates permissions, is the precondition for system collaboration. Evaluation methods and benchmarks. Reproducible test sets and metrics are needed to measure an agent's security performance. Risk classification and terminology unification. A shared language is needed for cross-organization collaboration and regulatory alignment.&lt;/p&gt;

&lt;p&gt;A notable difference appears here. NIST's path leans toward standards and evaluation infrastructure, while the domestic framework emphasizes governance systems and accountability. The two are complementary and together form the two legs of agent security governance.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. The Implementation View: CAICT's Agent Deployment Risk Management Guide
&lt;/h2&gt;

&lt;p&gt;If the framework answers what to manage, the deployment guide answers how to manage. The CAICT agent deployment risk management guide sits closer to engineering practice. Its value lies in translating abstract requirements into concrete deployment-stage actions. Several practices can be extracted.&lt;/p&gt;

&lt;p&gt;Run a permission check before launch. Verify the tools and data scope each agent can call, item by item, and remove permissions that were temporarily opened for debugging. Add a human gate to critical actions. Operations involving funds, deletion, or external sending default to human confirmation. Establish a behavior-log baseline. Run for a period to build a normal-behavior profile, then alert on deviation. Treat memory as a data asset. Define what may be written to long-term memory, how long it is kept, and how its source is traced. Prepare a circuit-breaker plan. Specify under what conditions an agent is stopped, who decides, and how rollback happens.&lt;/p&gt;

&lt;p&gt;None of these actions is complex. That is exactly why they are the layer most easily skipped between principle and capability.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Security Engineering on the System Side
&lt;/h2&gt;

&lt;p&gt;The framework's shift from the model side to the system side is the most consequential move for practitioners. A large share of the seven measures does not live inside the model at all. Least privilege, rate limiting, observability, audit, and circuit breaking are system-layer functions. They sit between the agent and the services it touches.&lt;/p&gt;

&lt;p&gt;This is where the surrounding infrastructure becomes decisive. Every tool call an agent makes is a request that crosses a boundary. That request needs routing, validation, accounting, and logging before it reaches the target. A centralized API gateway is the natural enforcement point for several of the measures above. It can apply per-tool allowlists and quota rules, enforce rate limits on critical actions, record each call for audit, and hold credentials centrally so they are not scattered across the agent's environment. A platform such as 4sapi implements that layer: unified routing, metering, and access control that make the agent's tool calls observable and bounded. In governance terms, the gateway is the enforcement arm of the least-privilege and observability measures, and it works regardless of which model drives the agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  7. The Accountability Problem
&lt;/h2&gt;

&lt;p&gt;The cross-system risk has a governance dimension that engineering alone cannot close. When agent A calls agent B and B operates system C, the responsibility for a failure is ambiguous. A tool call may appear legitimate at the source and harmful only after several hops. Audit logs that record each hop are necessary but not sufficient; they establish what happened, not who is accountable.&lt;/p&gt;

&lt;p&gt;The practical answer is to define the boundary explicitly in advance. Each agent should declare the set of actions it may take and the systems it may reach. The gateway is the mechanism for enforcing that declaration. By acting as the single chokepoint for outbound calls, it turns an ambiguous multi-hop chain into a sequence of recorded, permission-checked steps. Accountability still requires organizational decisions, but the technical preconditions, knowing what was called, by whom, and with what permission, are created at the access layer.&lt;/p&gt;

&lt;h2&gt;
  
  
  8. Conclusion: The Three Landing Points of Governance Capability
&lt;/h2&gt;

&lt;p&gt;Looking back at this evolution, the implementation of agent security governance converges on landing points that engineering must satisfy.&lt;/p&gt;

&lt;p&gt;First, definability. Risk must be decomposed into specific stages and specific actions, not kept as a vague "security risk." Second, observability. When risk occurs, there must be permission, mechanism, and evidence to stop and roll back. A third implied landing point is disposability: the ability to act on what is observed through defined approval flows, automatic degradation, and circuit breaking, which the seven measures make explicit.&lt;/p&gt;

&lt;p&gt;Principles solve whether to do something. Capability implementation solves whether it can be done. As agents begin to actually act on behalf of users, the weight of the latter grows.&lt;/p&gt;

&lt;p&gt;For developers and architects on the front line, now is the window to move security capability forward into the design stage. Waiting until the agent is already running in production makes the cost of fixing it much higher. The framework gives direction. The deployment guide gives steps. The system layer, including the API gateway that governs tool calls, gives the enforcement point. Together they turn governance from advocacy into something that can be built, measured, and operated.&lt;/p&gt;




&lt;p&gt;International access: &lt;a href="https://4sapi.com" rel="noopener noreferrer"&gt;https://4sapi.com&lt;/a&gt;&lt;br&gt;
Domestic access: &lt;a href="https://4sapi.org" rel="noopener noreferrer"&gt;https://4sapi.org&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>tutorial</category>
      <category>devops</category>
      <category>api</category>
    </item>
  </channel>
</rss>
