<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: KarmaBox</title>
    <description>The latest articles on DEV Community by KarmaBox (@karmabox).</description>
    <link>https://dev.to/karmabox</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4171469%2Fae808189-6abd-4e75-bfde-e2d8c6fcfc51.png</url>
      <title>DEV Community: KarmaBox</title>
      <link>https://dev.to/karmabox</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/karmabox"/>
    <language>en</language>
    <item>
      <title>Ten golden tasks before you move work to a local model</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Fri, 20 Nov 2026 04:05:02 +0000</pubDate>
      <link>https://dev.to/karmabox/ten-golden-tasks-before-you-move-work-to-a-local-model-3an4</link>
      <guid>https://dev.to/karmabox/ten-golden-tasks-before-you-move-work-to-a-local-model-3an4</guid>
      <description>&lt;p&gt;Somewhere in this week's queue is a task that ran on a frontier API and could have run on a machine in your office. Which one? Most teams answer that by reading benchmark tables, and the tables cannot say. They rank models against public test sets. Your queue is private, narrow, and comes with its own files attached, so the question it poses is different from the question any benchmark answers.&lt;/p&gt;

&lt;p&gt;The gap between the two questions used to be papered over by a large capability gap: local models trailed by enough that the details did not matter. That cushion is thinner now. In an interview published on November 19, Pete Warden, one of the longest-serving engineers in deep learning, put local open models within months of the big commercial ones on benchmark capability, and argued the real barrier has moved to usability. Two more open-weight models landed the same week, Radixark's DaisyRose-27B on November 18 and Tencent's Hy4 Flash on November 17. When the benchmark gap is months and shrinking, benchmarks stop settling the argument. Your own work has to.&lt;/p&gt;

&lt;p&gt;A golden set is the cheapest way to make your own work do that. It is ten real tasks pulled from your recent queue, each with an answer you can check mechanically, run against a candidate model to decide where that class of work should live. The name borrows from gold-standard test data in machine learning. The habit is closer to an acceptance test: small, specific, and binding, with the verdict written down before anyone argues from vibes.&lt;/p&gt;

&lt;p&gt;We run agents on our own hardware all day, so we have an interest in where this lands, and we will get to it near the end. The instrument itself needs nothing from us.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why benchmarks cannot make this call
&lt;/h2&gt;

&lt;p&gt;A benchmark measures general ability across a public distribution of tasks. Three things go missing on the way to your queue.&lt;/p&gt;

&lt;p&gt;The distribution is wrong. Benchmark items are chosen to separate models from each other. Your items are chosen by your customers and your calendar, and they cluster tightly: the same invoice layouts, the same handbook, the same four report formats. A model can lag on olympiad math and sail through every task your ops queue produced this quarter. Nothing in the leaderboard tells you which situation you are in.&lt;/p&gt;

&lt;p&gt;Context is the second mismatch. Benchmarks mostly test what a model can supply from its own weights. Office work mostly tests whether a model can follow what you hand it: the price sheet is attached, the policy document is in the folder. When the material rides along with the task, the model's stored knowledge matters less and its ability to read carefully matters more, and that ability saturates earlier in model size.&lt;/p&gt;

&lt;p&gt;And the scoring hides the cost that actually hurts. A benchmark scores itself in milliseconds. Your real cost sits in the human minutes spent checking output, and that number is invisible to every leaderboard. A Harvard working paper that circulated in October 2026 looked at 718 companies using AI coding agents and found code volume up about 30 percent without matching output gains, while review time rose 49 percent. The bottleneck moved to the checking. An instrument for deciding where work runs has to price the checking in, which is exactly what the expected-answer column of a golden set does.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a checkable task looks like
&lt;/h2&gt;

&lt;p&gt;The set only works if every item can be scored without a meeting. Four check modes cover most queues, in descending order of how much you should lean on them.&lt;/p&gt;

&lt;p&gt;Exact match. The answer is a value: a vendor ID, a due date, a reorder quantity. Extraction and calculation tasks live here, and they make the best golden items because a miss is unarguable.&lt;/p&gt;

&lt;p&gt;Contains. The answer must include certain strings, in any phrasing: a retrieval question over your own handbook ("what is the refund window for annual plans?") checks as pass if the number of days and the one exception both appear.&lt;/p&gt;

&lt;p&gt;Schema-valid. The output has to parse: the meeting transcript becomes action-item JSON that validates against a schema, the product table comes back as CSV with the right columns. Formatting work checks itself.&lt;/p&gt;

&lt;p&gt;Rubric. A person scores the output against two or three written criteria. Drafting tasks end up here. Use this mode sparingly, because it reintroduces the human minutes the other three modes were saving, and write the rubric down anyway so two reviewers fail the same answer.&lt;/p&gt;

&lt;p&gt;A workable rubric is shorter than people expect. For a customer-reply draft, ours is three lines: states the resolution in the first two sentences; quotes the actual policy clause rather than gesturing at it; contains nothing the customer did not ask about. Pass requires all three. The discipline is the same as the other modes, written before the run, binary per criterion, no points for style. If a criterion needs a debate to apply, it is two criteria, or it is taste, and taste does not belong in the set.&lt;/p&gt;

&lt;p&gt;Picking items teaches you something before any model runs. A task you cannot write a check for is a task you have not finished defining, and that is useful to learn while it is still cheap. It usually means the task is judgment work wearing a routine task's clothes, and judgment work was never a local-first candidate to begin with.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before you move a task to a local model, what should you test?
&lt;/h2&gt;

&lt;p&gt;Four steps, one afternoon.&lt;/p&gt;

&lt;p&gt;Pull ten real items of one task type from last month's queue. Real, because synthetic prompts flatter models: they arrive clean, and your queue does not. Last month's, because the mess is representative: the scanned PDF, the vendor who renamed a column, the email with the forwarded thread stapled to it. One task type per set, because a mixed set produces a score that describes nothing in particular. Sample on purpose rather than at random: seven typical items, plus the two worst inputs the quarter produced, plus one known trap (the vendor whose name collides with another, the invoice with a credit line in it). A set of ten clean items certifies that the model handles clean items, which nobody doubted.&lt;/p&gt;

&lt;p&gt;Freeze the materials. The local candidate gets exactly what the current flow gets: the same attachments, the same instructions, the same reference files. If you quietly improve the prompt while switching models, the comparison leaks, and you will not know which change did the work.&lt;/p&gt;

&lt;p&gt;Write the expected answer next to each item before the first run. Afterwards, you will grade on a curve. The expectations go in a file, and the file goes in version control next to your prompts.&lt;/p&gt;

&lt;p&gt;Run, score, record. Model name, quantization level, date, score. A score without its configuration becomes a rumor within a month.&lt;/p&gt;

&lt;p&gt;The file format matters less than having one. This shape has worked for us:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# task_fit.yaml - one task type per file
task_type: invoice-extraction
model: "local-9b-q4"      # record quantization; scores do not transfer across it
date: 2026-11-20
items:
  - id: t01
    input_ref: "queue/2026-10/inv-0441.pdf"
    materials: ["vendors.csv"]
    check: exact
    expect: {"vendor_id": "V-209", "total": "4180.00", "due": "2026-12-01"}
  - id: t02
    input_ref: "queue/2026-10/inv-0457.pdf"
    check: schema
    expect_schema: "invoice.schema.json"
  - id: t03
    input_ref: "queue/2026-10/policy-q-112.txt"
    materials: ["handbook-2026.md"]
    check: contains
    expect_contains: ["14 days", "annual plans excluded"]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# score.py (sketch)
import yaml

spec = yaml.safe_load(open("task_fit.yaml"))
passed = 0
for item in spec["items"]:
    answer = ask_local(item)              # your client here
    passed += run_check(item, answer)     # exact / contains / schema / rubric
print(f'{spec["task_type"]}: {passed}/{len(spec["items"])} on {spec["model"]}')
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The quantization comment earns its place. Quantization stores each model weight in fewer bits so the model fits smaller memory; Warden's November 19 interview walks the arithmetic for a 9-billion-parameter model, roughly 9 GB at 8-bit and small enough for a 4 GB graphics card at Q4. A model that scores 9 out of 10 at one quantization level has told you nothing about a more aggressive one, so the score file records the level, and you test at the level you will actually run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three bands, and why the lines sit where they do
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7yckf5ilqe1z9z92w4bf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F7yckf5ilqe1z9z92w4bf.png" alt=" " width="799" height="167"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2godqlawuj1hukk7bbr2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2godqlawuj1hukk7bbr2.png" alt="score 10 golden questions" width="799" height="410"&gt;&lt;/a&gt;&lt;br&gt;
The lines encode one judgment: a task you must re-check more than a third of the time costs more in attention than it saves in fees. The fee side of that trade is small and public. Across current list prices, one agent task spans roughly $0.00086 to $0.0603 depending on the model (IntuitionLabs, 2026-09-05; InventiveHQ, 2026-10-19; assuming 4,000 tokens in, 800 out, 10% retries, 70% cache hits), which is why &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/how-much-does-an-ai-agent-cost" rel="noopener noreferrer"&gt;what one AI task costs&lt;/a&gt; rarely decides this question on its own. The attention side is the expensive one, and the Harvard review-time number above is what it looks like when nobody prices it.&lt;/p&gt;

&lt;p&gt;So the top band tolerates one miss in ten only because the check that caught it in testing keeps running in production as a spot-check. The middle band is honest about its cost: every item gets a human signature, which pays off when review takes seconds (confirm three extracted fields) and burns money when review re-does the task (re-read the whole contract). If review costs what the task costs, 6 to 8 is a polite way of saying no. The bottom band is a verdict about today, with an expiry date attached.&lt;/p&gt;

&lt;p&gt;One more rule that has saved us arguments: a task that scores 10 on the golden set but fails twice in its first production week goes back to the middle band without a meeting. The set is a sample. Production is the population.&lt;/p&gt;

&lt;p&gt;When an item fails, sort the failure before touching anything, because the three causes point at three different fixes. A material problem means the input itself defeated the model: the scan too blurry to read, the attachment missing. That is a preprocessing fix or a routing rule (scans go elsewhere), and no model change is implied. An instruction problem means the model did the wrong thing confidently: read the European date as American, summarized when you wanted extraction. One added sentence in the task instruction usually clears it, and the fix is free, which is why you re-run before concluding anything. A capability problem is what remains when the input was clean and the instruction was explicit and the answer is still wrong. Only the third bucket argues against the model, and in our runs it is the smallest of the three, which is worth knowing before you let one failed run end the experiment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep the set, not the scores
&lt;/h2&gt;

&lt;p&gt;Models now improve on a monthly cadence; two open-weight releases landed within 48 hours of each other this week alone. The set is how you notice without re-litigating. Same ten items, same checks, new line in the score log each quarter. A task moving from 7 to 9 across two quarters is a diff you can discuss in review, and the score log becomes the history of what your office learned to trust.&lt;/p&gt;

&lt;p&gt;Two maintenance habits keep the instrument honest. When an item ages out, the vendor redesigned their invoice, the handbook chapter was rewritten, replace it and note the swap in the log; a set that silently drifts measures nothing. And resist upgrading the set to twenty items. Ten is the budget that keeps the quarterly re-run an hour instead of an afternoon, and an instrument that is cheap to run is an instrument that actually gets run.&lt;/p&gt;

&lt;p&gt;Ownership settles the last argument. The set belongs to whoever owns the queue, the ops lead for invoices, the support lead for replies, because they are the ones who can tell a representative item from a convenient one, and they feel the standing cost of each band in their own week. Scores go wherever that team already posts its numbers. The day the set has no owner, it stops being re-run, and six months later someone quotes a stale verdict in a planning meeting as if it were current. An unmaintained golden set fails in the direction of overconfidence, which is the expensive direction.&lt;/p&gt;

&lt;p&gt;The same ten-question habit, pointed at memory instead of tasks, is how you verify &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-memory-portability" rel="noopener noreferrer"&gt;an agent's memory survived a model switch&lt;/a&gt;. Build it once, use it at both doors.&lt;/p&gt;

&lt;h2&gt;
  
  
  A set from one queue
&lt;/h2&gt;

&lt;p&gt;Consider an ops team that handles supplier invoices, forty a week, currently extracted by a cloud model and pasted into the ledger by hand. They pull ten October invoices, including the two scanned ones everyone hates, write expected values for vendor, total and due date (an hour and a half, mostly spent finding the ugly cases), and run them against a 9B model at Q4 on an office machine.&lt;/p&gt;

&lt;p&gt;First run: 7 out of 10. The two scans fail on the total field, and one digital invoice fails because the model reads a European date format as American. That is the middle band, local drafts with a signature on each, and already workable since checking three fields takes seconds.&lt;/p&gt;

&lt;p&gt;They make one change, an instruction spelling out the date convention, and re-run: 9 out of 10, with the remaining miss on the worst scan. Top band, with the spot-check watching for layout drift, and the scanned invoices routed separately. The whole exercise fit in an afternoon, and the verdict came out of their own paper, which is what makes it hold up in the next planning meeting.&lt;/p&gt;

&lt;p&gt;The two scans get a decision of their own rather than a shrug. Either they go through OCR cleanup before the model sees them, and the preprocessed versions join next quarter's set as items, or the routing rule sends anything tagged as a scan to the stronger cloud model and the log records why. Both are fine answers. The bad answer is averaging them into the score and letting one input class quietly drag a whole task type back to the expensive path.&lt;/p&gt;

&lt;p&gt;Numbers in this sketch are for the shape of the exercise; your pass rates will differ, which is the point of running it on your queue instead of ours.&lt;/p&gt;

&lt;h2&gt;
  
  
  How KarmaBox runs the split
&lt;/h2&gt;

&lt;p&gt;Our own product, KarmaBox, is built for exactly this split: the routine work runs on a machine you own, and the hard cases go to built-in cloud models only when you approve the spend. Here is how the pieces above map onto it.&lt;/p&gt;

&lt;p&gt;A golden answer, in production form, looks like this. In an internal test of our cross-border commerce avatar team (2026-11-15, test account), an avatar that had been taught a cost structure once took a single instruction and returned the landed-profit calculation in 28 seconds: $12.55 net per unit, break-even ACoS 41.84%, matching a manual recalculation. When any cost input is missing, it asks instead of estimating a range. Specific, numeric, checkable against a human redo: exactly the kind of item that belongs in a set, and exactly the kind of work the top band is for.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fax23o8yx2awz9812x963.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fax23o8yx2awz9812x963.png" alt="TPE yoga mat" width="800" height="643"&gt;&lt;/a&gt;&lt;br&gt;
A verdict only matters if it sticks, so the boundary is enforced rather than suggested. Since the local-loop release (2026-10-22), a task pinned to run locally fails loudly if it cannot complete on the machine; it does not quietly fall back to a cloud call. Your golden-set verdict stays true after you stop watching, and a failed local run arrives as a report, which is itself a data point for the next retest. The behaviour is documented under &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/what-is-sovereign-ai-avatar" rel="noopener noreferrer"&gt;what a sovereign AI avatar is&lt;/a&gt;, alongside our unplugged test (2026-10-28 to 30): 197 local inferences in one day with zero business data leaving the machine.&lt;/p&gt;

&lt;p&gt;The bottom band has a path that does not involve provisioning anything. Cloud models are built into the platform, no API keys to set up; when a task would spend money on a cloud model, the &lt;a href="https://www.karmaboxai.com/en/concept/factory/ai-avatar" rel="noopener noreferrer"&gt;avatar&lt;/a&gt; asks first and waits for approval. Moving a task between bands is a configuration change plus a yes or no from the person who owns the budget.&lt;/p&gt;

&lt;p&gt;Volume, once a task passes, is a dial rather than a re-verdict. The same machine runs several tasks in parallel, and instances scale on demand; in the published small-business example, a team ran 2 to 3 parallel instances in normal weeks and turned the dial to 15 for the holiday rush, then back down after. The golden set certifies the task type once; the instance count is just how many copies of a certified worker you want this month.&lt;/p&gt;

&lt;p&gt;And if you are deciding whether to own the machine at all, the honest operations ledger (hardware, electricity, the hour a month of upkeep) is in &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/self-hosted-ai-agent" rel="noopener noreferrer"&gt;our self-hosted agent guide&lt;/a&gt;; the &lt;a href="https://www.karmaboxai.com/en/concept/factory/product" rel="noopener noreferrer"&gt;device itself &lt;/a&gt;runs the mid-size open models this whole exercise is about. If none of that fits your stack, the YAML, the bands and the quarterly re-run work the same without us.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the set does not settle
&lt;/h2&gt;

&lt;p&gt;A golden set addresses allocation risk: the cost of putting work in the wrong place. It does not raise the capability ceiling: a task every local candidate fails stays a cloud task, whatever your preferences, and no re-test schedule changes that until the models do. Ten items are a sample, so a rare input class that never made the set will fail in production first; the spot-check exists to catch it, and the set absorbs it as a new item afterwards. A set also scores one configuration at a time, so comparing two local candidates means two full runs of the same ten items, never one blended score. Scores age with model releases, which is why the verdict carries an expiry date. And the set says nothing about where data may sit or which actions need a human signature; capability was never the whole decision, and &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/private-ai-capability-paradox" rel="noopener noreferrer"&gt;the three decisions no model makes for you&lt;/a&gt; stay yours in either band.&lt;/p&gt;

&lt;p&gt;Which ten items from last month's queue would make your first set?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;The Data Exchange, &lt;a href="https://thedataexchange.media/pete-warden-moonshine/" rel="noopener noreferrer"&gt;Pete Warden on Local AI, Open Models, Voice Interfaces, and the Future of Hardware&lt;/a&gt;, 2026-11-19&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Radixark, &lt;a href="https://www.radixark.com/blog/daisyrose-27b" rel="noopener noreferrer"&gt;DaisyRose-27B&lt;/a&gt;, 2026-11-18&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Tencent, &lt;a href="https://hy.tencent.ai/research/hy4-flash" rel="noopener noreferrer"&gt;Hy4 Flash&lt;/a&gt;, 2026-11-17&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;ComplexDiscovery, &lt;a href="https://complexdiscovery.com/harvard-researchers-find-ai-coding-agents-shifted-the-bottleneck-to-code-review/" rel="noopener noreferrer"&gt;Harvard Researchers Find AI Coding Agents Shifted the Bottleneck to Code Review&lt;/a&gt;, 2026-10 &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;IntuitionLabs, &lt;a href="https://intuitionlabs.ai/articles/llm-api-pricing-cost-per-task-comparison" rel="noopener noreferrer"&gt;LLM API Pricing: Cost per Task Comparison&lt;/a&gt;, 2026-09-05&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;InventiveHQ, &lt;a href="https://inventivehq.com/blog/llm-api-cost-comparison" rel="noopener noreferrer"&gt;LLM API Cost Comparison&lt;/a&gt;, modified 2026-10-19&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>Five entry types that belong in your AI agent's memory file</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Thu, 19 Nov 2026 04:00:39 +0000</pubDate>
      <link>https://dev.to/karmabox/five-entry-types-that-belong-in-your-ai-agents-memory-file-n24</link>
      <guid>https://dev.to/karmabox/five-entry-types-that-belong-in-your-ai-agents-memory-file-n24</guid>
      <description>&lt;p&gt;Most engineering teams version their prompts now. Very few version what the agent has learned since: the corrections, the house rules, the "we tried that in March, it broke billing" warnings that accumulate in daily use. Prompts are code. Accumulated memory is state. Teams that treat state as something the vendor will look after tend to discover the difference during a migration.&lt;/p&gt;

&lt;p&gt;This matters more in a week like this one. OpenAI shipped GPT-6.2 Astra and Sol on November 17, Google followed with Gemini 4 Neon on November 18, and Microsoft announced MAI Code 2 in the same 48 hours. Whether you should chase any of them is a separate question (we wrote about &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-avatar-vs-ai-model" rel="noopener noreferrer"&gt;the difference between an avatar and a model&lt;/a&gt; earlier this month). This post is about the narrower engineering problem: if you did switch, what file would you carry over, and would you trust what's in it?&lt;/p&gt;

&lt;p&gt;AI memory portability is the ability to move what an assistant has learned, the facts, corrections, preferences and project context it accumulated, onto a different model or runtime without re-teaching it. It is a property of where the memory lives, not of how capable the model is. A memory that lives in a file you hold survives any switch. A memory that lives anywhere else survives some switches and not others, and the difference is decided by format, not by luck.&lt;/p&gt;

&lt;p&gt;So let's talk about the file.&lt;/p&gt;

&lt;h2&gt;
  
  
  A chat log is not a memory file
&lt;/h2&gt;

&lt;p&gt;Every major assistant lets you export your conversation history, usually as JSON or HTML. That export is useful. It is also not a memory file. It is a transcript: every instruction you ever gave, including the ones you reversed a week later, with nothing marking which one won.&lt;/p&gt;

&lt;p&gt;Suppose you told the agent in June that invoices go out on the 25th, and in September that they now go out on the 1st. The transcript contains both sentences with equal weight. A new model reading that transcript has to re-derive the current rule from timestamps and context, which is exactly the re-teaching work the file was supposed to spare you.&lt;/p&gt;

&lt;p&gt;A memory file is the distilled version: current state, with history attached as metadata rather than as the payload. The distinction sounds obvious written down. Almost no export format on the market today produces it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The five entry types
&lt;/h2&gt;

&lt;p&gt;After a year of running agents on our own infrastructure, the entries that earn their place in the file sort into five types. (The taxonomy follows what gets lost in practice, which we walked through in &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-agent-memory-ownership" rel="noopener noreferrer"&gt;agent memory ownership&lt;/a&gt;; here we only care about structure.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stable facts and preferences.&lt;/strong&gt; "Reply to customers in German." "The owner's timezone is UTC+8." Low churn, high value, trivially portable as plain statements.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Corrections.&lt;/strong&gt; "The payments repo signs webhooks with the legacy key, not the one in &lt;code&gt;.env.example.&lt;/code&gt;" This is the type that hurts most to lose, because each one was paid for with a debugging session. Corrections are also the type most likely to be superseded later, so they need lifecycle fields.&lt;/p&gt;

&lt;p&gt;**Project and business context. **Current clients, price floors, naming conventions, who approves what. Changes monthly rather than yearly. Often lives half in the agent's memory and half in documents, which is fine as long as the memory entry points at the document instead of paraphrasing it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Episodic summaries.&lt;/strong&gt; Compressed accounts of what happened: "Week 32: migrated the staging cluster, two rollbacks, root cause was the pinned driver." Useful for continuity, dangerous when they silently become the only record. Keep them, but mark them as summaries so nobody mistakes compression for evidence.&lt;/p&gt;

&lt;p&gt;**Derived data. **Embedding vectors, retrieval indexes, caches. This type goes in the file's manifest but never counts as the asset itself, for reasons the re-indexing section will make concrete. If your export contains vectors but not the source text they were computed from, you have exported the derived cache and left the actual asset behind.&lt;/p&gt;

&lt;p&gt;One schema that has held up for us:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{
  "id": "mem_2026_0412",
  "type": "correction",
  "statement": "Payments repo signs webhooks with the legacy key, not the one in .env.example.",
  "scope": "repo:payments",
  "taught_by": "zoe",
  "taught_at": "2026-08-14T09:12:00Z",
  "status": "active",
  "supersedes": "mem_2026_0291",
  "review_after": "2027-02-01",
  "source_refs": ["chat:2026-08-14#t142"]
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three of these fields do the portability work, and they are precisely the ones a chat export lacks.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;status&lt;/code&gt; plus &lt;code&gt;supersedes&lt;/code&gt; encode which version of a rule is current, so a new model never has to re-litigate your history. &lt;code&gt;taught_by&lt;/code&gt; and &lt;code&gt;taught_at&lt;/code&gt; are provenance: when two entries conflict, you can resolve by authority and recency instead of guessing. &lt;code&gt;scope&lt;/code&gt; keeps a correction about the payments repo from leaking into advice about the analytics repo, which is the single most common failure we see when memory gets injected wholesale into a context window.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;review_after&lt;/code&gt; is optional and underrated. Memory that can expire gracefully stays trustworthy; memory that only accumulates eventually poisons retrieval with stale rules.&lt;/p&gt;

&lt;h2&gt;
  
  
  What stays out of the file
&lt;/h2&gt;

&lt;p&gt;Credentials, API keys, session tokens: never memory entries. They rotate on a different schedule, they have different access control needs, and an exported memory file will travel to places a key must not. Keep secrets in a secret manager and let memory entries reference capabilities by name. (Identity and credentials deserve their own treatment, which the &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-agent-identity-sovereignty" rel="noopener noreferrer"&gt;agent identity&lt;/a&gt; piece covers.)&lt;/p&gt;

&lt;p&gt;Raw conversation logs stay out too, for the superseding reason above. So do model-specific prompt tricks. If an entry only works because one particular model interprets a phrasing quirk a certain way, it is configuration for that model, not knowledge about your business. Tag it as such or accept that it dies in the move.&lt;/p&gt;

&lt;h2&gt;
  
  
  An export format that survives
&lt;/h2&gt;

&lt;p&gt;The format we have settled on is deliberately boring: one JSONL file for the entries, one Markdown mirror for humans, one manifest.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;memory-export/
  entries.jsonl        # one entry per line, schema above
  entries.md           # same content, grouped by scope, human-readable
  manifest.json        # counts, date range, schema version, index fingerprints
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;JSONL instead of one big JSON array because it diffs, streams and merges line by line. The Markdown mirror exists because the person auditing a migration at 11 p.m. is not going to read JSONL, and because a second representation catches serialization bugs (if the two disagree, the export is broken). The manifest records what embedding model the current indexes were built with, which turns the re-indexing step from archaeology into a checklist item.&lt;/p&gt;

&lt;p&gt;The whole directory belongs in version control. A diff of &lt;code&gt;entries.jsonl&lt;/code&gt; reads as a changelog of what the team taught the agent that week, and a memory change that goes through review catches a surprising number of wrong or overly broad rules before they calcify. If your team already reviews prompt changes, this is the same muscle pointed at state.&lt;/p&gt;

&lt;p&gt;To be fair to the hosted platforms: account-level memory on the major assistants does persist across that vendor's own model updates. If you moved from one model version to the next inside the same product this week, your saved memories came along. The file earns its keep at the boundaries vendors do not cross for you: moving to a different vendor, moving to a different runtime (cloud to local, or onto your own hardware), and proving after the move that nothing silently changed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Merging: the second life of the format
&lt;/h2&gt;

&lt;p&gt;A memory file that only ever has one writer is the easy case. The day a teammate runs the same avatar on their machine, or a staging copy learns something production should know, you have two files that must become one, and the format either planned for this or it didn't.&lt;/p&gt;

&lt;p&gt;Line-oriented JSONL gets you halfway: concatenate, then deduplicate by &lt;code&gt;id&lt;/code&gt;. The other half is conflict resolution, and the fields from the schema earn their keep again. Two entries with the same scope and contradictory statements resolve by &lt;code&gt;taught_at&lt;/code&gt; recency unless one &lt;code&gt;supersedes&lt;/code&gt; the other explicitly; entries from different scopes never conflict by definition, which is a quiet argument for keeping scopes narrow. When recency and authority disagree (the older entry came from the owner, the newer one from an intern's session), the merge tool should stop and ask rather than pick. A merge that needs zero human decisions is usually a merge that silently lost one.&lt;/p&gt;

&lt;p&gt;Deletions need special handling. If you remove an entry from one copy and merge with a copy that still has it, the entry comes back. The standard fix is a tombstone: the entry stays in the file with &lt;code&gt;status: "retracted"&lt;/code&gt; and an empty statement, so the deletion itself propagates. Memory that can be deleted everywhere, provably, matters more than it sounds; the moment memory contains anything about people, retraction stops being a nicety and becomes an obligation (the &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/who-owns-ai-agent-memory-company" rel="noopener noreferrer"&gt;company-side ownership questions&lt;/a&gt; make the same point from the offboarding angle).&lt;/p&gt;

&lt;h2&gt;
  
  
  How entries reach the context window
&lt;/h2&gt;

&lt;p&gt;A file on disk teaches the model nothing. At runtime, something has to decide which entries ride along with each request, and this injection policy is as much a part of portability as the schema, because it is what the new model actually experiences.&lt;/p&gt;

&lt;p&gt;The pattern that has survived contact with our own usage is two tiers. A small always-on core, stable facts and preferences, rides with every request. Everything else is retrieved per task, filtered by scope first and similarity second. The scope filter runs before the vector search, which is what keeps a payments correction from surfacing in an analytics task even when the two texts embed close to each other.&lt;/p&gt;

&lt;p&gt;The budget math is worth doing explicitly, because memory competes with the actual task for context. Forty always-on entries at an average 25 tokens each is 1,000 tokens on every call (that assumption counts the statement text only; formatting overhead adds maybe a fifth). At ten thousand calls a month that is ten million tokens of standing overhead before anyone asks a question. The lesson we took: promotion to the always-on tier should be rare and audited, and an entry that has not been retrieved in months is a candidate for archive, not for a bigger core. The file can afford to be a warehouse. The context window cannot.&lt;/p&gt;

&lt;p&gt;Two tiers also simplify migration testing. The always-on core is small enough to eyeball in the new binding's first system prompt; the retrieved tier is exactly what the golden set exercises question by question.&lt;/p&gt;

&lt;h2&gt;
  
  
  The re-index trap
&lt;/h2&gt;

&lt;p&gt;An embedding is the list of numbers a model assigns to a piece of text so that similar texts get nearby numbers; retrieval works by comparing distances between these coordinates. The part that bites people: the coordinates are only meaningful inside the model that produced them. Two embedding models are two different coordinate systems, and a distance measured between a point from each is noise, even when the models come from the same family one version apart.&lt;/p&gt;

&lt;p&gt;So if you switch embedding models and keep the old index, retrieval does not fail loudly. It degrades into confident nonsense: queries embedded in the new space are compared against stored vectors from the old space, and the nearest neighbors are near by accident.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8wszxu925b2upxjwnv9v.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8wszxu925b2upxjwnv9v.png" alt="Mixing embedding models breaks retrieval until you re-index" width="800" height="420"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Example scenario: a 200-query retrieval check against the same memory store (top-5 hit rate; the figures illustrate the mechanism, not a benchmark). The middle bar is the trap: nothing errors, results just quietly stop matching.&lt;/em&gt;&lt;br&gt;
The engineering consequences are short:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqkq5ktvzexdau55cn498.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqkq5ktvzexdau55cn498.png" alt=" " width="800" height="203"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Re-embedding is cheap relative to the damage of skipping it. Ten thousand entries averaging 60 tokens is roughly 600k tokens of embedding work (one pass, no retries assumed), which current hosted embedding APIs price in cents and a local model does for electricity. The deeper cost of hosted embeddings is schedule, since a deprecation notice forces the re-index on the vendor's calendar. Pinning an open-weight embedding model on your own hardware moves that onto yours, and the local menu keeps improving, with Radixark's DaisyRose-27B (released November 18) the latest open-weight entrant in a week otherwise dominated by closed flagship launches.&lt;/p&gt;
&lt;h2&gt;
  
  
  Verifying a migration: the golden set
&lt;/h2&gt;

&lt;p&gt;Format gets memory across; verification proves it arrived. The cheapest instrument we know is a golden set: ten questions whose answers depend on specific memory entries and can be checked mechanically.&lt;/p&gt;

&lt;p&gt;Good golden questions have crisp expected answers. "Which key signs webhooks in the payments repo?" is checkable. "Summarize our deployment philosophy" is not. Pull one question from each scope that matters, bias toward corrections (the expensive type), and store expected answers next to the questions:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# golden_set.yaml
- id: g01
  question: "Which signing key do webhooks in the payments repo use?"
  expect_contains: ["legacy key"]
  depends_on: ["mem_2026_0412"]
- id: g02
  question: "What day do invoices go out?"
  expect_contains: ["1st"]
  depends_on: ["mem_2026_0533"]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# verify.py (sketch)
import yaml

golden = yaml.safe_load(open("golden_set.yaml"))
failures = []
for g in golden:
    answer = ask_new_binding(g["question"])   # your client here
    if not all(s.lower() in answer.lower() for s in g["expect_contains"]):
        failures.append((g["id"], answer[:120]))
print(f"{len(golden) - len(failures)}/{len(golden)} passed")
for fid, preview in failures:
    print("FAIL", fid, "-&amp;gt;", preview)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it three times: against the old binding (to prove the questions are answerable at all), against the new binding on day one, and again after fixes. Expect the middle run to find something.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj505ifj9gn9ajtepfosk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fj505ifj9gn9ajtepfosk.png" alt="A 10-question golden set finds what the migration dropped" width="800" height="381"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Example scenario: in our runs the day-one failures are rarely lost entries. They are entries the new model reads differently, a date format here, an implicit unit there. Cheap to fix once named, invisible until asked.&lt;/em&gt;&lt;br&gt;
When a question fails, triage it into one of three buckets before touching anything, because the fixes point in different directions. If the entry is missing from the store, the export or merge dropped it: a format problem, fix the pipeline and re-import. If the entry is present and retrieved but the answer is wrong, the new model is misreading the statement: a phrasing problem, rewrite the entry to be model-agnostic (spell out units, use ISO dates, avoid pronouns) and add the old phrasing to a regression note. If the entry is present but never retrieved, the new embedding space ranks it differently: a retrieval problem, check the re-index actually ran and consider widening the scope filter before blaming the model. In our experience the second bucket dominates, which is good news, since rewriting a handful of statements is an afternoon, and every rewrite makes the file more portable for the switch after this one.&lt;/p&gt;

&lt;p&gt;Keep the old binding alive for a week after the switch. A golden set catches what you thought to test; the week catches what you didn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  How KarmaBox handles the file
&lt;/h2&gt;

&lt;p&gt;Our own product, KarmaBox, is built on the premise this post argues for: an avatar's memory lives in a file on a machine you own, and the model is the replaceable part. Here is how each problem above maps onto something you can configure, with the dates of what we have published.&lt;/p&gt;

&lt;p&gt;The portable file itself is the default, not an export feature. Each &lt;a href="https://www.karmaboxai.com/en/concept/factory/ai-avatar" rel="noopener noreferrer"&gt;AI avatar&lt;/a&gt; keeps its accumulated memory in a file that travels with it, and since the unified re-embodiment release (2026-10-24) the same avatar moves between cloud, a Mac and a &lt;a href="https://www.karmaboxai.com/en/concept/factory/product" rel="noopener noreferrer"&gt;KarmaBox device&lt;/a&gt; with its memory intact, no retraining step. The setting that matters to you: back the memory file up to your own disk on whatever schedule you back up code.&lt;/p&gt;

&lt;p&gt;Model switching is decoupled from key juggling. Cloud models are built into the platform, so trying this week's releases does not mean provisioning new API keys; when a task would spend money on a cloud model, the avatar asks first and waits for approval. That turns "should we try the new flagship" from an infrastructure ticket into a yes/no decision, with the memory file constant across the comparison.&lt;/p&gt;

&lt;p&gt;The on-device claim is measured, not asserted. In our published unplugged test (2026-10-28 to 30, network cable out), one device ran 197 local inferences in a day with zero business data leaving the machine, and its 886 conversations, 2 knowledge bases and 24 memory entries stayed readable on the device page, which reports "nothing has left". The full setup is documented under &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/what-is-sovereign-ai-avatar" rel="noopener noreferrer"&gt;what a sovereign AI avatar is&lt;/a&gt;. The point for this post: a memory file you can read while offline is a memory file whose location you do not have to take on faith.&lt;/p&gt;

&lt;p&gt;And the golden-set habit has a live example. In an internal test of our cross-border commerce avatar team (2026-11-15, test account), an avatar that had been taught a cost structure once answered a single instruction with the full landed-profit calculation, $12.55 per unit, break-even ACoS 41.84%, in 28 seconds, matching a manual recalculation. A question like that, specific, numeric, checkable, is exactly what belongs in a migration golden set.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcgpk1dbkl9m2sobtjgd1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcgpk1dbkl9m2sobtjgd1.png" alt="TPE yoga mat" width="800" height="643"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;What should you do if you run your stack somewhere else entirely? Everything above the product section works without us: the schema, the JSONL export, the manifest fingerprint, the golden set. The file format is the portability. We would rather you have it anywhere than lose it everywhere.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the file does not fix
&lt;/h2&gt;

&lt;p&gt;A clean migration proves your memory arrived. It does not make the new model better at using it. Capability differences between models are real, tone differences survive no file format, and the re-embedding pass costs time even when it costs little money. We call the remaining exposure re-validation cost: after any switch you still owe a period of watching outputs more closely, golden set or not, because the file carries what the agent knows, never how the new model will behave with it.&lt;/p&gt;

&lt;p&gt;Budget for that week. It is much smaller than re-teaching six months of corrections, which is the alternative on offer.&lt;/p&gt;

&lt;p&gt;If you had to rebind your agent to a different model next Monday, how many of its memories could you prove survived by lunch?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;OpenAI, &lt;a href="https://openai.com/index/introducing-gpt-6-2-astra-sol/" rel="noopener noreferrer"&gt;Introducing GPT-6.2 Astra and Sol&lt;/a&gt;, 2026-11-17&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Google, &lt;a href="https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-neon/" rel="noopener noreferrer"&gt;Introducing Gemini 4 Neon&lt;/a&gt;, 2026-11-18&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Microsoft AI, &lt;a href="https://microsoft.ai/news/introducing-mai-code-2-and-mai-code-2-flash/" rel="noopener noreferrer"&gt;Introducing MAI Code 2 and MAI Code 2 Flash&lt;/a&gt;, 2026-11-17&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Radixark, &lt;a href="https://www.radixark.com/blog/daisyrose-27b" rel="noopener noreferrer"&gt;DaisyRose-27B&lt;/a&gt;, 2026-11-18&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>llm</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Microsoft/onnxruntime is 404. Here is the audit we ran on our own pipeline</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Wed, 18 Nov 2026 09:45:08 +0000</pubDate>
      <link>https://dev.to/karmabox/microsoftonnxruntime-is-404-here-is-the-audit-we-ran-on-our-own-pipeline-1530</link>
      <guid>https://dev.to/karmabox/microsoftonnxruntime-is-404-here-is-the-audit-we-ran-on-our-own-pipeline-1530</guid>
      <description>&lt;p&gt;Most teams can name the package registry they depend on. Far fewer can say how many times their CI reaches for github.com directly. We counted ours this morning, and the trigger was not hypothetical.&lt;/p&gt;

&lt;p&gt;On November 17, github.com/microsoft/onnxruntime started returning 404. No notice, no archive banner, no redirect. We checked it ourselves at 10:15 Beijing time on November 18 and again two hours later: still 404. The thread on Hacker News filled up the same day, and the first comment set the tone: a team whose evals started failing in their pipelines that morning, with no advance warning they could find.&lt;/p&gt;

&lt;p&gt;Here is the part that decides what you should do about it. The PyPI package is still up. So are the docs at onnxruntime.ai and the NuGet package. What vanished is the repository itself: source, submodules, release assets, issue history, the security discussions people link to in audits. If your stack only ever touches the wheel, you felt nothing. If anything in your build clones, pins, or downloads from that repo, it broke on Monday.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/when-a-github-repo-disappears" rel="noopener noreferrer"&gt;Upstream risk&lt;/a&gt; is the chance that a repository, model, or artifact source you build on goes offline with no notice, no export window, and no contract that says otherwise. Microsoft has not said why the repo is gone, and it may well come back. That changes nothing about the audit below, because the next one to vanish will not announce itself either.&lt;/p&gt;

&lt;h2&gt;
  
  
  The audit: five passes over your repo
&lt;/h2&gt;

&lt;p&gt;Run everything from the repo root. Each pass answers one question: where does my build trust someone else's hosting? For each, you get the command, what the output means, and what to do with a hit. Budget an hour for a mid-sized service, most of it spent reading, and keep the results; the mirror section below works straight off this list.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pass 1: lockfiles
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -n &lt;span class="s2"&gt;"github.com"&lt;/span&gt; package-lock.json yarn.lock pnpm-lock.yaml \
  requirements*.txt poetry.lock Pipfile.lock go.sum Cargo.lock 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What it catches: packages installed from a repository instead of a registry. Registry entries resolve to the registry, which holds its own copy of every published artifact. Anything matching here resolves to a repo. In Python lockfiles, look for &lt;code&gt;git+https&lt;/code&gt; lines; in JS lockfiles, &lt;code&gt;resolved: "git+..."&lt;/code&gt; or a &lt;code&gt;github.com&lt;/code&gt; tarball URL; in Go, &lt;code&gt;go.sum&lt;/code&gt; entries are proxied by default through proxy.golang.org, so a hit in &lt;code&gt;go.mod&lt;/code&gt; with a &lt;code&gt;replace&lt;/code&gt; directive pointing at GitHub matters more than the sum file itself.&lt;/p&gt;

&lt;p&gt;What to do with a hit: decide, per line, whether the package has a registry release you can switch to. Many git-pinned dependencies exist only because someone needed one unreleased fix years ago, and the fix has long since shipped in a tagged version. Switching those back to the registry is the cheapest de-risking you will do all week. Verify the swap the boring way: build both, diff the installed tree (&lt;code&gt;pip download&lt;/code&gt; the tagged version and compare against what the git pin produced, or diff &lt;code&gt;node_modules&lt;/code&gt; listings), and run the test suite before merging. The remainder, the ones that genuinely live only in a repo, go on the mirror list.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pass 2: submodules
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git config --file .gitmodules --list 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What it catches: source trees fetched fresh on every clone. Every URL in that output is resolved at checkout time, which includes the checkout your CI does on each run. A vanished submodule breaks every new clone of your own repository, which is a strange and unpleasant failure: your repo is fine, your history is fine, and no fresh machine can build it.&lt;/p&gt;

&lt;p&gt;What to do with a hit: submodules are the strongest mirror candidates because git makes relocating them painless. Mirror the upstream (next section), then point the submodule URL at your mirror and keep the upstream as a comment:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git submodule set-url third_party/somelib https://git.internal/mirrors/somelib.git
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Commit hashes do not change, so nothing else in your history moves.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pass 3: Dockerfiles, Makefiles, and scripts
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -rn &lt;span class="s2"&gt;"github.com.*releases/download\|raw.githubusercontent.com\|git clone"&lt;/span&gt; \
  Dockerfile* Makefile makefiles/ scripts/ 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What it catches: release tarballs and raw-file downloads baked into images and setup scripts. Dockerfiles love a &lt;code&gt;curl -L&lt;/code&gt; straight into a release asset, and base-image build scripts love piping an installer from a raw URL. These URLs return 404 the moment the repo does, and they fail at image build time, which often means they fail during an incident, exactly when you are rebuilding something.&lt;/p&gt;

&lt;p&gt;What to do with a hit: download the asset once, store it in your artifact store (the same place your own build outputs go), and change the Dockerfile line to pull from there. Record the original URL and the file's checksum next to it, so provenance survives:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -LO https://github.com/ORG/REPO/releases/download/v1.2.3/tool.tar.gz
sha256sum tool.tar.gz &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; MIRROR_MANIFEST.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Pass 4: CI workflow actions
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -rn &lt;span class="s2"&gt;"uses:"&lt;/span&gt; .github/workflows/ 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;|&lt;/span&gt; grep -v &lt;span class="s2"&gt;"^\s*#"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What it catches: the pass teams forget. Each uses: someorg/some-action@v4 line is a repository someone else hosts, fetched at run time, executed with your credentials. Pinning to a commit SHA protects you from tampering. It does not protect you from deletion; the SHA still has to be fetched from a repo that exists.&lt;/p&gt;

&lt;p&gt;What to do with a hit: triage by blast radius. Actions maintained by your CI vendor are about as durable as the CI platform itself; if those vanish, you have bigger problems, so leave them. Third-party actions from individual maintainers are the fragile tier. For each one, either vendor it (most actions are a few hundred lines of JavaScript or a composite YAML you can copy into &lt;code&gt;.github/actions/&lt;/code&gt; in your own repo) or fork it into your own org and point &lt;code&gt;uses:&lt;/code&gt; at the fork. Vendoring is usually less work than it sounds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mkdir -p .github/actions/some-action
&lt;span class="c"&gt;# copy the action's action.yml and dist/ into it, commit, then:&lt;/span&gt;
&lt;span class="c"&gt;# uses: ./.github/actions/some-action&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Pass 5: model weights and AI-stack extras
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -rn &lt;span class="s2"&gt;"huggingface.co\|cdn-lfs\|\.onnx\|\.gguf\|\.safetensors"&lt;/span&gt; \
  --include=&lt;span class="s2"&gt;"*.py"&lt;/span&gt; --include=&lt;span class="s2"&gt;"*.yaml"&lt;/span&gt; --include=&lt;span class="s2"&gt;"*.toml"&lt;/span&gt; . 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null &lt;span class="o"&gt;|&lt;/span&gt; head -50
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What it catches: the heaviest upstream dependency most AI teams have, and the most casually fetched. Weights are tens of gigabytes, so people deliberately avoid keeping second copies, and the download URL quietly becomes the backup. Hosting policies for model files change more often than for code: several model families have moved behind license-acceptance gates after release, which breaks anonymous CI downloads even while the files nominally stay up.&lt;/p&gt;

&lt;p&gt;What to do with a hit: copy the exact files you serve, in the exact quantization you serve, to storage you control, and make the serving config read your copy first. A second disk is enough. While you are there, sweep the small companions that travel with weights and break loading when absent: the tokenizer files, the generation config, the chat template. They are kilobytes, they live in the same upstream repo as the weights, and a surprising number of serving stacks fetch them separately at startup. Copy the whole model directory rather than the big file alone. The download script that fetched them originally becomes the refresh script, pointed at your copy as the primary and the public URL as fallback.&lt;/p&gt;

&lt;h2&gt;
  
  
  The tally
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;grep -rhoE &lt;span class="s2"&gt;"github.com/[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+"&lt;/span&gt; \
  --include=&lt;span class="s2"&gt;"*.json"&lt;/span&gt; --include=&lt;span class="s2"&gt;"*.lock"&lt;/span&gt; --include=&lt;span class="s2"&gt;"*.yml"&lt;/span&gt; --include=&lt;span class="s2"&gt;"*.yaml"&lt;/span&gt; \
  --include=&lt;span class="s2"&gt;"*.toml"&lt;/span&gt; --include=&lt;span class="s2"&gt;"*.txt"&lt;/span&gt; --include=&lt;span class="s2"&gt;"Dockerfile*"&lt;/span&gt; . 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null \
  &lt;span class="o"&gt;|&lt;/span&gt; sort -u
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One line per unique repo your build trusts. Read the list slowly. Every entry is a thing that can go 404 on a Tuesday without asking you.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpfwme6pj5dx5iucfdzz3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpfwme6pj5dx5iucfdzz3.png" alt=" "&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One more pass decision worth writing down: what not to mirror. Transitive registry packages are already buffered by the registry and, if you set up the wheelhouse or proxy below, by your cache too; mirroring their repos adds nothing. Build outputs you can regenerate from your own source do not need copies, they need the recipe. And the long tail of small tools whose loss would cost you an hour of finding a replacement is not worth a line in the manifest. The test for every candidate is the same: if this vanished tonight, would the business stop, or would someone sigh and move on? Mirror the first category. Sigh at the second.&lt;/p&gt;

&lt;h2&gt;
  
  
  The mirror: one afternoon
&lt;/h2&gt;

&lt;p&gt;A mirror is a copy you control, refreshed on a schedule. For the repos that survived triage:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# one-time, per repo&lt;/span&gt;
git clone --mirror https://github.com/ORG/REPO.git /mirrors/REPO.git
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A &lt;code&gt;--mirror&lt;/code&gt; clone carries every branch, tag, and ref. Two things it does not carry, and both have bitten people who thought they were covered:&lt;/p&gt;

&lt;p&gt;One gap is Git LFS objects. A mirror clone copies the LFS pointer files, which are tiny text stubs, and leaves the actual binaries on the LFS server. If the upstream uses LFS (model repos often do), fetch the real objects explicitly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cd&lt;/span&gt; /mirrors/REPO.git &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; git lfs fetch --all
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The other gap is release assets. Releases are attached to the repo in GitHub's database, outside git's object store entirely. A mirror clone knows the tag a release points at and nothing about the tarballs attached to it. Grab those separately; note that the API paginates, so for a project with a long release history loop until empty, or let the CLI do it per release:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;gh release list --repo ORG/REPO --limit 200 &lt;span class="o"&gt;|&lt;/span&gt; awk &lt;span class="s1"&gt;'{print $1}'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="nb"&gt;read&lt;/span&gt; tag&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  gh release download &lt;span class="s2"&gt;"$tag"&lt;/span&gt; --repo ORG/REPO --dir &lt;span class="s2"&gt;"/mirrors/assets/REPO/$tag/"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the refresh, which is the part that makes it a mirror rather than a snapshot:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/cron.d/mirror-refresh  (or a scheduled CI job)&lt;/span&gt;
15 3 * * 1  mirror  &lt;span class="k"&gt;for&lt;/span&gt; d &lt;span class="k"&gt;in&lt;/span&gt; /mirrors/*.git&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; git -C &lt;span class="s2"&gt;"$d"&lt;/span&gt; remote update --prune&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Weekly is a sane default. The refresh is incremental, so after the first clone it costs seconds per repo. If you mirror more than a handful of repos, run the refresh with an authenticated token rather than anonymously: unauthenticated clone and API traffic hits rate limits quickly, and a refresh job that half-completes on throttling gives you mirrors of uneven age, which is worse than mirrors of known age. The read-only token from the credentials note below is the right one to use here.&lt;/p&gt;

&lt;p&gt;For registry packages, the equivalent move is a local package store. Python's is built in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip download -r requirements.txt -d ./wheelhouse
&lt;span class="c"&gt;# restore path: pip install --no-index --find-links=./wheelhouse -r requirements.txt&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For Node, a pull-through proxy such as Verdaccio caches every package your builds request and keeps serving them if the upstream entry disappears. For containers, most registries (Harbor, Artifactory, even a plain registry:2 with a proxy config) can run as a pull-through cache for Docker Hub and ghcr.io. While you are in the Dockerfiles anyway, pin base images by digest (&lt;code&gt;FROM python:3.12-slim@sha256:...&lt;/code&gt;) rather than by tag: a tag is a moving pointer on someone else's registry, and a digest plus your pull-through cache together mean the image you built with is the image you can always rebuild with. The common shape across all three ecosystems: your build asks your cache, your cache asks the internet once, and from then on the internet is an optimization.&lt;/p&gt;

&lt;p&gt;Where to put the mirrors is less important than having them. A NAS in the office works. So does a persistent disk on the box that runs your CI. One useful rule: put the mirror in a different failure domain from the thing it protects. A mirror that lives in the same cloud account as your CI protects you from upstream deletion and from nothing else; a disk in the office additionally survives your cloud account having a bad day. The storage for a typical service's full dependency set is measured in tens of gigabytes, which is to say: less than one model checkpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pointing CI at your copies
&lt;/h2&gt;

&lt;p&gt;A mirror nobody reads is a warm feeling, so the last step is making the build prefer it. Doing this incrementally beats a big-bang migration, because each step is testable on its own.&lt;/p&gt;

&lt;p&gt;Order the work by blast radius: start with the dependency whose disappearance would stop you longest (for most AI teams that is weights, then the inference runtime, then submodules), and leave the long tail of small packages to the registry cache.&lt;/p&gt;

&lt;p&gt;The fallback order inside each build step should read: your mirror first, public URL second, fail third. In a Dockerfile that is usually two lines and an &lt;code&gt;||&lt;/code&gt;; in CI config, most runners let you define the mirror as the primary source and the public host as a named fallback. Resist the temptation to flip it (public first, mirror as backup): that configuration looks the same on a green day and silently rots, because the mirror path never gets exercised until the day it must work.&lt;/p&gt;

&lt;p&gt;Which leads to verification. You validate a mirror the same way you validate &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/when-a-github-repo-disappears" rel="noopener noreferrer"&gt;offline operation&lt;/a&gt;: cut the path and watch. Add a CI job, weekly is plenty, that runs the build inside a network namespace where github.com and the registry hosts are unreachable, and passes only if the build completes from mirrors alone. On most runners the cheapest version is a container with an /etc/hosts that pins the public hosts to a blackhole address. The first run of that job is usually educational: it finds the two or three fetches every team forgot they had.&lt;/p&gt;

&lt;p&gt;Checksums close the loop. Keep a &lt;code&gt;MIRROR_MANIFEST.txt&lt;/code&gt; that maps each mirrored artifact to its sha256 and the original URL, and have the refresh job verify it. If an upstream force-pushes or re-tags (it happens), you want the refresh to fail loudly rather than quietly absorb the rewrite, because a silent change in mirrored content is exactly the class of thing the security team should look at before the build does. Weights deserve the same treatment with higher stakes: hash the exact files you serve when you first copy them, and have the serving host verify the hash at load time. A corrupted or swapped weight file does not fail loudly on its own; it just answers differently.&lt;/p&gt;

&lt;p&gt;Two small operational notes that save grief later. Give the mirror host its own read-only credential story: CI should be able to fetch from mirrors with a token that can do nothing else, and the refresh job's write credential should live on the mirror host only, never in CI secrets, so a compromised pipeline cannot rewrite your copies. And write the three-line README in &lt;code&gt;/mirrors/&lt;/code&gt; saying what refreshes when and who owns it. Mirrors fail silently when the one person who set them up leaves; the README costs a minute and outlives them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a team actually does with this
&lt;/h2&gt;

&lt;p&gt;Consider a ten-person team running a product with a retrieval service, two fine-tuned models in production, and ordinary CI. Their audit produces: 3 git-pinned Python packages (two have tagged releases now, one is genuinely repo-only), 2 submodules, 6 third-party actions (4 from the CI vendor, 2 from individuals), a Dockerfile that curls one release tarball, and 160 GB of weights fetched from a hosting URL at deploy time.&lt;/p&gt;

&lt;p&gt;The afternoon's work: switch two lockfile lines back to the registry, mirror the third repo plus both submodules (three &lt;code&gt;git clone --mirror&lt;/code&gt; commands, one LFS fetch), vendor the two individual actions, move one tarball into the artifact store, and copy the weights to the NAS with the serving config reading the NAS path first. The cron entry and the weekly blackhole job go in before anyone forgets. Total new storage: under 200 GB, dominated entirely by the weights. Nothing about the team's daily workflow changes; the public internet is still the convenient path, just no longer the only one.&lt;/p&gt;

&lt;p&gt;That is the whole program. It is not elegant, and it does not need to be.&lt;/p&gt;

&lt;p&gt;The week after, the useful move is making the default stick so the list never regrows. Put the mirror-first fallback into whatever template new services are stamped from, so the next repo starts with the right fetch order instead of inheriting the audit as a chore. Add the tally command to the new-service checklist, and have the weekly blackhole job cover every service, including the ones that never got the initial treatment. From then on the audit stops being an event; it is a property the template enforces and the weekly job proves.&lt;/p&gt;

&lt;p&gt;One honest caveat before the product section. A mirror addresses upstream risk and nothing else. If the upstream shipped a backdoored commit last month, your mirror preserves it faithfully; that is security risk, and it needs review, not copies. And a mirror of an abandoned project still receives no fixes; that is maintenance risk, and at some point it needs a fork with an owner or a migration. The weekly refresh even gives you a natural checkpoint for noticing the second kind: a mirror that has pulled nothing new for six months is telling you something about the upstream's health. Keep the three risks apart when you decide what deserves the effort.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we apply this one layer up
&lt;/h2&gt;

&lt;p&gt;Our own product, KarmaBox, is built on the other side of that bet: the runtime, the models, and the avatar's memory live on a machine you own. The device ships with its models preinstalled, so the day-to-day work an &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/what-is-sovereign-ai-avatar" rel="noopener noreferrer"&gt;AI avatar&lt;/a&gt; does there, chat, writing, retrieval, image understanding, transcription, video, runs without reaching upstream at all. The local loop is enforced: if a local job fails, it reports failure rather than quietly falling back to a cloud endpoint, which is the same design argument as mirror-first fallback order above. In a three-day unplugged test we ran in October (October 28 to 30, on our own device), the box logged 197 local inferences with zero business data leaving it. The avatar's memory sits in a file the owner can carry to another machine. When a job genuinely needs a cloud model, that path is built in, with no keys for the owner to manage, and anything that costs money asks for approval first.&lt;/p&gt;

&lt;p&gt;The audit above is the same idea at a smaller scale: know what you pull from other people's machines, and hold a copy of whatever you cannot afford to lose.&lt;/p&gt;

&lt;p&gt;The repo may be back by the time you read this. Suppose it is. How many entries on your tally would survive the next one, and which of them would you miss first?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Hacker News, &lt;a href="https://news.ycombinator.com/item?id=50539610" rel="noopener noreferrer"&gt;Microsoft's ONNX Runtime has disappeared from GitHub&lt;/a&gt;, November 17, 2026&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;PyPI, &lt;a href="https://pypi.org/project/onnxruntime/" rel="noopener noreferrer"&gt;onnxruntime&lt;/a&gt;, checked November 18, 2026&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Microsoft, &lt;a href="https://onnxruntime.ai" rel="noopener noreferrer"&gt;ONNX Runtime documentation&lt;/a&gt;, checked November 18, 2026&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>devops</category>
      <category>git</category>
      <category>opensource</category>
      <category>ci</category>
    </item>
    <item>
      <title>Find out what your AI tools actually send home</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Tue, 17 Nov 2026 11:20:31 +0000</pubDate>
      <link>https://dev.to/karmabox/find-out-what-your-ai-tools-actually-send-home-4bn6</link>
      <guid>https://dev.to/karmabox/find-out-what-your-ai-tools-actually-send-home-4bn6</guid>
      <description>&lt;p&gt;&lt;em&gt;Startup connections, telemetry defaults, update channels, dependency pings: four checks you can run this afternoon.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;On November 16, a researcher reverse-engineering Windows 11 found a system component, MissionControlAggregator.dll, that records which AI coding agents run on the machine. The feature is called CLICodingAgentUsage. It keeps a bitmap of which agent CLIs have been used (Codex, Copilot, Cursor and others), and that record travels with the "optional diagnostic data" setting, which is on by default. The researcher verified the behavior in a Windows 11 ARM64 virtual machine on build 26200 and reported seeing no prompts or code in the payload, only the usage record. Microsoft had not commented as of this writing (&lt;a href="https://xusheng.dev/posts/windows-coding-agent-telemetry/" rel="noopener noreferrer"&gt;xusheng.dev&lt;/a&gt;, November 16, 2026).&lt;/p&gt;

&lt;p&gt;Nobody found that by reading a privacy policy. Someone ran the binary and watched what it did.&lt;/p&gt;

&lt;p&gt;The same method works for every AI tool on your machine. The privacy page tells you what the vendor intends to collect. Traffic tells you what the software does. Between those two there are four channels where data leaves without you asking: the connections a tool opens at startup, the telemetry it sends by default, the update checks it runs on its own schedule, and the reporting built into its dependencies. Each channel can be checked from outside, with tools you probably already have, in about half an hour per machine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Start with DNS
&lt;/h2&gt;

&lt;p&gt;Before a process can send anything, it has to look up a hostname. Watching DNS for a few minutes while you launch a tool gives you the destination list with almost no setup.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo tcpdump -i any -n port 53
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run that in one terminal, launch the AI tool in another, and read the lookups. On a quiet machine the output is short enough to read line by line. An inference tool looking up its API domain is expected. The lines worth attention are the other ones: analytics domains, error trackers, a CDN that only hosts an update manifest.&lt;/p&gt;

&lt;p&gt;If the machine is too noisy for a live capture, move the question to the resolver. A Pi-hole on the network, or NextDNS on the device, logs every lookup per client. Let it run for a working day, then filter the log by the hours you used the tool. You get the same list, with timestamps, and you did not have to babysit a terminal.&lt;/p&gt;

&lt;p&gt;DNS gives you who. It does not tell you what was sent or which process asked. The next two checks fill that in.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tie the connections to processes
&lt;/h2&gt;

&lt;p&gt;On macOS and Linux, &lt;code&gt;lsof&lt;/code&gt; shows open sockets per process:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;lsof -i -P -n &lt;span class="o"&gt;|&lt;/span&gt; grep -i &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;toolname&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it while the tool is working and you see every live connection with the destination address and port. Run it while the tool is idle and you learn something subtler: which connections stay open when nothing is happening.&lt;/p&gt;

&lt;p&gt;On Linux, &lt;code&gt;strace&lt;/code&gt; catches connections that are too short-lived for &lt;code&gt;lsof&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;strace -f -e &lt;span class="nv"&gt;trace&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;network -o /tmp/net.log &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;tool&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --version
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A version check should not need the network. It often uses it anyway, because the same startup path runs an update ping. The log file shows every socket call the process and its children made.&lt;/p&gt;

&lt;p&gt;On macOS, an outbound firewall does this continuously. LuLu (free, from Objective-See) or Little Snitch prompts on every new outbound connection per process. The first day is noisy. After that, the accumulated ruleset is an inventory of everything on the machine that talks to the network, which is exactly the document you were trying to produce.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put a proxy in the middle
&lt;/h2&gt;

&lt;p&gt;Hostnames and processes still leave one question open: what is in the payload. For HTTP traffic, mitmproxy answers it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mitmproxy
&lt;span class="nv"&gt;HTTPS_PROXY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;http://127.0.0.1:8080 &lt;span class="nv"&gt;HTTP_PROXY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;http://127.0.0.1:8080 &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;tool&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Most CLIs honor the proxy environment variables. After you trust mitmproxy's certificate (its docs cover this per platform), you see full request paths, headers, and body sizes. A telemetry POST with a 2 KB JSON body is a different finding from one shipping 400 KB per request.&lt;/p&gt;

&lt;p&gt;Some tools pin their certificates and refuse to connect through the proxy. That refusal is also a finding: the tool insists on a channel you cannot inspect. Whether that is acceptable depends on what data the tool can reach, and it belongs in your notes either way.&lt;/p&gt;

&lt;h2&gt;
  
  
  If the machine is Windows
&lt;/h2&gt;

&lt;p&gt;The record that opened this piece was found on Windows, and the checks above have direct equivalents there. In an elevated prompt, &lt;code&gt;netstat -b -n&lt;/code&gt; lists connections together with the executable that opened them. &lt;code&gt;pktmon&lt;/code&gt; has shipped with Windows since 2018 and captures packets without installing anything: pktmon &lt;code&gt;start --capture&lt;/code&gt;, reproduce the behavior, &lt;code&gt;pktmon stop&lt;/code&gt;, then convert the trace to a format Wireshark reads with pktmon's conversion command. For per-process network events with filters, Sysinternals Process Monitor does what &lt;code&gt;strace&lt;/code&gt; does on Linux.&lt;/p&gt;

&lt;p&gt;The diagnostic-data channel itself has a built-in viewer. Settings, then Privacy &amp;amp; security, then Diagnostics &amp;amp; feedback, then "View diagnostic data" enables the Diagnostic Data Viewer, which shows the events Windows has queued for upload. That is the queue the coding-agent usage record rides in, so you can read for yourself what your machine is about to say about you.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fka4oorc7rjn5anl2sdxq.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fka4oorc7rjn5anl2sdxq.png" alt="where one workstation's AI-tool traffic goes"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Example scenario: DNS lookups from one developer workstation over 24 hours, grouped by destination category. Inference calls dominate by count, and nobody is surprised by them. The telemetry category is the one that was never discussed with anyone.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  Telemetry is a setting, and the default is usually on
&lt;/h2&gt;

&lt;p&gt;Almost none of what the capture finds is hidden. It is documented, toggled by an environment variable or a settings key, and enabled by default. A partial list that covers a lot of developer machines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;DO_NOT_TRACK&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1                &lt;span class="c"&gt;# consoledonottrack.com convention&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;DOTNET_CLI_TELEMETRY_OPTOUT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 &lt;span class="c"&gt;# .NET CLI&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;HOMEBREW_NO_ANALYTICS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1       &lt;span class="c"&gt;# Homebrew&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;NEXT_TELEMETRY_DISABLED&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1     &lt;span class="c"&gt;# Next.js&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;HF_HUB_DISABLE_TELEMETRY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1    &lt;span class="c"&gt;# Hugging Face libraries&lt;/span&gt;
&lt;span class="kd"&gt;export&lt;/span&gt; &lt;span class="nv"&gt;SAM_CLI_TELEMETRY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0           &lt;span class="c"&gt;# AWS SAM CLI&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Azure CLI keeps its switch in config rather than the environment (&lt;code&gt;az config set core.collect_telemetry=false&lt;/code&gt;), and the Google Cloud CLI does the same (&lt;code&gt;gcloud config set disable_usage_reporting true&lt;/code&gt;). VS Code reads "&lt;code&gt;telemetry.telemetryLevel": "off"&lt;/code&gt; from its settings. On Windows, the same Diagnostics &amp;amp; feedback page from the section above holds the off switch: turning off optional diagnostic data closes the upload queue you just read.&lt;/p&gt;

&lt;p&gt;Each project documents its toggle, usually with a reasonable explanation of what gets collected. The pattern across all of them is the direction of the default. Collection is on until the user finds the page that says how to turn it off. The party that sets the default is the party that wants the data, so the default favors collection. Your audit does not need to argue with any of this. It needs to record which toggles exist on your machine and which state they are in.&lt;/p&gt;

&lt;h2&gt;
  
  
  Updates and dependencies report separately
&lt;/h2&gt;

&lt;p&gt;The tool you audited is not the only sender in its own install. Auto-updaters contact vendor servers on their own schedule, whether or not you use the tool that day. Model runtimes contact a registry when they pull weights. Some packages ping an analytics service at install time (many of these honor DO_NOT_TRACK). Error-reporting SDKs ship stack traces, and stack traces can carry file paths, hostnames, and fragments of whatever was in scope when the exception fired.&lt;/p&gt;

&lt;p&gt;A data processing agreement, when you have one, covers the product's intended flows. The telemetry of the product's dependencies is typically nobody's contract. If the tool runs where confidential work happens, the dependency layer deserves the same capture treatment as the tool itself: same tcpdump, same proxy, pointed at an install and an update instead of a chat session.&lt;/p&gt;

&lt;p&gt;The instinct generalizes past telemetry. A post on the front page of Hacker News this week describes running coding agents without handing them credentials at all (&lt;a href="https://blog.antonio.cx/running-agents-with-lend" rel="noopener noreferrer"&gt;blog.antonio.cx&lt;/a&gt;, November 17, 2026). Separating what a tool can read from what it can transmit is one control. Deciding what it can read in the first place is the same control, applied a layer earlier.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to write down
&lt;/h2&gt;

&lt;p&gt;The capture session produces a pile of hostnames. The deliverable is smaller. For each AI tool on the machine, four lines:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;Endpoints observed (from DNS, lsof, or the proxy)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Which of those the vendor documents, and where&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Which telemetry toggles exist, and their current state&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;What you could not inspect (pinned channels, closed-source updaters)&lt;br&gt;
Keep the file in the repo next to the tooling docs and re-run the capture after major version updates, because defaults change between versions. One more line worth adding per tool: does the vendor state where inference happens, as a specific commitment, separate from where data is stored? If the documentation never says, write "unspecified." For anything processing client work, an unspecified inference location is an answer, and it is the answer you needed the audit to surface.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How we handle this at KarmaBox
&lt;/h2&gt;

&lt;p&gt;KarmaBox is our product, and it exists because we wanted the answer to this audit to be short.&lt;/p&gt;

&lt;p&gt;It is a device you run yourself (one GB10, 128 GB of unified memory, around 170 W, sits in an office). Avatars, the AI workers on the platform, run their inference locally for local-model tasks, and business data (documents, conversations, knowledge bases, task records) stays on the device. The audit above collapses into one question: what leaves this box?&lt;/p&gt;

&lt;p&gt;The device answers that question continuously instead of once. The data boundary panel shows outbound events as they happen. In our internal test from October 28 to 30, 2026, the device ran 197 local inference calls over 72 hours with zero business-data exits; 886 conversation segments, 2 knowledge bases, and 24 memory records stayed on the device, and the panel read "nothing has left" throughout.&lt;/p&gt;

&lt;p&gt;Two behaviors matter for the failure cases above. Local inference fails closed: if the local model errors out, the task reports failure rather than silently rerouting to a cloud endpoint (deployed October 22, 2026). A silent fallback is precisely the kind of traffic a packet capture exists to catch, so the platform refuses to generate it. And when a task genuinely needs a cloud model, the cloud models are built into the platform, there are no API keys for you to configure, and if the call will cost money the avatar asks you first and waits. A boundary crossing becomes a decision you approved with a receipt attached, rather than a default you discover in a DNS log.&lt;/p&gt;

&lt;p&gt;The rules sit at the platform layer, outside the model, so swapping the underlying model does not loosen them. Sensitive work goes to avatars configured for local models; public-facing work can go to cloud avatars. That split is a &lt;a href="https://www.karmaboxai.com/en/concept/factory/product" rel="noopener noreferrer"&gt;configuration you choose on the device&lt;/a&gt;, per avatar.&lt;/p&gt;

&lt;p&gt;What this does not cover: integrations you connect (a calendar API is a new data flow with its own jurisdiction), people with physical access to the device, and the outbound channels an agent can be talked into using, which is its own problem with its own controls (&lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-agent-data-leakage" rel="noopener noreferrer"&gt;AI agent data leakage&lt;/a&gt; covers those). And a box you run is hardware you must patch: the OS-level checks above apply to our device the same as to any other machine. If you are comparing vendors on these points, the &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-vendor-sovereignty-checklist" rel="noopener noreferrer"&gt;AI vendor sovereignty checklist&lt;/a&gt; asks the questions in a form you can send to a sales team.&lt;/p&gt;

&lt;p&gt;Which tool on your machine would you least like to see a day of DNS logs for?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://xusheng.dev/posts/windows-coding-agent-telemetry/" rel="noopener noreferrer"&gt;Windows 11 Is Collecting Data on Which Coding Agents and Developer Tools You Use&lt;/a&gt; — xusheng.dev, November 16, 2026&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://blog.antonio.cx/running-agents-with-lend" rel="noopener noreferrer"&gt;Running coding agents without handing them my credentials&lt;/a&gt; — blog.antonio.cx, November 17, 2026&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://learn.microsoft.com/en-us/windows/privacy/configure-windows-diagnostic-data-in-your-organization" rel="noopener noreferrer"&gt;Diagnostic data in Windows&lt;/a&gt; — Microsoft Learn, accessed November 17, 2026&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://consoledonottrack.com" rel="noopener noreferrer"&gt;Console Do Not Track&lt;/a&gt; — consoledonottrack.com, accessed November 17, 2026&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables" rel="noopener noreferrer"&gt;Hugging Face Hub environment variables&lt;/a&gt; — Hugging Face documentation, accessed November 17, 2026&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>privacy</category>
      <category>security</category>
      <category>tooling</category>
    </item>
    <item>
      <title>Your Self-Hosted Agent Runs. Now Put the Gate Outside the Model</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Mon, 16 Nov 2026 14:12:19 +0000</pubDate>
      <link>https://dev.to/karmabox/your-self-hosted-agent-runs-now-put-the-gate-outside-the-model-52om</link>
      <guid>https://dev.to/karmabox/your-self-hosted-agent-runs-now-put-the-gate-outside-the-model-52om</guid>
      <description>&lt;p&gt;Most self-hosting guides end at docker run. The container starts, the agent answers in the chat pane, and the guide declares victory. The part that decides whether the agent is safe to leave alone with your inbox starts right after that command, and it has very little to do with the model.&lt;/p&gt;

&lt;p&gt;A self-hosted agent that can send email and move money needs three things the model never touches: somewhere to keep the keys that the agent process can use but not read in bulk, a policy file the agent can't edit, and a log the agent can't rewrite. All three fit in an afternoon. What follows is the version we'd hand to a teammate, with the code small enough to read in one sitting.&lt;/p&gt;

&lt;p&gt;Here's why the third point matters more than it looks. n8n's Nod, released November 10, puts a separate confirmation step in front of sending, paying, posting and deleting, and n8n says plainly in the same announcement that the step is model-driven and can make mistakes. That candor is useful. It tells you where the remaining risk sits: anywhere a model is asked to approve its own actions, the approval is a probability.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keys: where the agent's secrets live
&lt;/h2&gt;

&lt;p&gt;The first decision after the container is up is how the agent reaches a model. For a stack you assemble yourself, the simplest answer is your own provider account: change the endpoint and the key, keep everything else. Whose key it is matters less than one property: swapping the model must not drag the credentials, memory, policy and log along with it.&lt;/p&gt;

&lt;p&gt;Where the key lives matters as much as whose it is. The pattern we use:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Model keys and service credentials go in the OS keychain or a secrets file readable only by the agent's service user (chmod 600, owned by that user).&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The agent reads a key at the moment it calls a service. It never gets a "list all secrets" capability.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Nothing secret goes into the agent's memory store or its prompt context. If the model can see a key, a prompt injection can ask the model to print it.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That last rule is the one people break first, usually by pasting a token into a system prompt "just for testing." Keep a grep in your pre-commit hook for anything that looks like sk- or Bearer in prompt templates; it costs nothing and catches the habit early.&lt;/p&gt;

&lt;p&gt;Two operational facts come with renting a model, whoever's key you use. Whatever the agent sends to the model vendor falls under that vendor's data policy, so a task whose content must not leave the building belongs on a local model. And rate limits become your problem in a busy week, which is a good reason to keep a second provider's key configured even while you mostly use one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The gate: a policy file the model never sees as editable
&lt;/h2&gt;

&lt;p&gt;The core of the setup is a function that sits between "the model wants to do X" and "X happens." The model proposes an action with a type and a target. The gate looks the action type up in a policy file the owner wrote, and returns one of three answers: allow, ask a human, or deny. The model's own opinion about whether the action is fine is not an input.&lt;/p&gt;

&lt;p&gt;The policy file first. TOML, because humans edit it:&lt;/p&gt;

&lt;h1&gt;
  
  
  Owner-edited. The agent process can read this file but not write it.
&lt;/h1&gt;

&lt;p&gt;default = "ask"&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[actions]
"fs.read"       = "allow"
"draft.create"  = "allow"
"email.send"    = "ask"
"payment.send"  = "ask"
"file.delete"   = "ask"
"policy.write"  = "deny"

[allowlists]
"email.send" = ["accounts@supplier-one.example", "ops@client-two.example"]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details carry most of the weight. default = "ask" means an action type nobody anticipated goes to a human instead of slipping through. policy.write = "deny" means the agent can't loosen its own rules, even if a prompt asks it to. And the allowlist lets replies to known addresses go out on their own while anything to a new address still waits for you.&lt;/p&gt;

&lt;p&gt;Make the "can read but not write" part real at the filesystem level: the policy file belongs to your user, the agent runs as a different service user with read permission only. A rule the agent could edit is a suggestion.&lt;/p&gt;

&lt;p&gt;Now the gate (Python 3.11+, standard library only):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;
"""Minimal action gate: the model proposes, this file decides."""
import hashlib, json, sys, time, tomllib
from pathlib import Path

POLICY = Path("policy.toml")
LOG = Path("actions.log.jsonl")

def load_policy():
    with POLICY.open("rb") as f:
        return tomllib.load(f)

def decide(policy, action, target):
    rule = policy["actions"].get(action, policy.get("default", "ask"))
    allowed = policy.get("allowlists", {}).get(action)
    if rule == "ask" and allowed and target in allowed:
        rule = "allow"   # known recipients skip the human, by owner rule
    return rule

def append_log(entry):
    prev = "0" * 64
    if LOG.exists():
        lines = LOG.read_text().splitlines()
        if lines:
            prev = json.loads(lines[-1])["hash"]
    entry["prev"] = prev
    body = json.dumps(entry, sort_keys=True)
    entry["hash"] = hashlib.sha256((prev + body).encode()).hexdigest()
    with LOG.open("a") as f:
        f.write(json.dumps(entry, sort_keys=True) + "\n")

def gate(action, target, ask_human=input):
    policy = load_policy()
    rule = decide(policy, action, target)
    outcome = rule
    if rule == "ask":
        reply = ask_human(f"Agent wants {action} -&amp;gt; {target}. Approve? [y/N] ")
        outcome = "approved" if reply.strip().lower() == "y" else "refused"
    append_log({"ts": time.time(), "action": action,
                "target": target, "rule": rule, "outcome": outcome})
    return outcome in ("allow", "approved")

def verify_log():
    prev = "0" * 64
    for n, line in enumerate(LOG.read_text().splitlines(), 1):
        e = json.loads(line)
        h = e.pop("hash")
        if e["prev"] != prev:
            return f"broken chain at line {n}"
        if hashlib.sha256((prev + json.dumps(e, sort_keys=True)).encode()).hexdigest() != h:
            return f"edited entry at line {n}"
        prev = h
    return "log intact"

if __name__ == "__main__":
    if sys.argv[1:] == ["verify"]:
        print(verify_log())
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your agent's tool layer calls &lt;code&gt;gate(action, target)&lt;/code&gt; before every side effect and only proceeds on &lt;code&gt;True&lt;/code&gt;. The policy is reloaded on each call, so an edit you make takes effect on the next action without a restart. &lt;code&gt;ask_human&lt;/code&gt; defaults to a terminal prompt; in practice you'd swap it for a push notification or a chat message with approve and refuse buttons, and the rest of the file doesn't change.&lt;/p&gt;

&lt;p&gt;Two extensions are worth adding once the basics hold. The first is a rate cap per action class: even an allowed class like replies to known contacts should stop at a number per hour that you'd consider normal, because an agent stuck in a loop can do a lot of allowed damage fast. It's a counter keyed on action type, read from the same policy file. The second is a dry-run mode, where the gate logs what it would have decided and always returns False. Run a new tool in dry-run for a few days and you get a list of every action it would take before it takes any of them.&lt;/p&gt;

&lt;p&gt;A quick run to see it behave:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;from gate import gate, verify_log
print(gate("fs.read", "inbox/"))                                  # True  (allow)
print(gate("email.send", "ops@client-two.example"))               # True  (allowlisted)
print(gate("email.send", "stranger@elsewhere.example",
           ask_human=lambda q: "n"))                              # False (human said no)
print(gate("policy.write", "policy.toml"))                        # False (deny)
print(verify_log())             
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;




&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                              # log intact&lt;br&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;h2&gt;
&lt;br&gt;
  &lt;br&gt;
  &lt;br&gt;
  The log: append-only, chained, on your disk&lt;br&gt;
&lt;/h2&gt;

&lt;p&gt;Every call to the gate writes one JSON line, whether the answer was yes or no. Each line carries the SHA-256 of the line before it, so the file forms a chain. Edit any past line, even to flip a "refused" into "approved," and &lt;code&gt;python gate.py verify&lt;/code&gt; names the line that no longer matches.&lt;/p&gt;

&lt;p&gt;A hash chain on a file you control has one honest limit: someone with write access to the whole file can rewrite every line and recompute the chain. Two cheap habits close most of that gap. Copy the latest hash somewhere the agent can't reach once a day (a note on your phone works). And include the log directory in your nightly backup, so an old copy exists to compare against. For a single-owner agent that combination is usually enough; teams with compliance needs should ship the log to write-once storage instead.&lt;/p&gt;

&lt;p&gt;What earns the log its keep is the weekly read. Filter for &lt;code&gt;rule == "ask"&lt;/code&gt; and look at what the agent tried to do that needed you. That short list tells you which classes are safe to move to allow (four quiet weeks is our usual bar) and which ones keep surprising you.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the gate changes, in one picture
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv606v886ihafbbmsjl7y.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fv606v886ihafbbmsjl7y.png" alt="Who said yes? 40 injected instructions, two gate designs" width="800" height="552"&gt;&lt;/a&gt;&lt;br&gt;
The chart is a scenario, built to show the shape of the difference. It does not report a benchmark. What makes the shape reliable is mechanical: under the platform gate, the send and pay columns can only be non-zero if the policy file says allow for those classes or a human typed y. Nothing the model reads can change either of those.&lt;/p&gt;

&lt;p&gt;The weekly read of the log is also where the scenario's numbers would come from in your own setup. Count the held sends and payments, and count how many of them you refused. If you refuse almost none for a month, the class may be ready for an allowlist; if you refuse a steady few, the gate is doing the job you installed it for.&lt;/p&gt;

&lt;p&gt;Notice also what the gate leaves alone. Reads and drafts go through in both designs, because the owner allowed them. A prompt injection that makes the agent read a file it was already allowed to read, or write a silly draft, still happens. The gate narrows the damage to what your policy permits; how strict to make that policy is your judgment.&lt;/p&gt;

&lt;h2&gt;
  
  
  How KarmaBox does the same job without the glue code
&lt;/h2&gt;

&lt;p&gt;Our own product, KarmaBox, is a platform and network for running your own AI avatars, and each piece of the sixty lines above has a built-in counterpart there.&lt;/p&gt;

&lt;p&gt;The policy file becomes &lt;a href="https://www.karmaboxai.com/en/concept/factory/ai-avatar" rel="noopener noreferrer"&gt;per-avatar permission settings&lt;/a&gt;. Six action classes (sending externally, publishing publicly, deleting, spending money or trading, writing to the ledger, executing commands on a machine) are each set to do it directly, ask first, or never. The defaults follow the same logic as &lt;code&gt;default = "ask"&lt;/code&gt;: ledger writes, which only the owner sees and can undo, run directly; sending and publishing ask first; spending defaults to never. Since October 23, 2026, payment always needs the owner's own approval, with no switch to automate it. In policy.toml you could write payment.&lt;code&gt;send = "allow"&lt;/code&gt;. In KarmaBox you can't.&lt;/p&gt;

&lt;p&gt;The gate runs outside the model, so it survives a model swap. When someone other than the owner is directing an avatar, it checks with the owner first, which catches a stranger's steering email.&lt;/p&gt;

&lt;p&gt;The log becomes a receipt per task: what was asked, what the avatar did, what it caught, and what it handed back to a human. Work that matters can go on to an acceptance avatar, which checks the result line by line and never edits or sends anything itself.&lt;/p&gt;

&lt;p&gt;In an internal beta run on November 15, 2026 (test account), we asked a cross-border team's ads avatar to lower two Amazon keyword bids before the ads account was connected. It couldn't read the real keyword IDs, so it changed nothing and returned a pending plan instead of guessing an ID.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8iiv2sirdjmt2i3lml2e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8iiv2sirdjmt2i3lml2e.png" alt="The KarmaBox ads avatar replying that the Amazon ads account is not connected" width="800" height="380"&gt;&lt;/a&gt;&lt;br&gt;
Key custody works differently. In the script the agent borrows your keys. A KarmaBox avatar has its own name, outbound address, inbox and phone line, so client replies and verification codes go through the avatar's address rather than your account. Cloud models come built in, so there is no model key to store, and a task that would spend money on one waits for your approval. The data-policy point above still applies.&lt;/p&gt;

&lt;p&gt;The model can stay in the building too. Avatars run on a &lt;a href="https://www.karmaboxai.com/en/concept/factory/product" rel="noopener noreferrer"&gt;KarmaBox in your office&lt;/a&gt;, where chat, writing, retrieval, image understanding, transcription and video models run on the device. Since October 22, 2026, &lt;a href="https://www.karmaboxai.com/en/concept/factory/sovereign-ai" rel="noopener noreferrer"&gt;a local-loop enforcement gate&lt;/a&gt; makes local work fail rather than fall back to the cloud without telling you. In an unplugged run on October 28 to 30, it completed 197 local inferences in a day with no business data leaving.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fclhbmihe0w0s4q2xvo1e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fclhbmihe0w0s4q2xvo1e.png" alt=" " width="799" height="308"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you're building your own, the sixty lines above get you most of the way. If you'd rather not maintain them, the left column is a checklist for any platform and the right column is KarmaBox's answer. Either way the target is criterion five of a &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/what-is-sovereign-ai-avatar" rel="noopener noreferrer"&gt;sovereign AI avatar&lt;/a&gt;: it can act, and its actions are judged outside the model.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this leaves the model
&lt;/h2&gt;

&lt;p&gt;None of this makes the model less useful. It still plans the work, writes the drafts and picks the next step. It just stops being the last line of defense, which it was never good at being. When a newer model arrives you point the endpoint at it and the gate, the keys and the log stay exactly where they were.&lt;/p&gt;

&lt;p&gt;A gate, scripted or built in, addresses one risk: an agent acting beyond what its owner allowed. It doesn't write a good policy for you, since a class set to "allow" is enforced as faithfully as one set to "never." It doesn't protect a machine that nobody patches or backs up, and it doesn't make a cloud model's data policy go away for the tasks you send there. Those stay with the owner, whichever way you build it.&lt;/p&gt;

&lt;p&gt;The longer version, covering which layers must stay on your machine, what you can keep renting, and what self-hosting costs in hours and dollars, is our guide to the &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/self-hosted-ai-agent" rel="noopener noreferrer"&gt;self-hosted AI agent&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Which action class would you leave at "ask" the longest?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;n8n Labs — &lt;a href="https://blog.n8n.io/meet-nod/" rel="noopener noreferrer"&gt;Meet Nod&lt;/a&gt; — November 10, 2026&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Python Software Foundation — &lt;a href="https://docs.python.org/3/library/tomllib.html" rel="noopener noreferrer"&gt;tomllib: Parse TOML files&lt;/a&gt; — Python 3.11+ documentation&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>security</category>
      <category>selfhosted</category>
    </item>
    <item>
      <title>Your Prompt Says "Escalate If Unsure." Google Just Shipped What That Actually Takes</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Thu, 12 Nov 2026 09:27:41 +0000</pubDate>
      <link>https://dev.to/karmabox/your-prompt-says-escalate-if-unsure-google-just-shipped-what-that-actually-takes-4lpo</link>
      <guid>https://dev.to/karmabox/your-prompt-says-escalate-if-unsure-google-just-shipped-what-that-actually-takes-4lpo</guid>
      <description>&lt;p&gt;Most agent codebases already contain a confidence check. Few contain one that ever fires. It usually looks like a line of system prompt: "If you are not confident in your answer, escalate to a human." The model reads it, the model agrees with it, and the model then proceeds to be confident about everything, because nothing in that sentence defines confident, measures it, or routes anything anywhere when the measurement comes up short.&lt;/p&gt;

&lt;p&gt;On November 11, 2026, Google published a blunt opinion about where that check should live instead: outside the generating model entirely. The Gemini Decisions API is a separate, priced product that answers bounded questions and attaches a calibrated 0-to-1 confidence score to each answer. You do not have to buy it (we will get to self-hosted alternatives), and you should read it as an architecture document either way. A vendor that sells generation by the token just told its customers that judgment is a different workload, with different latency, different pricing, and an explicit instruction in its own launch post: use the score to route uncertain cases "to a fallback or human reviewer."&lt;/p&gt;

&lt;p&gt;Below, one concrete agent decision gets wired both ways. Same task, same policy question, two architectures: the judgment inlined into the generation prompt, and the judgment as a separate decision layer with thresholds, fallback routes, and an audit log. The differences are mechanical rather than philosophical, which is the nicest kind of difference to argue about.&lt;/p&gt;

&lt;h2&gt;
  
  
  The task we will wire up
&lt;/h2&gt;

&lt;p&gt;A support agent drafts refund responses. Policy, written by a human, says: refunds under $50 for damaged items can go out automatically; anything ambiguous waits for the support lead; anything over $200 or smelling of fraud always gets a person. The agent's job is to draft the reply and decide which bucket this case falls into.&lt;/p&gt;

&lt;p&gt;That second half is the judgment. The same wiring applies unchanged to outbound email, listing updates, or ticket closures; refunds just make the error costs easy to picture. Here is what it looks like when it lives inside the generation prompt.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture A: the self-grading prompt
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SYSTEM = """You are a support agent. Draft a refund decision.
Policy: auto-approve damaged-item refunds under $50.
If the request is ambiguous, or you are not confident,
do not decide; output ESCALATE instead.
Also rate your confidence 0-100 at the end."""

def handle(ticket: str) -&amp;gt; str:
    draft = llm.generate(SYSTEM + "\n\nTicket:\n" + ticket)
    if "ESCALATE" in draft:
        return queue_for_human(ticket, draft)
    return send(draft)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everyone has shipped a version of this, including us. It works in the demo and it fails in four specific places.&lt;/p&gt;

&lt;p&gt;**The score is decorative. **That trailing "confidence 0-100" is a token sequence the model produces because you asked for one. It is a plausible-sounding number, generated by the same next-token machinery as the prose around it. Public evaluations have repeatedly found self-reported confidence from general chat models to be poorly calibrated; a model that says 90 is at best weakly related to an outcome that happens 90% of the time. Nothing in the pipeline above checks the number against reality, stores it, or trends it. It is theater that compiles. (The one place a self-assessment helps: as a cheap pre-filter ahead of a real check, where a model flagging its own doubt can only add recall. The failure mode is treating it as the check.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The judge and the defendant share a brain&lt;/strong&gt;. The same forward pass that produced the draft evaluates the draft. When the generation goes wrong because the model misread the ticket, the self-evaluation misreads it the same way. Correlated errors are the whole reason code review exists and the whole reason auditors do not report to the CFO they audit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The policy is unversioned.&lt;/strong&gt; Your threshold lives in English, inside a prompt, probably inside a string literal, possibly inside three string literals that have drifted apart. When someone asks "what was the escalation rule on October 3rd," the honest answer is a git archaeology expedition. When compliance asks "who changed it," the answer is worse.&lt;/p&gt;

&lt;p&gt;**There is no ledger. **send(draft) fires and the decision evaporates. No record of what the score was, which rule version was live, or why this particular ticket sailed through. The first time you need that record is the first time something goes wrong, which is exactly when you do not have it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture B: a separate decision layer
&lt;/h2&gt;

&lt;p&gt;The alternative moves the judgment into its own call, against its own model, returning a score your code can act on. With Google's new product, the request is a bounded question over the same context. The shape below follows the launch post's description (yes/no with probability, multiple choice over 2–255 options, rubric scoring; one multimodal input up to 256K tokens; up to 1,000 questions in a single pass). Field names here are illustrative pseudocode; the exact schema is whatever the official docs say.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Decision request (shape per Google's launch post, 2026-11-11;
# field names illustrative, consult official docs)
decision = decisions_api.ask(
    context = ticket_text,
    questions = [
        {"id": "policy_ok",  "type": "yes_no",
         "text": "Does this request qualify for auto-approval "
                 "under the damaged-item policy pasted below?"},
        {"id": "fraud_risk", "type": "yes_no",
         "text": "Does this request show signs of refund abuse?"},
        {"id": "amount_tier", "type": "choice",
         "options": ["under_50", "50_to_200", "over_200"]},
    ],
    policy_text = POLICY_V7,
)
# -&amp;gt; each answer arrives with a probability and a 0-1 confidence
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The generating model still drafts the reply. It just no longer grades its own homework. Around that call, three short pieces of code do what the prompt sentence only gestured at.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The threshold, as config.&lt;/strong&gt; A policy object, versioned like any other config, reviewed like any other config:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;THRESHOLDS = {                     # policy v7, owner: support-lead
    "auto_approve":  {"policy_ok": 0.95, "fraud_risk_max": 0.10,
                      "amount": "under_50"},
    "queue":         {"policy_ok": 0.70},   # below auto, above floor
    # below 0.70, or over_200, or fraud signals: always human
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;The fallback route.&lt;/strong&gt; Low confidence is not an error; it is the third output. The launch post's own framing, routing uncertain cases to "a fallback or human reviewer," is a switch statement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;def route(decision, draft):
    a = decision["policy_ok"]
    if a.confidence &amp;gt;= 0.95 and passes_hard_rules(decision):
        return send(draft)
    if a.confidence &amp;gt;= 0.70:
        return queue(draft, reason="ambiguous", score=a.confidence)
    return queue(draft, reason="low_confidence", score=a.confidence)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;The ledger.&lt;/strong&gt; One append per routed decision, written before anything sends:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ledger.append({
    "ts": now(), "ticket": ticket_id,
    "question": "policy_ok", "answer": a.value,
    "confidence": a.confidence,
    "threshold_version": "v7",
    "route": route_taken,
    "reviewer": None,            # filled when a human acts
    "reviewer_verdict": None,
})
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the whole architecture. No machine-learning team appears in it. The hard part, deciding the numbers in THRESHOLDS, is an organizational exercise (the method: tier actions by blast radius, price both error types, run two weeks in shadow mode, grant automation downward from the most conservative setting). We wrote that method up separately; wiring is the part a codebase can enforce, so wiring is what gets the code blocks.&lt;/p&gt;

&lt;h2&gt;
  
  
  Phrasing the questions is half the engineering
&lt;/h2&gt;

&lt;p&gt;The decision call is only as good as the questions, and bounded questions have their own craft. Three rules cover most of it.&lt;/p&gt;

&lt;p&gt;One judgment per question. "Does this qualify under the policy and show no fraud signals?" is two questions wearing one question mark, and a single probability over a conjunction is uninterpretable: a 0.55 could mean confident-yes on policy with doubts on fraud, or the reverse. Split them, as the example does, and each score means one thing.&lt;/p&gt;

&lt;p&gt;Put the policy in the context, never in the question. A question that paraphrases the policy drifts from the policy; the prompt above pastes POLICY_V7 verbatim and asks whether the request qualifies under the pasted text. When the policy changes, the decision input changes with it, and the ledger's threshold_version row tells you which policy every past judgment was made against.&lt;/p&gt;

&lt;p&gt;Prefer choices to free rubrics where you can. A 2-to-255-option choice with normalized probabilities ("under_50 / 50_to_200 / over_200") gives you a distribution you can threshold; an open scale invites the model to cluster around flattering middles. Save rubric scoring for things that are rubrics in real life, like severity grades a human team already uses.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two architectures, item by item
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F576rst8rn4x66c9u47gy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F576rst8rn4x66c9u47gy.png" alt=" " width="800" height="635"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Two rows there deserve a paragraph each, because they carry most of the argument.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cost row: judgment is now the cheapest thing in your stack
&lt;/h2&gt;

&lt;p&gt;At $0.10 per million input tokens with output tokens free (Google's published price, 2026-11-11), a policy check over a 2,000-token ticket costs $0.0002, assuming no retries. For comparison, a single generation task under our standard accounting (4,000 tokens in, 800 out, 10% retry, 70% cache hit; InventiveHQ pricing survey, modified 2026-10-19) runs $0.00086 to $0.0603 depending on model tier. The check costs a fraction of the work it checks, at every tier.&lt;/p&gt;

&lt;p&gt;That kills the old excuse for Architecture A. Inlining judgment into the prompt was always partly a cost decision: a second model call per action felt extravagant. It is now a rounding error, and Google's own usage makes the point. The launch post describes the same API routing everyday internal requests between model tiers, with latency in those routed scenarios dropping by up to 40% (Google's own figure, about its own systems, 2026-11-11); the mechanics of that pattern are what &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/multi-model-routing-explained" rel="noopener noreferrer"&gt;multi-model routing&lt;/a&gt; is for.&lt;/p&gt;

&lt;p&gt;The cheapness cuts the other way too, and it is worth saying plainly: when every check costs $0.0002, nothing in the invoice stops you from letting checks replace checkers everywhere. Which action tiers still end at a person is a policy decision now. The bill will never again make it for you.&lt;/p&gt;

&lt;h2&gt;
  
  
  The calibration row: a score you can audit vs. a vibe you cannot
&lt;/h2&gt;

&lt;p&gt;A dedicated decision model computes its probability over a closed answer set; that number can be checked. Pull your last 500 human-decided cases, run them through the layer, bucket by reported score, compare each bucket with the known outcomes. If the 0.90 bucket is right about 90% of the time, calibration holds on your traffic. If it is right 72% of the time, you learned that before wiring it to anything irreversible.&lt;/p&gt;

&lt;p&gt;Run the same loop monthly and you get drift detection almost for free, because the ledger already holds score and outcome side by side. This is the quiet, compounding advantage of Architecture B: every routed decision deposits a data point, and the pile of data points is the answer to next quarter's question, "is the automation actually as good as its scores say." Architecture A deposits nothing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fva43kopfxl1puypjdhej.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fva43kopfxl1puypjdhej.png" alt="when the dial reads high" width="800" height="448"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  When the judge is down
&lt;/h2&gt;

&lt;p&gt;A decision layer is a dependency, and dependencies fail: timeouts, rate limits, a 5xx on the one call that gates everything else. What your code does in that moment is a safety decision you are making right now, at design time, whether you notice or not.&lt;/p&gt;

&lt;p&gt;Fail open (treat an unanswered question as confident) silently removes the gate during exactly the windows when a vendor is having an incident, which correlate with the windows when everyone else's retries are flooding the same API. Fail closed (route to the queue) degrades to more human work and no silent sends. For any action tier worth gating, closed is the only defensible default:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;def judged(ticket, draft):
    try:
        d = decisions_api.ask(..., timeout_ms=800)
    except (Timeout, ApiError) as e:
        ledger.append({"ticket": ticket.id, "route": "queue",
                       "reason": f"judge_unavailable:{e.code}"})
        return queue(draft, reason="judge_unavailable")
    return route(d, draft)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two refinements earn their keep. A judge outage can flood the queue past what reviewers absorb, and a flooded queue rubber-stamps; pair the fallback with a circuit breaker that pauses non-urgent action classes outright when the queue depth crosses its cap. And write the outage into the ledger as shown, because a gap in the score column is itself a drift signal: a week of "judge_unavailable" entries explains a weird calibration bucket better than any post-hoc guesswork.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the inline sentence is still good for
&lt;/h2&gt;

&lt;p&gt;None of this makes the prompt-level instruction useless. It makes it a different tool.&lt;/p&gt;

&lt;p&gt;As a pre-filter, self-assessment is free recall: a generator that flags its own doubt can only add cases to the review pile, never remove them, so wiring "output ESCALATE if unsure" in addition to the decision layer is pure upside. As an annotation, it is context the number cannot carry: a decision layer returns 0.63, and the generator's own note ("flagging: the order number in this ticket does not appear in the account history") is what lets the human clear the queue item in thirty seconds instead of five minutes. And in tiers where the blast radius is zero, internal drafts, suggestions a human always edits, Architecture A alone is proportionate; adding a priced check in front of a throwaway draft is ceremony.&lt;/p&gt;

&lt;p&gt;The rule that falls out: let the generator talk about its doubts, let the decision layer measure them, and let only the measurement touch the routing. Trust the commentary as color, never as a gate.&lt;/p&gt;

&lt;h2&gt;
  
  
  A migration path that is not a rewrite
&lt;/h2&gt;

&lt;p&gt;Nobody green-lights "pause the agent while we rebuild its judgment." The rewiring above installs incrementally, and each step pays for itself before the next one starts.&lt;/p&gt;

&lt;p&gt;Week zero: add the ledger alone. Wrap existing sends with an append that records action, input pointer, and outcome when known. No scores yet, no behavior change, and you already gain the record you currently lack when something goes wrong.&lt;/p&gt;

&lt;p&gt;Weeks one and two: add the decision call in shadow. Every action gets scored and logged; humans keep deciding everything; latency cost is one parallel call. The shadow data yields the two curves every threshold argument needs, the score distribution and the human-disagreement rate per band, measured on your traffic rather than a vendor benchmark.&lt;/p&gt;

&lt;p&gt;Week three: automate one band of one action class, the highest-scoring slice of the cheapest, most reversible tier. Everything else keeps flowing to people. The ledger now accumulates measured error rates for the automated slice.&lt;/p&gt;

&lt;p&gt;Week four onward: loosen one band at a time, each loosening justified by the measured error rate of the last, never by the backlog. Re-run calibration monthly on the most recent 500 human-decided cases. Review THRESHOLDS like a firewall rule: named owner, dated changes, reasons in the log. Incident-time changes move in one direction only, toward more review; relaxation waits for the postmortem.&lt;/p&gt;

&lt;p&gt;A team that stops halfway through this list, ledger plus shadow scores and no automation at all, has still replaced "are we comfortable with the agent deciding" with a measured disagreement curve. That is most of the value, available before the first automated send.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fagqeph2jfaz2uym185f8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fagqeph2jfaz2uym185f8.png" alt="six weeks live" width="800" height="448"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What the decision layer does not fix
&lt;/h2&gt;

&lt;p&gt;Named limits, so nobody files this under silver bullets.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Calibration drift&lt;/strong&gt; survives the rewiring. A layer calibrated on last quarter's tickets walks off true when the product, the customers, or the input language changes. The ledger makes drift visible; it does not make it impossible. Google's calibration claims for gemini-decision-1 are measured on Google's benchmarks (vendor figures, 2026-11-11), never on your refund queue.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The accountability gap&lt;/strong&gt; survives it too. Routing a case to a human puts a human in the loop; it does not make them awake, informed, or responsible. A queue that dumps forty items an hour on someone with a day job degrades into rubber-stamping within weeks. Cap queue volume, track the disagreement rate, and treat a reviewer who never overrules the model as a broken alarm.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Vendor dependency&lt;/strong&gt; gets worse, by one API. A bought decision layer is an external dependency at the most sensitive point in your stack, the component that decides what your agent may do. The self-hosted version (an open-weight model with a constrained output head, or a small fine-tuned classifier) starts with worse scores and costs you the maintenance, and in exchange nobody can reprice, deprecate, or silently revise your judgment layer. Our rule of thumb: buy judgment for low-stakes tiers, keep the high-stakes gate on infrastructure whoever answers for the money controls. Teams writing this into contracts can lift the questions from a &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/sovereign-ai-procurement-standard-2026" rel="noopener noreferrer"&gt;procurement standard&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;And one whole class of decisions stays off the table at any score.&lt;/strong&gt; The same week Google priced judgment, United Airlines confirmed it will not let third-party AI agents complete ticket purchases (The Verge, 2026-11-11): agents can search flights, and the transaction ends at a person, partly because federal rules require the passenger to personally attest to declarations like the no-explosives question. No threshold crosses that line, because the meaning of an attestation is that a specific person made it. Signature-class actions belong above every threshold, routed to a person every time, at confidence 1.0 as much as at 0.3.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this lands in an agent stack
&lt;/h2&gt;

&lt;p&gt;Agents raise the stakes on all of the above because the output of the judgment is an action: send, purchase, delete, publish. One of the six criteria for a &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/what-is-sovereign-ai-avatar" rel="noopener noreferrer"&gt;sovereign AI avatar&lt;/a&gt; is that actions are judged outside the model, by rules the owner sets. Architecture B is that criterion, written in code: the generating model proposes, a separate layer scores, versioned config decides, a ledger remembers.&lt;/p&gt;

&lt;p&gt;Our own product, KarmaBox, is a platform and network for running your own AI avatars, and it ships this wiring as configuration rather than as a weekend project. Per-action policies (send, publish, spend, delete each get their own rules) are the THRESHOLDS object with an owner attached; approval cards are the queue() branch, pushing the ambiguous case to the person who answers for it; the action ledger is the append, kept in the owner's records rather than a vendor console. Where the judgment itself runs is a deployment choice: sensitive review flows can sit on machines you control, public-facing tasks can ride cloud avatars. The decision method, the tiers and the numbers, stays the team's own homework either way.&lt;/p&gt;

&lt;p&gt;If you wire up exactly one piece of this on Monday, make it the ledger. Thresholds you can argue about forever; the ledger is twenty lines, and it is the only component that turns this quarter's automation into next quarter's evidence. The second-best Monday candidate is shadow scoring, for the same reason: both produce evidence while changing nothing a customer sees.&lt;/p&gt;

&lt;p&gt;What is the one action your agent takes today that you could not reconstruct the reasoning for, six weeks later, if a customer asked?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Google, &lt;a href="https://blog.google/innovation-and-ai/technology/developers-tools/gemini-decisions-api/" rel="noopener noreferrer"&gt;Introducing the Gemini Decisions API&lt;/a&gt;, 2026-11-11&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;The Verge, &lt;a href="https://www.theverge.com/ai-artificial-intelligence/1031623/united-unauthorized-ai-agents" rel="noopener noreferrer"&gt;United won't let your AI agent book your flight&lt;/a&gt;, 2026-11-11&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>architecture</category>
      <category>python</category>
    </item>
    <item>
      <title>Your Agent's Share Link Is a Bearer Token, and Its SSH Socket Is You</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Wed, 11 Nov 2026 14:47:45 +0000</pubDate>
      <link>https://dev.to/karmabox/your-agents-share-link-is-a-bearer-token-and-its-ssh-socket-is-you-e96</link>
      <guid>https://dev.to/karmabox/your-agents-share-link-is-a-bearer-token-and-its-ssh-socket-is-you-e96</guid>
      <description>&lt;p&gt;Two security stories broke within a day of each other this week. They look different, but they describe the same problem.&lt;/p&gt;

&lt;p&gt;On November 10, Ars Technica reported research from security firm Claroc: Grok Build, the vibe-coding agent from SpaceXSI, has been leaking data from thousands of users through its share links. The URLs are randomized, so in theory only someone who knows the exact address can open an app. A design decision in the platform, however, makes it possible to enumerate thousands of live projects, and clicking through opens the app and, in all but rare cases, everything stored with it. The exposed haul in Claroc's sampling: internal business plans, ad proposals, medical exam results, galley checklists from a celebrity cruise line, employee names and ID numbers. The embarrassing detail: 24 of the leaking apps were built by SpaceXSI itself, including a confidential onboarding plan for its enterprise sales team, a sales territory map of 594 accounts, and internal system addresses.&lt;/p&gt;

&lt;p&gt;The day before, a post on Hacker News described a quieter exit. A user reported that a desktop AI coding setup had forwarded their local SSH agent to the remote execution environment, meaning agents running on machines they did not control could authenticate as them, including pushing to private repositories. The mechanism is not a rumor; the openai/codex repository has a merged pull request, #28131 from June 2026, implementing an explicit opt-in flag, &lt;code&gt;--forward-ssh-agent&lt;/code&gt;, that validates and refreshes the forwarded agent socket across reconnects. The capability is real, documented, and deliberate; the HN report is one user's account of being surprised by where their identity ended up.&lt;/p&gt;

&lt;p&gt;In the first story, data walks out through a link. In the second, your identity walks out through a socket. Neither exit required an exploit, both shipped as features, and both are present in ordinary developer setups today. This post is an audit of those two exits.&lt;/p&gt;

&lt;h2&gt;
  
  
  Exit one: why a random URL is not an access control
&lt;/h2&gt;

&lt;p&gt;The Grok Build failure is worth understanding precisely, because the pattern is everywhere, and because nothing about it required breaking cryptography.&lt;/p&gt;

&lt;p&gt;An unguessable URL is a bearer token: whoever holds the string gets the resource, no further questions. As a security primitive this is legitimate and widely used; the entropy of the random suffix does the work a password would do. The arithmetic only holds under one assumption: the token never appears anywhere an attacker can read.&lt;/p&gt;

&lt;p&gt;That assumption is what the design decision destroyed. If any surface of the platform lists projects, indexes them, exposes sequential identifiers, or lets a crawler walk from one share to another, the randomness of the suffix becomes decoration. Claroc did not guess URLs; per Ars' description, the platform made it possible to enumerate live projects. The platform had one job here: make sure nobody could pull a list of those links. That is the job that failed.&lt;/p&gt;

&lt;p&gt;A rule you can use in your own audit: a bearer URL is exactly as private as the most public place it is ever written down. Share links live in chat logs, in email bodies scanned by filters, in browser histories synced to clouds, in analytics referrer fields, in the platform's own index. Each of those is a place your "private" link is stored by someone who never promised you privacy. The link does not expire when the meeting ends or the demo finishes, and almost nobody goes back to revoke links, because almost nobody keeps a list of the links they have minted.&lt;/p&gt;

&lt;p&gt;Note the pattern, because it repeats: the cryptography was fine, the policy was fine, the leak lived in an adjacent design decision nobody audited as a security surface. The sharing feature was built by people solving "make collaboration easy." The enumeration surface was invisible to them because it was not in the threat model of a sharing feature.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs6l3m3j761bjy7uwj99j.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs6l3m3j761bjy7uwj99j.jpg" alt="a random URL" width="800" height="419"&gt;&lt;/a&gt;&lt;br&gt;
It is worth being precise about who was responsible for what. Making the suffix random enough was the vendor's job, and they did it. Keeping the project list from being enumerable was also the vendor's job, and they did not, and no user could have seen that until Claroc looked. The only thing users ever controlled was the decision of what to store behind such links in the first place. That is the whole case for auditing your own side of every boundary: the side you can see is the only side you can fix.&lt;/p&gt;
&lt;h2&gt;
  
  
  Exit two: the socket that signs as you
&lt;/h2&gt;

&lt;p&gt;The SSH agent story needs one paragraph of mechanism for anyone who has only ever used the agent without looking at it.&lt;/p&gt;

&lt;p&gt;Your SSH agent holds your private keys and answers signing requests over a Unix socket, advertised in &lt;code&gt;SSH_AUTH_SOCK&lt;/code&gt;. Agent forwarding extends that socket to a remote machine: processes there can ask your local agent to sign authentication challenges. The keys never leave your laptop, which is the detail that makes forwarding feel safe, and the signing capability absolutely leaves your laptop, which is the detail that matters. Anything on the remote end that can reach the socket can authenticate as you, to any system that trusts your key, for as long as the session lasts.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;local laptop                      remote execution env
┌───────────────┐                 ┌─────────────────────┐
│ ssh-agent     │◄── socket ─────►│ any process here    │
│ (your keys)   │   forwarded     │ can request a       │
└───────────────┘                 │ signature as YOU    │
                                  └─────────────────────┘
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The ssh documentation has warned about this for as long as the flag has existed: forward only to hosts you trust, because the socket is reachable by root on the remote end. The warning was written for a world where the remote end was a machine you picked and the session was your hands on a keyboard.&lt;/p&gt;

&lt;p&gt;Forwarding to a server you administer, for a session you watch, is a calculated risk ops people have taken for decades. The new ingredient is the autonomous occupant. An AI agent in that remote environment is a process that runs for hours, acts on its own schedule, executes code it just wrote, and ingests instructions from files and web pages it was asked to read. Prompt injection turns "can reach the socket" from a hypothetical into an attack path: a malicious README that convinces an agent to run &lt;code&gt;git push&lt;/code&gt; somewhere it should not is now operating with your identity.&lt;/p&gt;

&lt;p&gt;The codex PR deserves to be read fairly: &lt;code&gt;--forward-ssh-agent&lt;/code&gt; is opt-in, and the engineering in #28131 is about keeping the socket valid across reconnects, which is solid work for people who asked for the feature. The HN user's surprise is the actual finding. Forwarding your identity to an autonomous agent is a different risk from forwarding it to a session you drive by hand, and the flag's name does not warn you about that difference.&lt;/p&gt;

&lt;h2&gt;
  
  
  The common geometry
&lt;/h2&gt;

&lt;p&gt;Put the two exits side by side and the shared structure is visible:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqi9tql7y4869stlh563o.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqi9tql7y4869stlh563o.png" alt=" " width="799" height="489"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Both failures happened where two individually reasonable features met, and combining features is exactly what agent platforms do all day. Neither vendor shipped a vulnerability in the CVE sense; both shipped conveniences whose interaction with real usage produced an exit. That distinction matters for your audit, because scanners find vulnerabilities and nobody scans for conveniences. An agent is a thing that combines your tools, your credentials, your data, and a remote execution environment, automatically, on a loop. Every convenience the platform adds is a new connection between those pieces, and the leaks happen at the connections.&lt;/p&gt;

&lt;p&gt;The same week supplied a third data point from the other direction: Fireworks disclosed on November 9 that attackers accessed some customers' model weights, datasets, and billing information, per TechCrunch. Different exit, same lesson: whatever sits inside somebody's boundary is exposed to that boundary's failures, and the only variable you choose is whose boundary your valuables sit inside.&lt;/p&gt;

&lt;p&gt;The sovereign-AI framing gives this a name. Condition two of a sovereign AI avatar: the data an agent works with stays within the boundary its owner set. The useful move for a developer is to treat identity as data, the most sensitive record you own: when a document leaks, the damage is its contents; when your identity leaks, whoever holds it can act as you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where else the bearer pattern lives in your stack
&lt;/h2&gt;

&lt;p&gt;Grok Build is the example with a headline. The pattern ships in tools you already use, and the audit should walk all of them.&lt;/p&gt;

&lt;p&gt;Meeting recordings and their transcripts travel as "anyone with the link" pages. Cloud documents and notebooks default to the same mode the moment someone hits share under deadline. Agent platforms mint trace viewers and eval dashboards with tokenized URLs, and those traces contain your prompts, which contain your data. Object storage pre-signed URLs are the one well-behaved member of the family, because they carry an expiry by construction; they are built around a token that expires on a schedule, which is exactly what the rest of the family lacks.&lt;/p&gt;

&lt;p&gt;The well-behaved version of the pattern needs three properties, and you can demand them from any tool: expiry by default, revocation that actually works, and a ledger of minted links someone can review. A sharing feature with all three is an access control. A sharing feature with none of them is a publishing button with a delay.&lt;/p&gt;

&lt;h2&gt;
  
  
  The socket has a family too
&lt;/h2&gt;

&lt;p&gt;SSH agent forwarding is the member of its family that made the news this week. The family is ambient identity: credentials reachable from an execution environment without anyone handing them over per use.&lt;/p&gt;

&lt;p&gt;The cloud metadata endpoint, 169.254.169.254, answers with instance credentials to any process that asks from inside. A mounted docker.sock makes a container root on its host. A logged-in browser profile, driven by an agent that browses, is a wallet of session cookies. CLI config files hold OAuth refresh tokens that outlive any session. A kubeconfig in the home directory is a cluster. None of these was placed for the agent; all of them are reachable by it, because the agent runs where they live.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwuanfbcsnm6xc4iq7ejx.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwuanfbcsnm6xc4iq7ejx.jpg" alt="ambient identity" width="799" height="437"&gt;&lt;/a&gt;&lt;br&gt;
The audit question is uniform across the family: what identity-shaped things can a process in this environment reach without asking a human, and which of them would still be dangerous in the hands of a contractor who obeys instructions found in documents. Enumerate, then either remove, scope, or gate each one.&lt;/p&gt;
&lt;h2&gt;
  
  
  Why "the model is trustworthy" is the wrong variable
&lt;/h2&gt;

&lt;p&gt;One mechanism paragraph, because it explains why this audit targets the environment rather than the model.&lt;/p&gt;

&lt;p&gt;A language model has no privilege separation between instructions and data. Everything in context, the system prompt, your request, the README it just read, the web page it fetched, arrives as the same kind of token stream, and the model has no way to verify where an instruction came from. Vendors add mitigations, and the mitigations help, but the property is structural: anything the agent reads is a potential instruction. Which means the security boundary cannot be drawn around the model's intentions. It has to be drawn around what the environment lets any occupant do: what is reachable, what is signable, what can leave. That is the entire logic of auditing sockets and tokens instead of auditing prompts.&lt;/p&gt;
&lt;h2&gt;
  
  
  The audit: one afternoon, five commands
&lt;/h2&gt;

&lt;p&gt;Nothing below needs new tools. Budget an afternoon.&lt;/p&gt;

&lt;p&gt;Start with the bearer-token inventory. List every place your agent tooling mints share links: coding agents' app shares, notebook links, trace viewers, playground snapshots. For each, answer three questions: can anything enumerate it (try the obvious: list endpoints, sequential IDs, sitemap, search engine &lt;code&gt;site:&lt;/code&gt; query), does it expire, and can you revoke it. A link that is enumerable, immortal, and irrevocable is a public page with extra steps, and should carry nothing you would not put on one.&lt;/p&gt;

&lt;p&gt;Then the credential manifest, for every remote or containerized agent run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# in the agent's execution environment, not your laptop:
env | grep -iE 'token|key|secret|auth'
ls -la "$SSH_AUTH_SOCK" 2&amp;gt;/dev/null &amp;amp;&amp;amp; echo "agent socket PRESENT"
cat ~/.git-credentials 2&amp;gt;/dev/null
curl -s --max-time 1 http://169.254.169.254/ &amp;gt;/dev/null &amp;amp;&amp;amp; echo "cloud metadata REACHABLE"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything that prints is something the agent can use, which means something a prompt injection can use. Run it in every distinct environment your agents touch, laptop, container, remote sandbox, CI runner, because the manifest differs per environment and the dangerous one is usually the environment nobody considered an environment. The question for each line is never "do I trust the model"; it is "would I hand this to a contractor whose instructions include every file he reads."&lt;/p&gt;

&lt;p&gt;Then split the identities. The agent gets its own: a deploy key scoped to one repo, a short-lived token minted per task, an account whose blast radius you chose on purpose. The test for the scope is the morning-after question: if this credential appears on a paste site tonight, what is the total damage, and the answer should be a shrug about one repository rather than an incident call about your identity. Your personal key signs nothing you did not type. If the platform offers agent forwarding, keep it off by default; turn it on per session, and only when you can say exactly what the remote end will be signing.&lt;/p&gt;

&lt;p&gt;Then watch the egress once. Run a representative agent task and look at what actually leaves: network connections, uploads, pushes. One honest look recalibrates the mental model better than any policy document, and the look is cheap: a proxy in front of the sandbox, or even &lt;code&gt;lsof -i&lt;/code&gt; sampled during the run, is enough to learn whether your picture of the agent's traffic matches the agent's traffic.&lt;/p&gt;

&lt;p&gt;Last, write down the boundary you decided. Which data may ride in which direction, which identity acts where, who approves exceptions. One page. The Grok Build victims whose medical results and sales maps are now browsable did not decide a boundary; they inherited one from a sharing feature's defaults.&lt;/p&gt;

&lt;h2&gt;
  
  
  A worked example: the demo that outlived the demo
&lt;/h2&gt;

&lt;p&gt;Consider a four-person agency that built a client demo on an agent platform in March. The demo needed realistic data, so someone exported a slice of the client's order history into it, eleven thousand rows, because the demo looked better with real numbers. The share link went to three people at the client. The deal closed in April. Nobody thought about the demo again.&lt;/p&gt;

&lt;p&gt;The link is still live in November. It lives in three inboxes, one forwarded thread nobody tracked, a browser history synced across two laptops, and the platform's own project index, wherever that is readable from. The agency's security posture, meanwhile, is decent by any checklist: SSO everywhere, disks encrypted, dependencies scanned. None of that apparatus has the demo in scope, because the demo does not exist in any inventory. It was never deployed; it was only shared.&lt;/p&gt;

&lt;p&gt;The audit above catches this in the first pass: the inventory question "what links have we minted" surfaces the demo, the three-questions test flags it (enumerable: unknown; expires: no; revocable: nobody remembers how), and the boundary page afterward says client data rides into demos only as synthetic rows. Total cost of prevention: one afternoon in March. Cost of the alternative: whatever a leak of eleven thousand rows of someone else's order history turns out to cost, at a moment the client chooses, not the agency.&lt;/p&gt;

&lt;h2&gt;
  
  
  The default stack, if you only remember five settings
&lt;/h2&gt;

&lt;p&gt;For readers who want the end state without re-deriving it: links expire by default, and anything shared externally gets a calendar entry to revoke it. Agents authenticate as themselves, with per-task tokens, never as you. Forwarding of any ambient identity, SSH agent included, stays off until a specific session needs it. Egress from agent environments goes through an allowlist, so surprise destinations fail loudly. And outward actions, push, send, publish, pass an approval gate that emits a receipt.&lt;/p&gt;

&lt;p&gt;Every one of those settings adds friction, and each is the cheap kind: seconds per event, in exchange for closing exits that stay invisible until the headline is about you.&lt;/p&gt;

&lt;h2&gt;
  
  
  How KarmaBox draws this boundary
&lt;/h2&gt;

&lt;p&gt;KarmaBox is our product: a platform and network for running your own AI avatars, built so that condition two is enforced by architecture instead of discipline.&lt;/p&gt;

&lt;p&gt;Avatars run on the owner's deployment, so the execution environment and the data boundary are the same place; what an avatar reads and produces stays on the machine unless a policy says otherwise, which is the property the &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-agent-data-leakage" rel="noopener noreferrer"&gt;agent data leakage&lt;/a&gt; guide walks through surface by surface. Credentials follow the same rule: keys stay in the owner's custody per the &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-api-key-self-custody-checklist" rel="noopener noreferrer"&gt;API key self-custody checklist&lt;/a&gt;, and outward actions, pushing, sending, publishing, are policy-gated per operation and wait for the owner's approval, so an injected instruction meets a gate instead of a socket. Setup specifics are in the docs.&lt;/p&gt;

&lt;p&gt;The configuration split for this article's readers: work that touches private repos, customer data, or your own credentials runs on avatars inside your boundary; experiments with public code can use whatever is convenient.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the audit does not buy you
&lt;/h2&gt;

&lt;p&gt;The audit has limits of its own. It addresses exposure risk: fewer tokens in the wild, fewer sockets reachable, one page of decided policy. It does not address platform risk: Grok Build's users could have run this entire audit and still been burned, because the enumeration surface was on the vendor's side of the fence; the only defense was not storing sensitive data behind a sharing feature at all. It does not address judgment risk: an agent with a correctly scoped deploy key can still push bad code through the correct gate. And local execution narrows exits without closing the human one: whoever can read the records can still leak them, which no architecture fixes.&lt;/p&gt;

&lt;p&gt;Also, the honest cost: per-task tokens and approval gates are friction, and the afternoon of audit is real hours. The cost of skipping them is the one in this week's news: 594 accounts on a sales map, browsable by URL.&lt;/p&gt;

&lt;p&gt;One of the two exits in this post is probably open in your setup right now. Which one are you going to check first, the links you have minted or the socket you are forwarding?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://arstechnica.com/security/2026/11/grok-build-coding-agent-leaks-oodles-of-confidential-info-some-belonging-to-spacexsi/" rel="noopener noreferrer"&gt;Grok Build coding agent leaks oodles of confidential info, some belonging to SpaceXSI&lt;/a&gt;, Ars Technica, November 10, 2026.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://github.com/openai/codex/pull/28131" rel="noopener noreferrer"&gt;openai/codex on GitHub&lt;/a&gt;, merged June 2026, accessed November 11, 2026.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-agent-data-leakage" rel="noopener noreferrer"&gt;AI agent data leakage&lt;/a&gt;, Karma Box, 2026.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>security</category>
      <category>ssh</category>
      <category>devops</category>
    </item>
    <item>
      <title>Ember Is $0.10 per Million Input Tokens. The Wiring Is the Real Work</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Tue, 10 Nov 2026 15:11:53 +0000</pubDate>
      <link>https://dev.to/karmabox/ember-is-010-per-million-input-tokens-the-wiring-is-the-real-work-347p</link>
      <guid>https://dev.to/karmabox/ember-is-010-per-million-input-tokens-the-wiring-is-the-real-work-347p</guid>
      <description>&lt;p&gt;Nobody ships a release by hand. One engineer writes the build file. A farm of identical workers runs it ten thousand times a night. The engineer is expensive and runs rarely; the workers are cheap and run all the time. On November 9, 2026, Meta described its two newest models in the same shape: "Muse Wonder plans, Muse Ember handles the volume."&lt;/p&gt;

&lt;p&gt;We build agents this way, and if you do too, the vendor news from that Monday changes your parts list more than your architecture. What it does not change is the wiring, and the wiring is where the real decisions live. This post is a build guide for that wiring: how we write the rules the planner produces, how we route each task category, how we keep a local executor from silently falling back to the cloud, and how we accept a new executor before it touches production volume.&lt;/p&gt;

&lt;p&gt;A planner/executor split is an agent design in which a capable model plans the work and writes the rules for a kind of task, and a smaller, cheaper model executes those rules across many items. It is not a fallback chain where the small model steps in when the big one fails. The two models do different jobs on every run, and each can be chosen, priced and replaced on its own.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Meta and Anthropic shipped on November 9
&lt;/h2&gt;

&lt;p&gt;TechCrunch's Russell Brandom reported Meta's release of Muse Ember that day. According to the report, Meta calls Ember its lightest and most cost-efficient model so far, "designed for complex reasoning execution and the work agents repeat all day: tool calls, coding subtasks, routing, extraction, and long document parsing." Input price: 10 cents per million tokens. The output price was not in the coverage we reviewed, so every Ember figure in this post is the input half only.&lt;/p&gt;

&lt;p&gt;The report also sketches the pairing in practice. In a customer-support agent, a heavier model establishes rules for a category of issue, and a lighter model applies those rules "across tens of thousands of cases." Meta trained Ember to read screenshots, charts and scanned documents, and to work from voice memos, meeting recordings and video without a separate model.&lt;/p&gt;

&lt;p&gt;Anthropic shipped Claude Fable 5.2 and Claude Mythos 5.2 the same day. The announcement lists Fable at $8 per million input tokens and $40 per million output in standard mode, and gives it two roles that map onto the same split: "Advisor (where Haiku, Sonnet, or Opus models consult it to help solve harder problems)" and "Orchestrator (where it delegates simpler tasks to smaller models)." The same post tells users to start most coding and knowledge work on Claude Opus 5.5 and move "upward to Fable or Mythos, or downward to Claude Sonnet 5.5 or Claude Haiku 5.5" depending on the job.&lt;/p&gt;

&lt;p&gt;One pricing detail in that announcement is worth copying into your own evaluation sheets. Anthropic compares per task, not per token: on its coding benchmark, Fable 5.2 at High effort scored "about level with Opus 5.5 at Max effort, for roughly half the cost per task." The number that matters for a finished item depends on how many tokens the model burns getting there, which a price sheet alone will not tell you.&lt;/p&gt;

&lt;p&gt;So two vendors, on one Monday, told customers to stop running one model for everything. Neither said where the cheap half should live or who keeps the rules. That part lands on whoever builds the agent, which is the rest of this post.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which half of the split reads the sensitive data?
&lt;/h2&gt;

&lt;p&gt;Before any routing decision, look at what each half actually touches.&lt;/p&gt;

&lt;p&gt;The planner sees a sample. To write extraction rules for refund requests it needs the policy, a handful of example tickets, maybe the edge cases a person flagged last month. It runs a few times a day, or a few times a week when the policy changes.&lt;/p&gt;

&lt;p&gt;The executor sees the whole queue. A support desk handling 30,000 tickets a month means the executor reads all 30,000, plus whatever it pulls in to act on them: order history, the attached screenshot, the scanned invoice, the voice memo a field rep left. Meta's own list of Ember's jobs (extraction, long document parsing, tool calls) is a list of work that requires reading the underlying data in full. The multimodal pitch sharpens it: a model that takes call recordings and signed-contract photos is reading exactly the material a confidentiality clause is written about.&lt;/p&gt;

&lt;p&gt;So the two halves have opposite profiles. Planner: low volume, high judgment, narrow exposure. Executor: high volume, rule-following, broad exposure. When someone asks whether your agent "sends data to the cloud," the accurate answer depends on which half they mean. In most planner/executor builds, nearly all of the data flows through the executor.&lt;/p&gt;

&lt;p&gt;That is the question to hold onto while wiring: the agent you are building will read every invoice, ticket or contract your company handles. Where should the half that reads everything run?&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpead4lftnoe88stge388.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpead4lftnoe88stge388.jpg" alt="one model writes the rules" width="800" height="430"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Will a local executor pay for itself?
&lt;/h2&gt;

&lt;p&gt;Run the arithmetic before deciding, because the answer is unflattering to the local option.&lt;/p&gt;

&lt;p&gt;Take a document-extraction job: 30,000 items a month at about 4,000 input tokens each (our assumption; scanned pages and long threads run higher). That is 120 million input tokens a month. At Ember's reported $0.10 per million, the input side costs about $12 a month. The same input on Fable 5.2 at $8 per million is about $960. Both figures leave out output tokens, retries and caching, and Ember's output price is unverified, so treat them as the input half of the bill only.&lt;/p&gt;

&lt;p&gt;Twelve dollars a month is two coffees. A machine under the desk does not pay for itself on token savings at this scale, and a build guide that pretends otherwise is selling you something. The executor layer is now cheap to rent.&lt;/p&gt;

&lt;p&gt;Which moves the question. If renting the executor costs almost nothing, the reasons to run it on your own hardware have to come from somewhere other than the bill. Two survive contact with the arithmetic: where the data goes, and who keeps the rules.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the sovereign-avatar standard points
&lt;/h2&gt;

&lt;p&gt;Those two reasons have names in a framework we use for all of this. A &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/what-is-sovereign-ai-avatar" rel="noopener noreferrer"&gt;sovereign AI avatar&lt;/a&gt; is an AI agent that runs on compute its owner controls, keeps its data and memory in the owner's domain, works within permissions the owner sets, and keeps a memory that survives a change of model. Two of its criteria map directly onto the two halves of the split.&lt;/p&gt;

&lt;p&gt;Criterion one says inference runs on compute the owner controls. Applied to this design it points at the executor: the layer that reads everything, follows rules rather than improvising, and happens to be what small local-capable models are now built for. Put the executor on owned hardware and the bulk of the data stays in the building, while the planner, which sees a sample, can stay on the best hosted model you can afford.&lt;/p&gt;

&lt;p&gt;Criterion six says memory belongs to the owner and follows the avatar, whichever model is underneath. Applied to this design it points at the rules, and the rules deserve their own section.&lt;/p&gt;

&lt;p&gt;The planner still sees samples, so keep its view small. Three habits cover most of it. Replace names, amounts and identifiers in sample documents with placeholders that keep their format (a twelve-digit number stays twelve digits). Describe recurring edge cases in words, "some suppliers print the VAT number in the footer," instead of attaching the documents where they occur. Log every sample sent to the hosted planner, with the date and who approved it, so exposure is a list you can read instead of a guess.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we write a rules file an executor can follow
&lt;/h2&gt;

&lt;p&gt;In TechCrunch's support-desk example, the heavy model "establishes rules" and the light model applies them across tens of thousands of cases. We treat those rules as a file, in the same way the build farm treats its build file: the one artifact worth keeping when every machine around it gets swapped.&lt;/p&gt;

&lt;p&gt;The shape matters more than the syntax. Ours look roughly like this (illustrative, trimmed):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# rules/invoice_extraction.yaml -- versioned, reviewed like code
version: 14
approved_by: finance-lead
approved_on: 2026-11-07
fields:
  - name: supplier_name     # string
  - name: invoice_number    # string
  - name: invoice_date      # date, must not be in the future
  - name: net_amount        # money
  - name: vat_amount        # money
  - name: vat_number        # string; some suppliers print it in the footer
checks:
  - "net_amount + vat_amount == gross_amount"   # to the cent
  - "invoice_date &amp;lt;= today"
on_check_failure: route_to_human                # never guess
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A page or two covers a task category. Every line usually stands for a mistake someone caught once, which is why it is the piece of work a team least wants to redo.&lt;/p&gt;

&lt;p&gt;Three handling habits make the file an asset instead of a liability. Keep it as text the team can read, review and diff, one file per task category. Version it like code, so a planner revision shows up as a diff a person approves. Store it with the agent's memory, outside any single vendor account, so a new executor or a new planner reads the same file.&lt;/p&gt;

&lt;p&gt;One more column earns its place: where each rule came from. A comment like &lt;code&gt;# source: ticket 4812, 2026-09&lt;/code&gt; next to the footer-VAT rule tells the next maintainer the line is load-bearing and which case breaks if it goes. Planners propose rules, people approve them, and the file remembers why.&lt;/p&gt;

&lt;p&gt;Where the file lives decides how free you are later. Rules that exist only inside one vendor's orchestration product, generated by the planner and fed to the executor without ever becoming a file you hold, mean that swapping the executor regenerates them and swapping the planner means hoping for compatible output. Rules that live as plain versioned files next to the agent carry over unchanged through either swap. That is &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-agent-memory-ownership" rel="noopener noreferrer"&gt;agent memory ownership&lt;/a&gt; applied to one team: context that lives with the avatar survives a model change, and context that lives inside a tool leaves with the tool.&lt;/p&gt;

&lt;p&gt;The file also makes debugging honest. When a baseline run fails, you can tell whether the rule was wrong or the executor misread a correct rule, because the rule sits there in plain text. Inside a vendor's orchestration layer, those two failures look identical.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we route tasks without a silent cloud fallback
&lt;/h2&gt;

&lt;p&gt;Routing starts with task categories, not models. For each category, three facts decide placement: items per month, data sensitivity, and how open-ended the task is. The table below is illustrative; the rows are typical of a small professional-services firm.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Furn780vzh43mkp2w532c.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Furn780vzh43mkp2w532c.png" alt=" " width="799" height="637"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Before moving a category local, check throughput. The invoice row is 30,000 items a month, about 1,000 a day. At 20 seconds per invoice end to end (an assumption; measure on your own documents and hardware) that is about 5.6 hours of work a day, which fits overnight with room to spare. At 60 seconds it becomes about 17 hours a day, and the box has no headroom left for anything else. The acceptance run below gives you the real seconds-per-item figure, and that figure decides whether one machine is enough.&lt;/p&gt;

&lt;p&gt;The last column is the one frameworks get wrong by default. The classic failure in multi-model agents is the silent fallback: the local executor times out, the harness retries on a hosted model without telling anyone, and 30,000 sensitive documents leave the building over a weekend because a GPU driver crashed. If a category is local for data reasons, its failure policy is "queue and wait" or "hold for a person," never "try the cloud."&lt;/p&gt;

&lt;p&gt;Here is that policy as code. It is not tied to any product's API; it is the logic we want in whatever harness runs the agent:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;from dataclasses import dataclass

@dataclass
class Route:
    executor: str          # "local" or a hosted model name
    on_failure: str        # "queue", "hold", or "retry_hosted"

ROUTES = {
    "invoice_extraction":   Route("local", "queue"),
    "ticket_triage":        Route("local", "queue"),
    "contract_compare":     Route("local", "hold"),
    "web_research":         Route("hosted-mid", "retry_hosted"),
}

class ExecutorUnavailable(Exception):
    pass

def run(task_category, item, executors, queue, review):
    route = ROUTES[task_category]
    try:
        return executors[route.executor](item)
    except ExecutorUnavailable:
        if route.on_failure == "queue":
            queue.put((task_category, item))      # data stays put
            return None
        if route.on_failure == "hold":
            review.put((task_category, item))     # a person decides
            return None
        if route.on_failure == "retry_hosted":
            return executors["hosted-backup"](item)
        raise
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details are deliberate. The failure policy hangs on the task category, not the model, so changing a category's executor never changes what happens when it breaks. And there is no default branch that sends work to a hosted model; a category opts into that explicitly, in a diff someone reviews.&lt;/p&gt;

&lt;p&gt;A queue policy needs one more piece to be safe: visibility. Work that waits silently is work that disappears. Whatever runs the queue should expose its depth, and someone should get a message when the oldest item passes an age threshold, say four hours during business hours. The failure this prevents is mundane. The box went down on Friday, the queue ate the weekend's invoices, and nobody noticed until a supplier called about a late payment. An alert on queue age turns that into a Monday-morning task instead of an apology.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2dx2gszm67ant4xmr4ib.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2dx2gszm67ant4xmr4ib.jpg" alt=" " width="800" height="410"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How we accept an executor before it goes live
&lt;/h2&gt;

&lt;p&gt;A cheap executor is only cheap when it gets the job right, and benchmark scores say little about whether a model pulls the right VAT number from your suppliers' scanned invoices. Each category gets its own acceptance run against a baseline, and the baseline is 50 recent items with human-checked outputs.&lt;/p&gt;

&lt;p&gt;The run itself is mechanical. In pseudocode, because the harness varies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# 50 recent items with human-checked outputs live in golden/
run-agent --rules rules/invoice_extraction.yaml \
          --input golden/items/ --output candidate/
compare-fields golden/expected/ candidate/ \
          --report acceptance-2026-11.md   # match rate, tool calls, retries, cost/item
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Compare field by field: for extraction, the share of fields that match exactly; for triage, the share of tickets landing in the same queue; for comparisons, the share of clauses a reviewer agrees with. Count tool calls and retries per item, because a model that loops costs more than its price suggests. Measure cost per item on the same set, with token assumptions written down.&lt;/p&gt;

&lt;p&gt;The composition of those 50 items decides what the test can see. Pull them from the live queue and keep the mix honest: clean typed invoices, bad scans, the supplier who prints the VAT number in the footer. Refresh a third of the set every quarter so it tracks what the queue looks like now. Items the executor once failed on stay in the set permanently; they are the regression suite. The report from each run records four numbers per category, match rate, tool calls per item, retries per item, cost per item, along with the model version and the rules file version the run used. Having both versions on one line is what keeps the next argument short.&lt;/p&gt;

&lt;p&gt;Set the bar before the run. "No new errors on items the current executor gets right" is our default for extraction; drafting can be looser. Write the result and the date next to the routing table, so that six months on, anyone can see why invoices run where they run.&lt;/p&gt;

&lt;p&gt;When the local executor misses the bar, two options are defensible. Keep the category on a hosted executor and accept where the data goes, in writing. Or split the category again: the local model handles the items it passes (clean, typed invoices), and only items failing a check go to a person, or to a hosted model with per-batch approval. That keeps most of the volume local and makes the exceptions explicit. Lowering the bar until the local model passes, without saying so, is the one move that costs you the whole point of the test.&lt;/p&gt;

&lt;p&gt;Rerun the acceptance whenever either model changes. A new planner writes rules that read differently; a new executor follows the same rules differently. The baseline makes both visible.&lt;/p&gt;

&lt;h2&gt;
  
  
  When the hosted executor is the better call
&lt;/h2&gt;

&lt;p&gt;A local executor wins when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;the category reads client data, personal data or anything under a confidentiality clause&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;volume is steady and high, so the machine stays busy&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the task is well specified and a smaller model passes the baseline&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;work has to continue through an internet outage&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the failure policy you want is "wait," never "send it somewhere else"&lt;br&gt;
A hosted executor wins when:&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the data is public or already published&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;volume is low or spiky, so a machine would sit idle&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the task needs a capability local models lack, such as reading poor scans reliably, and the hosted model passes where the local one fails&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;at prices like Ember's input rate, the monthly bill is smaller than the time cost of maintaining a machine&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;nobody on the team wants to own patches and uptime for another box&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What this wiring fixes, and what it leaves open
&lt;/h2&gt;

&lt;p&gt;The split plus an owned rules file addresses two risks. Data concentration risk drops, because the layer that reads everything can sit on hardware the owner controls while only redacted samples reach the hosted planner. Executor lock-in drops, because a rules file the team holds lets either model be replaced without starting over.&lt;/p&gt;

&lt;p&gt;Everything else stays where it was. Planner quality risk remains: a bad rule from the hosted planner gets applied 30,000 times by a faithful executor, so rule changes need human review. The capability gap remains: on the hardest inputs, hosted frontier models still beat anything that fits in an office machine, and the acceptance run is how you find out which categories that affects. Running your own executor adds the operational work a hosted API hides, patches and disks and locked-down remote access. And redacting samples before they reach the planner is a separate job this design does not do for you.&lt;/p&gt;

&lt;h2&gt;
  
  
  How KarmaBox runs the split
&lt;/h2&gt;

&lt;p&gt;Our own product, &lt;a href="https://www.karmaboxai.com/en/concept/factory/product" rel="noopener noreferrer"&gt;KarmaBox&lt;/a&gt;, is built around this split. It is a device with 128 GB of unified memory that sits in the owner's office and hosts a team of avatars, and it maps onto the guide above in four places.&lt;/p&gt;

&lt;p&gt;Models route per task: routine, high-volume work goes to local models on the device, where it carries no per-token cost, and harder tasks go to a hosted frontier model. That is the routing table expressed as a per-avatar choice, and &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/multi-model-routing-explained" rel="noopener noreferrer"&gt;multi-model routing&lt;/a&gt; is the mechanism behind it.&lt;/p&gt;

&lt;p&gt;Device mode is the no-silent-fallback policy as a setting: every model and tool call goes to the on-device model, and if that model is unavailable the call fails instead of falling back to the cloud. An owner who wants invoice extraction to stay local sets that avatar to device mode and lets the research avatar use cloud models.&lt;/p&gt;

&lt;p&gt;Memory follows the avatar, so a rules file kept with the avatar survives a change of either model, across cloud models, local models and the owner's own GPU. On the hardest problems a hosted frontier model is still ahead, which is why the planner side often stays in the cloud.&lt;/p&gt;

&lt;p&gt;If you wire your own agents this way, which task category moves to a local executor first, and what is its failure policy?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;TechCrunch, Russell Brandom, "&lt;a href="https://techcrunch.com/2026/11/09/meta-releases-muse-ember-a-new-lightweight-model-for-agent-execution/" rel="noopener noreferrer"&gt;Meta releases Muse Ember, a new lightweight model for agent execution&lt;/a&gt;," November 9, 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Anthropic, "&lt;a href="https://www.anthropic.com/claude-fable-and-mythos-5-2" rel="noopener noreferrer"&gt;Introducing Claude Fable 5.2 and Claude Mythos 5.2&lt;/a&gt;," November 9, 2026. &lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>architecture</category>
      <category>agents</category>
    </item>
    <item>
      <title>284B on 64GB RAM: We Read the Small Print Before Downloading 56 GiB</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Mon, 09 Nov 2026 11:16:10 +0000</pubDate>
      <link>https://dev.to/karmabox/284b-on-64gb-ram-we-read-the-small-print-before-downloading-56-gib-37ng</link>
      <guid>https://dev.to/karmabox/284b-on-64gb-ram-we-read-the-small-print-before-downloading-56-gib-37ng</guid>
      <description>&lt;p&gt;Everyone who has bought a car knows how to read a fuel-economy sticker. The number is real. It was also measured on a test cycle that isn't your commute, and the small print tells you which cycle. A benchmark line in a GitHub README works the same way. On Sunday, 8 November, a repository called llama-modchip reached the Hacker News front page with this sticker: "284B DeepSeek on 64GB RAM, no GPU." We spent Monday morning on the small print.&lt;/p&gt;

&lt;p&gt;The model in question is a mixture-of-experts model. A mixture-of-experts (MoE) model is a network whose feed-forward weights are split into many separate blocks called experts, and it doesn't run all of them for each token; a small router picks a few experts per layer, token by token. So two numbers describe it: total parameters (everything in the file) and active parameters (the share that does work on any one token). For DeepSeek-V4-Flash the README gives 284.33B total and 13.17B active. Fitting a 284B MoE into 64GB is a problem about the first number. How fast it runs is mostly about the second.&lt;/p&gt;

&lt;h2&gt;
  
  
  The sticker, line by line
&lt;/h2&gt;

&lt;p&gt;The repo describes itself as "a set of ggml/llama.cpp kernel patches, quantization transcoders, and runtime tools". In plain terms it's a modified build of llama.cpp (the open-source program most people use to run models locally) plus tools that re-pack model files into smaller formats. The headline claims below are from the README at v2.4.0, commit a9635a5 (pushed 8 November 2026, 21:01 UTC); the same commit's ARCHITECTURE.md, FAQ.md, RELEASE_NOTES.md, DeepSeek deep-dive in docs/ and benchmark JSON files supply the detail further down:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;the DeepSeek-V4-Flash checkpoint shrinks from 76.87 GiB to 56.56 GiB, which is 1.71 bits per weight&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;peak memory at a 256K-token context is 57.61 GiB in one place and 58.9 GiB in another&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;on an AMD Ryzen 9 9955HX (Zen 5, 16 cores) with 64 GB and no GPU, prefill runs at 44.4 tok/s and decode at 9.1–14.7 tok/s (README self-reported, not re-tested by us)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;a 30B Nemotron model packed to 14.35 GiB runs at 25–60.4 tok/s on the same CPU (README self-reported, not re-tested by us)&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the files use a custom tensor type 40, which stock llama.cpp, Ollama and LM Studio "will reject"&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two terms before the arithmetic. A token is the unit a model reads and writes, roughly three-quarters of an English word. Prefill is the phase where the model reads your prompt; decode is the phase where it writes the answer one token at a time. They hit different limits in the hardware, which is why the README lists them separately.&lt;/p&gt;

&lt;p&gt;The repo was created on 29 October, so it's 11 days old, and it had 534 stars when we checked the API this morning. Its history shows the README changing several times on Sunday alone, including a commit that added "an explicit quantization bpw breakdown table" and removed "emoji clutter". The HN thread had 382 points and 168 comments when we pulled it. One of the first comments, from akie, is the question this whole post is about: "At which quantization? That's the only thing I want to know."&lt;/p&gt;

&lt;h2&gt;
  
  
  Total parameters decide the RAM, active parameters decide the speed
&lt;/h2&gt;

&lt;p&gt;An MoE router can choose any expert on the next token. You can't know in advance which 13B of the 284B you'll need, so all 284B have to be somewhere fast. On a GPU box that means video memory. On this setup it means system RAM. If part of the model sits on an SSD instead, every token that routes to a cold expert waits on a disk read, and the README's phrase for that is "page-cache thrashing".&lt;/p&gt;

&lt;p&gt;Speed is the other number. During decode, each new token reads the active weights from RAM once. The profiling table in docs/DEEPSEEK_V4_OPTIMIZATIONS.md says that's 3.20 GB per token before the repo's pruning options and 2.85 GB with them. You can sanity-check that from outside: 13.17B active parameters at 1.71 bits each is about 2.8 GB, and the parts kept at higher precision would push it up a bit. The numbers agree.&lt;/p&gt;

&lt;p&gt;That also gives a ceiling. The FAQ puts dual-channel DDR5 at about 75 GB/s. Divide 75 GB/s by 3.20 GB per token and you get about 23 tokens per second, the most this machine could decode if it did nothing but stream weights. The claimed decode with nothing pruned, 9.06–9.95 tok/s (README self-reported, not re-tested by us), would use roughly 29 GB/s of that. So the claim sits well under the physical limit. That doesn't prove it. It does mean nothing here breaks physics, which is worth knowing when a README reads the way this one does.&lt;/p&gt;

&lt;h2&gt;
  
  
  1.71 bits per weight, and where the 20 GiB went
&lt;/h2&gt;

&lt;p&gt;The bits-per-weight figure is plain division. 56.56 GiB is 60.73 GB, or 485.8 billion bits; spread across 284.33 billion parameters, that's 1.709 bits each. It adds up. The same division on the 76.87 GiB starting file gives 2.32 bits per weight.&lt;/p&gt;

&lt;p&gt;For scale, a normal 16-bit model stores 16 bits per weight, and most people who run models locally treat 4 bits as the point below which quality starts to fall off. One commenter, armcat, cited a paper saying the "peak size-to-performance ratio occurs at 4-bit". (We haven't read that paper; the summary is theirs.) And 1.71 is an average over very different tensors, which the README's breakdown table spells out:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;embeddings, the first dense layer, the 43 shared experts and some attention projections stay at Q4_K, Q5_K or F32&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the attention projections that were 8-bit get re-packed into the repo's "Fractal-64" format at 3.75 bits&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the 256 routed experts per layer, which hold most of the weight, go mostly to IQ1_S at 1.5625 bits, with some "sensitive boundary layers" kept at IQ2_XXS (2.06 bits) or IQ1_M (1.75 bits)&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;ARCHITECTURE.md names those layers (0, 1, 26, 28, 41 and 42 in one set) and explains how the 1.56-bit experts are packed: an importance score computed from the weights themselves, without calibration data. It's a reasonable idea. It's also the part a reader can't verify by reading, because whether a 1.56-bit expert still knows what it knew is an empirical question.&lt;/p&gt;

&lt;p&gt;One detail doesn't line up. The README says the starting point is "Unsloth UD-IQ1_S (76.87 GiB / 82.5 GB)". ARCHITECTURE.md describes the same 76.87 GiB file as "IQ2_XXS". HN user adrian17 spotted this first and later wrote: "only now noticed that 76.87GiB is 82.5GB, which happens to be size of the unsloth IQ1_S." Since the sizes match, the base is most likely the IQ1_S build. It's still the author's job to say so, and right now the two documents disagree.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why 76.87 GiB doesn't fit in 64 GB
&lt;/h2&gt;

&lt;p&gt;A "64 GB" machine has 64 GiB of RAM installed, and you never get all of it. The kernel, firmware reservations, the desktop and anything else running take a share. The README uses 62.5 GiB as the usable figure on a 64 GB Linux system. On that budget:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fky03m0omj28b63eljxa1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fky03m0omj28b63eljxa1.png" alt=" " width="800" height="263"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Sizes and the 62.5 GiB figure come from the README; bits per weight and room left are our arithmetic.&lt;/p&gt;

&lt;p&gt;The bigger file wouldn't simply refuse to start. llama.cpp maps model files into memory, so it would load, and then the operating system would keep evicting pages and reading them back from disk. The README's own table lists the unpatched base at 5.91 tok/s decode, with "OOM / Swap on 64 GB" in the prefill column (README self-reported, not re-tested by us). Whatever that run looked like, nobody wants to do agent work inside it.&lt;/p&gt;

&lt;p&gt;3.6 GiB of slack at peak is thin. A browser with a few tabs open will eat that. If you try this on a 64 GB box, run it headless, or plan on meeting the Linux out-of-memory killer.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd6w9cqhrjs7umghs6ks8.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fd6w9cqhrjs7umghs6ks8.jpg" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  How a 256K context fits in about 2 GiB
&lt;/h2&gt;

&lt;p&gt;This was the number that surprised us most. The KV cache is the model's working memory for a conversation: for every token it has read, each layer keeps some vectors so it doesn't have to recompute them. It grows with context length, and on many models a 256K-token cache is bigger than the weights.&lt;/p&gt;

&lt;p&gt;Here the gap between file size and peak memory is 58.9 minus 56.56, or 2.34 GiB, and that covers the cache plus compute buffers. So we worked out what a plain cache would cost. DeepSeek's attention stores one compressed "latent" vector per token per layer instead of full keys and values; the deep-dive gives it 512 dimensions, plus 64 for position. At 16 bits, across 43 layers and 262,144 tokens, that comes to about 11–12 GiB (our arithmetic, assuming one 512- or 576-wide FP16 vector per layer per token). It wouldn't fit.&lt;/p&gt;

&lt;p&gt;The answer is in upstream llama.cpp. The deepseek4 model code at the commit the repo builds on keeps recent tokens in a sliding window and stores older history in compressed caches, at ratios of 4:1 (DSV4_CSA_RATIO = 4) and 128:1 (DSV4_HCA_RATIO = 128), chosen per layer from the model file's metadata. An 11 GiB cache compressed 4:1 is about 2.7 GiB, and at 128:1 it's under 0.1 GiB, so a mix of the two lands near what the README reports. The code doesn't show which layers use which ratio. That's stored in the GGUF header, and we didn't download the file.&lt;/p&gt;

&lt;p&gt;Two practical consequences follow. Credit for the small cache goes to DeepSeek's design and the upstream implementation; the patch works around the cache (prefetching, and a fix for rolling it back after speculative decoding). And "256K context" here means the model can hold that much history in compressed form. It doesn't tell you how well the model pulls an exact detail from token 3,000 when it's at token 250,000. That's a quality question, and we found no long-context test for this model in the repo.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why stock llama.cpp can't read the file
&lt;/h2&gt;

&lt;p&gt;The README's compatibility note says Fractal-64 tensors are "GGML_TYPE_Q2_0, type ID 40" and that unpatched llama.cpp, Ollama or LM Studio "will reject them with unknown tensor type 40". GGUF files label every tensor with a numeric type ID, and a loader that doesn't know an ID refuses the file, so the story is plausible. We went looking for the definition.&lt;/p&gt;

&lt;p&gt;The repo's build script pins upstream llama.cpp to commit d2f8305 (10 August 2026). Here's ggml/include/ggml.h at that commit, pulled through GitHub's API:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;        GGML_TYPE_MXFP4   = 39, // MXFP4 (1 block)
        GGML_TYPE_NVFP4   = 40, // NVFP4 (4 blocks, E4M3 scale)
        GGML_TYPE_Q1_0    = 41,
        GGML_TYPE_Q2_0    = 42,
        GGML_TYPE_COUNT   = 43,
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Upstream already has a type called Q2_0, its ID is 42, and 40 belongs to NVFP4. Current master has the same values for those lines. The patch doesn't touch ggml.h at all. What it changes is the block layout in ggml-common.h:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; #define QK2_0 64
 typedef struct {
-    ggml_half d;              // delta (scale)
-    uint8_t qs[QK2_0 / 4];   // 2 bits per element
+    ggml_half d;              // master FP16 super-block scale (2 bytes)
+    uint8_t scales[4];        // 8 x 4-bit shifted sub-scales ...
+    uint8_t qs[16];           // lower 2 bits for 64 3-bit fractal indices
+    uint8_t qh[8];            // upper 1 bit for 64 3-bit fractal indices
 } block_q2_0;
-static_assert(sizeof(block_q2_0) == sizeof(ggml_half) + QK2_0 / 4, "wrong q2_0 block size/padding");
+static_assert(sizeof(block_q2_0) == 30, "wrong q2_0 block size/padding");
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Upstream Q2_0 is 18 bytes per 64 weights, or 2.25 bits. The patched Q2_0 keeps the name and becomes 30 bytes per 64 weights, or 3.75 bits, with 3-bit indices. That's why a stock build can't read these files correctly: both builds know a type called Q2_0, and the bytes behind it differ. The struct comment also says FP16 scale, while the README calls it a BF16 scale.&lt;/p&gt;

&lt;p&gt;What we couldn't reconcile is the number 40. In a build made from this patch on that commit, Q2_0 should still be 42. Maybe the files were written by an earlier fork with different numbering, or maybe the note is out of date. A stock binary won't load the model properly either way, so the practical advice stands. But if you hit an error that mentions 42, or a tensor size mismatch, instead of 40, this is probably why. There's a second effect worth knowing, as far as we can tell from the diff: since the patch redefines an existing upstream type in place instead of adding a new ID, a patched build would misread genuine upstream Q2_0 files. Keep that in mind before pointing it at other models in your folder.&lt;/p&gt;

&lt;p&gt;HN user ar-nelson tried the whole path on a Haswell-era Xeon and hit two different failures. The repo's launcher stopped at "check_tensor_dims: tensor 'blk.0.ffn_gate_tid2eid.weight' has wrong shape", and the bundled binary at "Illegal instruction (core dumped)", which they traced to the prebuilt binaries needing AVX-512.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one script we ran
&lt;/h2&gt;

&lt;p&gt;The README offers a "Hardware Readiness Check (Zero Model Download Required)" as a one-liner that downloads the script with curl and pipes it straight into python3. We didn't, because piping unread code into an interpreter is how you find out what it does after it has done it. Our first attempt to get the repo, a plain git clone, timed out on port 443 from our network. GitHub's API and tarball endpoint worked, which is how we ended up with the tarball at commit a9635a5, and scripts/check_cpu_readiness.py was the first file in it we read.&lt;/p&gt;

&lt;p&gt;It's 590 lines, standard library only. It reads CPU and memory facts (sysctl on macOS, /proc and lscpu on Linux), runs a short memory-bandwidth test by allocating up to 16 buffers of 128 MB and scanning them, and prints estimates. It makes no network calls and writes no files, and the shell commands it runs are fixed strings. That was enough for us to run it, in an empty directory, with Python's isolated mode so it couldn't import anything planted next to it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;mkdir -p /tmp/d09/work/devto_run &amp;amp;&amp;amp; cd /tmp/d09/work/devto_run
S=/tmp/d09/work/devto_repo/src/llama-modchip-a9635a566f/scripts/check_cpu_readiness.py
shasum -a 256 "$S"     # df09c236cc697ecb...c61d073aadae49c81227d9061ecfc40f1497
for i in 1 2 3; do python3 -I "$S" --json &amp;gt; run$i.json; done
python3 -I "$S"        # the human-readable report
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The machine was an Apple M4 Mac mini with 24 GB, 4 performance and 6 efficiency cores, macOS 15.6.1 and Python 3.9.6. We ran it on Monday 9 November at 11:33 China Standard Time. Each run took about 0.4 seconds.&lt;/p&gt;

&lt;p&gt;The four runs measured memory bandwidth at 81.2, 98.6, 95.1 and 96.0 GB/s, and estimated "Technical Prose + New Code" decode at 20.6, 24.6, 23.8 and 24.0 tok/s. All four gave the same verdict: Tier A, "production-ready interactive coding agent".&lt;/p&gt;

&lt;p&gt;Reading the code explains all of it. The script is an estimator for the 30B Nemotron model, and its banner says so. It says nothing about the 284B DeepSeek build, which needs 56.56 GiB, on a machine that has 24. The only thing it measures is bandwidth. Everything else comes from a formula: roughly, a reference speed from the author's Zen 5 machine, scaled by measured bandwidth over 48 GB/s and some fixed factors for SIMD support and cache size. So when the bandwidth reading moved 21% between run 1 and run 2, the speed estimate moved 19%, and the verdict didn't move at all.&lt;/p&gt;

&lt;p&gt;A few smaller things showed up on macOS. The script sets available RAM to 75% of the total by fiat (18.0 of 24.0 GiB). It reports /dev/shm as 0.0 GiB, because macOS has no /dev/shm, and still estimates a 9.5 ms restore from /dev/shm. Its recommended launch line uses taskset, a Linux tool, with &lt;code&gt;-t 10&lt;/code&gt;, which counts the Mac mini's six efficiency cores as full threads.&lt;/p&gt;

&lt;p&gt;None of that is sinister. It's an estimator that prints its output as a verdict. ar-nelson's Xeon also got Tier A before the binaries refused to run on it, which is the same lesson from the other side.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo5oywicqe1d3ecnnhfvw.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo5oywicqe1d3ecnnhfvw.jpg" alt="one script, one mac, four runs:same verdict, moving estimates" width="800" height="430"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What the speed numbers were measured at
&lt;/h2&gt;

&lt;p&gt;This is the sticker's test cycle. We didn't download the 56.56 GiB file or run inference, having no 64 GB x86 machine to run it on, so every speed figure in this section is README or docs self-reported and not re-tested by us.&lt;/p&gt;

&lt;p&gt;The decode range of 9.1–14.7 tok/s mixes two modes. Plain decode with all six routed experts per layer is 9.06–9.95 tok/s. The 11.5–14.7 range needs "Stockfish-PVS", the repo's speculative drafter, which guesses several tokens ahead and has the big model check them in one batch. Speculative decoding speeds up text the drafter can predict, like repeated code or JSON, and helps less on fresh prose. A fair one-line summary would be "about 9–10 tok/s, more on repetitive output".&lt;/p&gt;

&lt;p&gt;There are faster modes, Top-5 and Top-4, at 9.68–10.74 tok/s, which run fewer experts per token. The README is upfront that these change the output, and reports a KL divergence (a measure of how far the output probabilities drift) of 0.0042 to 0.0694 nats against the default. The readiness script's recommended launch line exports &lt;code&gt;LLAMA_MOE_TOP_K=4&lt;/code&gt; for the 30B model, so following its advice puts you in a pruned mode by default.&lt;/p&gt;

&lt;p&gt;The prefill figure of 44.4 tok/s comes from a table where prompts ran from 63 to 580 tokens, plus six agent turns that grew the cache to 2,418 tokens. Prefill fell from 44.36 to 38.44 tok/s across those six turns. None of the speed tables runs at 256K. The 256K number is a memory figure: the size of the process when started with &lt;code&gt;-c 262144&lt;/code&gt;. The README's own quickstart command for the 284B model uses &lt;code&gt;-c 32768&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Here's what those rates mean in wall-clock time, assuming they held at depth (the trend says they'd drop):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;reading a full 256K-token prompt at 44.36 tok/s takes 262,144 ÷ 44.36 ≈ 5,900 seconds, about 98 minutes; at 38.44 tok/s it's about 114 minutes&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;a 500-token answer at 9.06 tok/s takes about 55 seconds&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;the repo's slot restore, which reloads a saved cache in 13.1–32.6 ms, is the way around the first item for agent work: you pay for the long read once, save it, and reload it&lt;br&gt;
Put together, it looks like this. It's a 284B model that writes at roughly human reading speed, after a prompt read that's fine for a few thousand tokens and slow for a long document. For a background job that may be perfectly acceptable. The repo also ships ./modchip claude, which points Claude Code at the local server, and for an interactive coding agent that re-reads large chunks of a repository, the prefill is the number to plan around.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What to check before trusting the numbers
&lt;/h2&gt;

&lt;p&gt;Most of this is free, and the cheap checks come first.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsg5n2bs7lnv2nl1k5ndg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsg5n2bs7lnv2nl1k5ndg.png" alt=" " width="800" height="531"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The first row is the one to settle before anything else. The README's "bit-exact" and "1.00000 logit cosine similarity" compare the patched runtime with the unpatched runtime "on the same checkpoint". That shows the fast kernels compute the same thing as the slow ones, which is good engineering. It says nothing about how much the 1.71-bit file lost compared with the model it came from. The repo's 100-point quality gate results (benchmarks/results/*.json) are all for the 30B Nemotron model. Even there, 20 of the 100 points are for speed and RAM, so "100/100" means 80 quality points plus a full speed score, earned on five tool calls, three coding tasks and two needle tests.&lt;/p&gt;

&lt;p&gt;With a 64 GB machine and a 128 GB one side by side for a day, the plan would look like this. It uses standard llama.cpp tools. We haven't run it, so check the flags against your llama.cpp version.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# 128 GB machine, stock llama.cpp: save reference logits from the 76.87 GiB base
llama-perplexity -m DeepSeek-V4-Flash-0731-UD-IQ1_S.gguf -f wiki.test.raw \
  --kl-divergence-base base_logits.bin

# Same machine, patched binary: score the 56.56 GiB file against those logits
llama-perplexity -m DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf \
  --kl-divergence-base base_logits.bin --kl-divergence

# 64 GB machine, patched binary, drafting off: speed at depth plus peak memory
/usr/bin/time -v llama-bench -m DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf \
  -p 512 -n 128 -d 0,32768
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That KL test measures what the transcoding lost relative to the IQ1_S base. It won't tell you what the base had already lost relative to the full model; a full-precision reference for a 284B model needs far more memory than either machine has. Two smaller habits round it out. Before loading any GGUF from a fork, list its tensor types with the gguf Python package's gguf-dump tool. And read a readiness script before you run it; this one took us about ten minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  What reading a repo settles, and what it doesn't
&lt;/h2&gt;

&lt;p&gt;Reading the repo deals with claim-scope risk: whether a number means what the headline makes you think it means. On that the result is mixed and specific. The sizes add up. The bits-per-weight figure adds up. The speed claims sit under the physical ceiling. The labels "lossless", "256K" and "9.1–14.7" each hold under narrower conditions than they suggest.&lt;/p&gt;

&lt;p&gt;It doesn't deal with quality risk, whether a 1.71-bit DeepSeek-V4-Flash gives answers you'd accept. As far as we could find, nobody has published that test for this file, and the HN thread is split. walrus01, who said they had spent a lot of time with this model, wrote: "Even the 97GB Q2XS will show very significant divergence from the full precision thing." sparse-Matrix reported "Steady 12 to 14 TPS" on a Ryzen 7 5700U with 64 GB, without saying which model, and the 5700U row on the leaderboard is for the 30B one.&lt;/p&gt;

&lt;p&gt;It doesn't deal with build risk either. ar-nelson's failures, the AVX-512 requirement in the prebuilt binaries and the type-ID mismatch are things you only find by trying. aappleby, who did try the 30B model on a Ryzen 7945HX with 96 GB, wrote that "it runs just about the same as plain old llama.cpp on the cpu (except prefill seems much faster)", and found no DeepSeek option in the setup.&lt;/p&gt;

&lt;p&gt;And it doesn't deal with capacity risk. 3.6 GiB of headroom on a 64 GB machine is a setup that works until something else wants memory. The usual fix is more memory, which costs more than a second pair of RAM sticks and leaves quality risk where it was: a bigger machine can hold the 2.32-bit base unmodified, and that base still has its own quality questions.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4tz24q525rkeea87u5qn.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4tz24q525rkeea87u5qn.jpg" alt="seven claims" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where more memory fits in
&lt;/h2&gt;

&lt;p&gt;The other way past the 64 GB line is more memory. That's what we sell, so a 64 GB CPU box running a frontier-size model is in some sense our competition. Here's how the checks in the table above map onto a KarmaBox.&lt;/p&gt;

&lt;p&gt;Start with fit, since that's what the whole repo is working around. Our &lt;a href="https://www.karmaboxai.com/en/concept/factory/product" rel="noopener noreferrer"&gt;product page&lt;/a&gt; lists two desktop configurations: KarmaBox, with one GB10 and 128 GB of unified memory, and KarmaBox Pro, with two linked GB10s and 256 GB. KarmaBox draws about 170 W. On the 128 GB unit the unmodified 76.87 GiB UD-IQ1_S build leaves about 50 GiB for the cache and everything else (our arithmetic from the published sizes, with no benchmark behind it). At that size the 62.5 GiB budget stops being the constraint, and so does the reason for a custom tensor format.&lt;/p&gt;

&lt;p&gt;Then the last row of the checklist: watch peak memory with your usual software running. On a KarmaBox the usual software is the rest of the avatar team. The product page describes the same 128 GB holding office workflows, local video production, your knowledge base and the avatars together, and our &lt;a href="https://www.karmaboxai.com/en/concept/factory/sovereign-ai" rel="noopener noreferrer"&gt;local AI hardware&lt;/a&gt; page lists 3–4 concurrent tasks for KarmaBox and 6–8 for KarmaBox Pro. So the peak worth recording is the one taken while those tasks are running, with the model loaded and the cache filled to the depth you work at.&lt;/p&gt;

&lt;p&gt;The quality row needs a side-by-side on your own prompts, whatever the hardware. In Karma the model selector at the top of a chat switches between models, as the quick start in the &lt;a href="https://www.karmaboxai.com/en/concept/factory/docs" rel="noopener noreferrer"&gt;Karma docs&lt;/a&gt; shows, so the same prompt set can go to each candidate model and the answers can be compared before you settle which model each avatar uses.&lt;/p&gt;

&lt;p&gt;Last, decide which jobs need the local model at all. Our &lt;a href="https://www.karmaboxai.com/en/concept/factory/solutions/finance" rel="noopener noreferrer"&gt;local-first finance workflow&lt;/a&gt; page walks through a scenario at a 42-person tax firm. Anything touching a client's ledgers, original contracts or identity documents goes to avatars living on the box, and public research or website replies can go to cloud avatars for lower latency and stronger model capability. The firm writes that line into policy instead of judging it task by task, and the audit log records which machine ran each task, so the split can be checked afterwards. The page also recommends an outage drill: cut the network and confirm the on-box avatars finish their core work without it. Where that line falls decides whether a job needs a frontier-size model on your own hardware in the first place.&lt;/p&gt;

&lt;p&gt;If the reason you want a local model is to give a coding agent to colleagues who aren't engineers, our guide to &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-coding-agents-for-non-engineers" rel="noopener noreferrer"&gt;AI coding agents for non-engineers&lt;/a&gt; covers who should get one and with what access.&lt;/p&gt;

&lt;p&gt;Which of these numbers would you re-measure first on your own machine?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;synchopate, &lt;a href="https://github.com/synchopate/llama-modchip" rel="noopener noreferrer"&gt;llama-modchip repository&lt;/a&gt;, commit a9635a5, 8 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;GitHub, &lt;a href="https://github.com/synchopate/llama-modchip/issues" rel="noopener noreferrer"&gt;synchopate/llama-modchip issues #2 to #14&lt;/a&gt; (benchmark submissions), 3 to 8 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Hacker News, &lt;a href="https://news.ycombinator.com/item?id=50411961" rel="noopener noreferrer"&gt;"Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU"&lt;/a&gt;, 8 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;ggml-org, &lt;a href="https://github.com/ggml-org/llama.cpp/blob/d2f83055d6e3b379b5d34c4837122a918cf402c2/ggml/include/ggml.h" rel="noopener noreferrer"&gt;llama.cpp ggml/include/ggml.h and ggml/src/ggml-common.h at commit d2f8305&lt;/a&gt;, 10 August 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;ggml-org, &lt;a href="https://github.com/ggml-org/llama.cpp/blob/d2f83055d6e3b379b5d34c4837122a918cf402c2/src/models/deepseek4.cpp" rel="noopener noreferrer"&gt;llama.cpp src/models/deepseek4.cpp at commit d2f8305&lt;/a&gt;, 10 August 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://www.karmaboxai.com/en/concept/factory/product" rel="noopener noreferrer"&gt;Karma Box&lt;/a&gt;, KarmaBox product page, accessed 9 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>llm</category>
      <category>ai</category>
      <category>performance</category>
      <category>opensource</category>
    </item>
    <item>
      <title>My Agent Had My Email in Its Context. So I Put a 125-Line Guard on Its Way Out</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Fri, 06 Nov 2026 09:21:59 +0000</pubDate>
      <link>https://dev.to/karmabox/my-agent-had-my-email-in-its-context-so-i-put-a-125-line-guard-on-its-way-out-37a5</link>
      <guid>https://dev.to/karmabox/my-agent-had-my-email-in-its-context-so-i-put-a-125-line-guard-on-its-way-out-37a5</guid>
      <description>&lt;p&gt;Most developers who saw the Claude Code thread on Hacker News this week took away one fact: an AI coding agent put a user's email address into the User-Agent header of web requests. Fewer noticed the more useful detail buried in the replies. The fix Anthropic shipped is a piece of code that checks requests as they leave. It does not ask the model to behave better.&lt;/p&gt;

&lt;p&gt;I work at Karma Box, which builds a workspace that runs AI agents on hardware you own, so I'm not neutral on agents and privacy, and you should read this knowing that. To keep it checkable I've quoted the GitHub issues and the HN thread word for word, labelled every comment as one person's view, and kept my own work to things I ran on my own machine on 6 November 2026, with the versions written next to the results. Nothing here reproduces Claude Code's internals. I don't have them, and the test below uses placeholder data only.&lt;/p&gt;

&lt;p&gt;An egress guard is a piece of code that looks at every request on its way out of an agent's tools and refuses the ones that carry data they shouldn't, such as an email address, a key or a value you've told it to watch for. It doesn't rely on the model reading an instruction. It runs every time.&lt;/p&gt;

&lt;h2&gt;
  
  
  What happened, in the order it happened
&lt;/h2&gt;

&lt;p&gt;On 25 July 2026, a user opened issue #81138 on the Claude Code repository. The title is blunt: "Account email (PII) is injected into the model's system prompt without consent, disclosure". The body says Claude Code "takes personal data I gave Anthropic in order to authenticate, and injects it into the model's system prompt on every session". The reporter quoted the injected block from the v2.1.220 bundle:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# userEmail
The user's email address is &amp;lt;account email&amp;gt;.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;According to that report, the value came from &lt;code&gt;~/.claude.json&lt;/code&gt;, under &lt;code&gt;oauthAccount.emailAddress&lt;/code&gt;. The same report shows the builder skipping the field when an environment variable called &lt;code&gt;ANTHROPIC_UNIX_SOCKET&lt;/code&gt; is set. I haven't checked the bundle myself, so treat those details as the reporter's reading of it.&lt;/p&gt;

&lt;p&gt;On 29 October, a short "Tell HN" post linked that issue and a Reddit thread where someone said a model had shared their email address with a website. It got 40 points and little attention.&lt;/p&gt;

&lt;p&gt;On 5 November, issue #107757 went up. Its author wrote: "I receive dozens of scam emails per day because claude code divulged my email to arbitrary 3rd parties in the User-Agent request header while conducting web searches without my permission." They added that they were "migrating my company to Pi with OpenRouter". The Hacker News submission linking it reached 450 points and 157 comments that day. One commenter described the header they had seen as something like &lt;code&gt;my-session-slug/1 &amp;lt;my.name@example.com&amp;gt;&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The same day, an Anthropic contributor replied on the issue: "We introduced a built-in mod called cc-plugin-privacy to deterministically block this behavior in v2.1.317 and later". They added that the mod is "enabled by default, you can turn it off via /plugins."&lt;/p&gt;

&lt;p&gt;That's a quick, specific response, and it's worth saying so. The four months between July and November are a separate question. So is whether the scam emails actually came from these headers. Several commenters doubted that, pointing out that work addresses get scraped from LinkedIn and GitHub all the time. Nobody in the thread proved the link either way, and I can't either.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the model wrote the header
&lt;/h2&gt;

&lt;p&gt;The most useful comment in the thread came from a user called oefrha. In their view, this wasn't a hardcoded line in a web-search tool. It was the model itself:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"As long as the email is in the context, which it is IIRC, it can totally decide to run curl –H 'User-Agent: '."&lt;br&gt;
And later:&lt;/p&gt;

&lt;p&gt;"The problem here literally is the non-deterministic model cooking up commands with the user email address in the header, and the harness not blocking that (and the implemented fix is to deterministically block that)"&lt;br&gt;
That's one person's reading, and another commenter disagreed about how to interpret the original report. But it matches the shape of the fix Anthropic described, and it explains something that matters for anyone building with agents.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A language model that can run shell commands or fetch URLs can put anything from its context into what it sends. Email addresses in the system prompt are the obvious case. oefrha pointed out a less obvious one: "your email will be all over the place in the context as long as the agent is allowed to read git log". Every commit has an author line. An agent that reads history to understand a codebase has seen your address, and maybe your colleagues' addresses too.&lt;/p&gt;

&lt;p&gt;Many web services ask automated clients to identify themselves in the User-Agent, often with a contact address. A model that has seen that convention, and has an address sitting in its context, can decide that a polite bot includes one. Nobody has to code that. It just happens.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fct0fngsnxg5wqcxfx6gh.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fct0fngsnxg5wqcxfx6gh.jpg" alt="every tool that sends something is a way out" width="800" height="430"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;So there are two separate questions. What does the agent know? And what is allowed out through the door a request leaves through? The first is hard to keep small, because agents need context to be useful. The second is something you can check.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step one: see what a request carries
&lt;/h2&gt;

&lt;p&gt;Before writing a guard, I wanted to see a request the way the receiving server sees it. A commenter in the thread suggested exactly this: "fire up a little web server locally, have the agent make a request to it, then check the logs."&lt;/p&gt;

&lt;p&gt;The capture server is about fifteen lines with Python's standard library. It listens on 127.0.0.1, writes the path and User-Agent of every GET to a file, and replies "ok".&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;class Capture(http.server.BaseHTTPRequestHandler):
    log_path = None

    def do_GET(self):
        with open(self.log_path, "a") as f:
            f.write(json.dumps({"path": self.path, "ua": self.headers.get("User-Agent")}) + "\n")
        self.send_response(200); self.end_headers(); self.wfile.write(b"ok\n")
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A colleague ran the simplest version of this earlier the same morning. Default curl 8.7.1 announced itself as &lt;code&gt;curl/8.7.1&lt;/code&gt;. Python's urllib announced &lt;code&gt;Python-urllib/3.9&lt;/code&gt;. When they built a User-Agent from a variable holding a placeholder address, &lt;code&gt;demo.user@example.invalid&lt;/code&gt;, the server logged it exactly as sent. That's not a surprise. It's the point: the server records whatever arrives, and nothing between the client and the server had any opinion about it.&lt;/p&gt;

&lt;p&gt;If you run an agent, this is the cheapest test you can do this week. Point one of its fetch tools at a local server and read the log. You'll learn what your agent sends by default, which is more than most of us knew on Wednesday.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step two: a guard at the door
&lt;/h2&gt;

&lt;p&gt;The guard is a small forward proxy that sits between the client and the capture server. For every request it looks at the request line, with the URL decoded, and at every header. If it finds an email address, a key-shaped string, or a value you've registered as sensitive, it answers 403 and doesn't forward the request. Otherwise it passes the request on.&lt;/p&gt;

&lt;p&gt;The checks are deliberately plain:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
KEYS = [re.compile(p) for p in (r"\bsk-[A-Za-z0-9_-]{16,}", r"\bAKIA[0-9A-Z]{16}\b", r"\bghp_[A-Za-z0-9]{36}\b")]
DENY = {"demo.user@example.invalid"}          # 显式登记：上下文里已知的身份串


def findings(text):
    hits = []
    if any(s in text for s in DENY):
        hits.append("denylist")
    if EMAIL.search(text):
        hits.append("email")
    if any(k.search(text) for k in KEYS):
        hits.append("key")
    return hits


def inspect(request_line, headers):
    target = request_line.split(" ")[1] if " " in request_line else request_line
    decoded = urllib.parse.unquote_plus(target)
    hits = [("url", h) for h in findings(decoded)]
    for k, v in headers.items():
        hits += [("header:" + k, h) for h in findings(v)]
    return hits
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three kinds of check, each doing a different job. The email pattern catches anything that looks like an address. It's loose on purpose; a false positive costs you one failed request you can look at, and a false negative costs you an address you can't get back. The key patterns catch the shapes of a few common API keys. The denylist is the interesting one. If your harness knows the account email, the git author address, or a customer name the agent is working with, it can register those exact strings and refuse any request that carries them, whatever the pattern rules think.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;unquote_plus&lt;/code&gt; call matters more than it looks. An address in a query string usually arrives URL-encoded, as &lt;code&gt;demo.user%40example.invalid&lt;/code&gt;. A pattern check on the raw string would miss it. Decoding first catches it.&lt;/p&gt;

&lt;p&gt;The full script, with the proxy handler, the test cases and the timing loop, is 125 lines. It uses only the standard library and refuses to run unless you hand it a fresh, empty directory for its log, so it can't scribble over anything else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step three: seven requests
&lt;/h2&gt;

&lt;p&gt;I ran seven requests through the guard to the capture server. Three should pass, three should be blocked, and one is there to show what the guard can't see.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmt22dvfuw9l0xr6c8v8c.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmt22dvfuw9l0xr6c8v8c.png" alt=" " width="800" height="456"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The capture server's log showed four requests: the three that should pass, and the base64 one. None of the three blocked requests reached it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx9kfvvscdf3jr7qvc0yy.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fx9kfvvscdf3jr7qvc0yy.jpg" alt="seven requests through a 125-line egress guard" width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
I ran the whole thing three times. Same results each time. Machine: Apple M4 Mac mini with 24 GB, macOS 15.6.1, Python 3.9.6, curl 8.7.1, everything on 127.0.0.1, no TLS, placeholder data only.&lt;/p&gt;

&lt;p&gt;On cost: 200 sequential requests took 0.040 seconds direct and 0.067 seconds through the guard, about 0.13 milliseconds extra per request (local loopback, sequential, no TLS; a real network round trip will dwarf this). Inspecting a request is cheap. The work is in deciding what to look for.&lt;/p&gt;

&lt;p&gt;The base64 miss deserves a straight answer. A model that wanted to smuggle an address past a pattern check could encode it, split it, or spell it out. Pattern checks catch the shapes you list, and the denylist catches the exact strings you register. Neither understands meaning. If you need more, you narrow what the agent can send at all: fixed headers per tool, no free-form headers, an allowlist of hosts. That's a policy decision, and it costs flexibility.&lt;/p&gt;
&lt;h2&gt;
  
  
  The pitfall that ate my first run
&lt;/h2&gt;

&lt;p&gt;My first run reported every request as 200, including the ones that should have been blocked. The capture log had all seven. For a minute I thought the inspection code was broken.&lt;/p&gt;

&lt;p&gt;It wasn't. The machine has &lt;code&gt;NO_PROXY&lt;/code&gt; and &lt;code&gt;no_proxy&lt;/code&gt; set, and both lists include &lt;code&gt;127.0.0.1&lt;/code&gt; and &lt;code&gt;localhost&lt;/code&gt;. curl honours those lists, so it ignored my &lt;code&gt;-x&lt;/code&gt; flag and went straight to the server. The guard never saw a single request.&lt;/p&gt;

&lt;p&gt;The fix in the test is one flag, &lt;code&gt;--noproxy ""&lt;/code&gt;, which clears the exception list for that call. The lesson is bigger than the flag. A guard that lives on the network only works if every request actually goes through it, and environment settings you've forgotten about can route around it without any warning. I only noticed because I had a capture server that logged what arrived. Without it, I'd have shipped a guard that guarded nothing and a test that said everything was fine.&lt;/p&gt;

&lt;p&gt;There's a cheap way to catch this class of bug in any guard you build: count. The guard should know how many requests it saw, and the capture server knows how many it received. If the guard saw zero and the server logged seven, something is going around it. My script now prints which requests reached the capture server for exactly this reason, and if I'd had that line on the first run I'd have found the problem in seconds instead of a minute of doubting my regex.&lt;/p&gt;

&lt;p&gt;The same check works in production. Log a counter at the guard, compare it with what the tool layer thinks it sent, and alert when they drift apart. A guard that's been routed around doesn't fail loudly. It just stops seeing anything, and an empty refusal log looks exactly like a well-behaved agent.&lt;/p&gt;
&lt;h2&gt;
  
  
  Where the check belongs
&lt;/h2&gt;

&lt;p&gt;There's a second reason not to rely on a network proxy, and it's a bigger one. My test used plain HTTP. Almost every real request an agent makes is HTTPS. When a client talks HTTPS through a forward proxy, it sends a &lt;code&gt;CONNECT&lt;/code&gt; request with a host and port, and everything after that, including the headers and the full URL, is encrypted. A plain proxy can't see the User-Agent at all. You'd need TLS interception to inspect it, with its own certificates, trust and failure modes.&lt;/p&gt;

&lt;p&gt;So the place to check is earlier: in the tool, after the agent has decided what to send and before the request is built and encrypted. The same &lt;code&gt;inspect()&lt;/code&gt; function works there. I called it directly on three HTTPS-shaped requests. A normal search passed. A request with the placeholder address in the User-Agent and one with it in the query string were both flagged, by the denylist and the email pattern. Ten thousand inspections of the second request took 0.034 seconds on the same machine.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpe54jgj71bmcgisp2nc7.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpe54jgj71bmcgisp2nc7.jpg" alt="where to put the check" width="800" height="360"&gt;&lt;/a&gt;&lt;br&gt;
That's also, as far as I can tell from the public reply, the level at which Anthropic fixed this. Their description says the new mod is built in and blocks the behaviour "deterministically". I haven't seen its source. The contributor said they'd try to make it available, the way some other built-in mods are.&lt;/p&gt;

&lt;p&gt;For your own agents, the practical version looks like this:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;wrap every tool that makes network calls in one function that runs the check&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;register the identity strings the harness already knows (account email, git author, customer names in scope) in the denylist&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;log every refusal with the tool name and the field that matched, never the matched value&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;treat a refusal as a bug report about the agent's behaviour, and look at what it was trying to do&lt;br&gt;
None of that is clever. It's the same thing we already do for SQL: we don't ask the query author to promise they'll escape input, we use code that does it every time.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  What goes on the denylist
&lt;/h2&gt;

&lt;p&gt;The pattern rules are generic. The denylist is the part only you can write, because it holds the specific strings your agent has seen. It's worth an hour of thought.&lt;/p&gt;

&lt;p&gt;Start with whatever identifies the person running the agent. The account email is the obvious one, and in this week's story it was the one that leaked. Next is the git identity. Running &lt;code&gt;git config user.email&lt;/code&gt; in a repository shows the address that will end up on every commit the agent makes, and on every commit it reads. If your team signs commits with work addresses, those addresses are in the agent's context the moment it reads &lt;code&gt;git log&lt;/code&gt;, whether or not anything put them in the system prompt.&lt;/p&gt;

&lt;p&gt;Then think about what the agent is working on. A support agent sees customer names and order numbers. A finance agent sees account numbers. A coding agent in a client repository sees the client's internal hostnames. Some of these are personal data. Some are commercially sensitive. All of them are strings you can register for the length of the task and remove afterwards.&lt;/p&gt;

&lt;p&gt;Last, the secrets. Key patterns catch common shapes, but your own tokens may not match any public pattern. If the harness hands the agent a token to call an internal API, register that exact token too. The denylist doesn't care what a string looks like. It only cares whether it's there.&lt;/p&gt;

&lt;p&gt;Two practical notes from building this. Keep the denylist out of the agent's context; it's a list of things the agent shouldn't send, and handing it to the model defeats the purpose. And when the guard refuses a request, log which field matched and which rule fired, never the value. A log of refusals that quotes the secret it protected is just another place the secret lives.&lt;/p&gt;
&lt;h2&gt;
  
  
  What this guard covers, and what it doesn't
&lt;/h2&gt;

&lt;p&gt;This guard covers outbound leaks through headers and URLs from tools you control. That's a real class of problem, and it's the one in this week's story.&lt;/p&gt;

&lt;p&gt;It doesn't cover what the model provider sees. Whatever is in the context goes to the provider's servers with every call, and the guard sits after that. If you want less there, the only fix is less context. Our team checked our own agent setup that morning: across 668 files in the configuration and workspace, there was no email field, though there were a user identifier and a machine identifier. We didn't print either. That's a smaller footprint than an email, and it's still not nothing.&lt;/p&gt;

&lt;p&gt;It doesn't cover request bodies, which my test didn't inspect. Bodies need the same check, with size limits so you don't scan uploads byte by byte.&lt;/p&gt;

&lt;p&gt;It doesn't cover disguised data, as the base64 case showed.&lt;/p&gt;

&lt;p&gt;And it doesn't cover tools you don't control. If a vendor's built-in web tool runs on their side, your guard never sees the request. You're relying on their fix, which this week was a good one.&lt;/p&gt;

&lt;p&gt;So: outbound-leak risk, partly addressed. Context-exposure risk, unchanged. Encoded-data risk, open. Third-party-tool risk, outside your reach. Naming them separately makes it easier to see which one you're working on.&lt;/p&gt;
&lt;h2&gt;
  
  
  Pointing an agent you didn't build at the capture server
&lt;/h2&gt;

&lt;p&gt;The capture server isn't only for your own code. Most agent tools that can fetch a URL will happily fetch &lt;code&gt;http://127.0.0.1:&amp;lt;port&amp;gt;/anything&lt;/code&gt; if you ask them to, and that's enough to see what their fetch tool sends by default. It's the test the HN commenter suggested, and it takes a few minutes.&lt;/p&gt;

&lt;p&gt;A few things make the result more useful. Ask for a plain page fetch, and separately for something that looks like research ("look up this page and summarise it"), because an agent may build requests differently depending on the task. Run it once in a fresh session and once in a session where the agent has already read a repository with commit history, so you can see whether what it knows changes what it sends. And read the whole log line, not just the User-Agent. Query strings and custom headers are where unexpected values tend to turn up.&lt;/p&gt;

&lt;p&gt;The limits are worth stating. This shows you requests the agent sends to your server. It says nothing about requests a vendor's tool sends from the vendor's own infrastructure, which never touch your machine. A clean log on one day also doesn't prove much about the next release. Rerun it when the tool updates. That's cheap, and it's the only way to know that this week's fix is still in place in the version you're running.&lt;/p&gt;
&lt;h2&gt;
  
  
  Running it yourself
&lt;/h2&gt;

&lt;p&gt;The script takes one argument, an empty directory for its log:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;D=$(mktemp -d)
python3 d08p_egress_guard.py "$D"
rm -rf "$D"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It starts both servers on random ports on 127.0.0.1, runs the seven cases, prints which requests reached the capture server, times 200 requests direct and through the guard, and shuts down. Nothing leaves your machine. If your environment sets &lt;code&gt;NO_PROXY&lt;/code&gt; for localhost, the &lt;code&gt;--noproxy ""&lt;/code&gt; flag in the test handles it, but check yours anyway; that's the bug that hid every result on my first run.&lt;/p&gt;

&lt;p&gt;If you adapt it, change the denylist first. The patterns are generic. The denylist is where your knowledge of your own agent goes: the strings it has seen and should never send.&lt;/p&gt;

&lt;p&gt;For keys specifically, our &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-api-key-self-custody-checklist" rel="noopener noreferrer"&gt;API key self-custody checklist&lt;/a&gt; covers where to keep them so fewer of them end up in an agent's context in the first place. The longer guide on &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-agent-data-leakage" rel="noopener noreferrer"&gt;AI agent data leakage&lt;/a&gt; covers the other outbound channels, what GDPR's data minimisation rules expect, and how to test a vendor's agent as well as your own.&lt;/p&gt;

&lt;p&gt;What would your agent's capture log show if you pointed it at a local server today?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://github.com/anthropics/claude-code/issues/81138" rel="noopener noreferrer"&gt;GitHub, anthropics/claude-code issue #81138&lt;/a&gt;, "Account email (PII) is injected into the model's system prompt without consent, disclosure", 25 July 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://github.com/anthropics/claude-code/issues/107757" rel="noopener noreferrer"&gt;GitHub, anthropics/claude-code issue #107757,&lt;/a&gt; "Bug Report: Privacy violation in User-Agent header", 5 November 2026, including the contributor reply on cc-plugin-privacy. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Hacker News, "&lt;a href="https://news.ycombinator.com/item?id=50385272" rel="noopener noreferrer"&gt;Claude Code sends your email address to websites&lt;/a&gt;" discussion, 5 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>security</category>
      <category>python</category>
      <category>sovereignai</category>
    </item>
    <item>
      <title>I Ran an Exit Drill on My AI Agent's Workspace. It Found 62 Folders That Weren't There</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Thu, 05 Nov 2026 11:01:02 +0000</pubDate>
      <link>https://dev.to/karmabox/i-ran-an-exit-drill-on-my-ai-agents-workspace-it-found-62-folders-that-werent-there-5hcc</link>
      <guid>https://dev.to/karmabox/i-ran-an-exit-drill-on-my-ai-agents-workspace-it-found-62-folders-that-werent-there-5hcc</guid>
      <description>&lt;p&gt;Every office building runs fire drills. Nobody expects a fire on drill day. The point is to find out, while nothing is burning, that the stairwell door sticks and half the floor doesn't know where to meet. Almost nobody runs that drill for the place where their AI agents work.&lt;/p&gt;

&lt;p&gt;I write for KarmaBox, which sells a workspace that runs on hardware you own, so a post that ends with "your agent's state is scattered" is convenient for me. Here is what I did to keep it checkable. Every number below comes from four real runs on 5 November 2026, on a copy of one real avatar workspace (an avatar is our name for an AI agent with its own job, files and permissions), on an Apple M4 Mac mini with 24 GB of memory. The full script is in this post. Where I couldn't find something out, I say so.&lt;/p&gt;

&lt;p&gt;An exit drill is a timed rehearsal of leaving a tool: you inventory what your agents depend on, pack it, restore it somewhere fresh, check it, and write down what didn't come along. It isn't a backup test, because a backup test asks whether your files survived; a drill asks whether your work can.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I ran it today
&lt;/h2&gt;

&lt;p&gt;On 4 November, &lt;a href="https://techcrunch.com/2026/11/04/meta-acquires-ai-coding-startup-conductor/" rel="noopener noreferrer"&gt;TechCrunch&lt;/a&gt; reported that Meta had acquired Conductor, a tool for running several AI coding agents side by side. The announcement, quoted word for word in the Hacker News thread, gave users three dates: sign-ups and downloads closed that day, the cloud would go read-only on 13 November, and on 20 November it would shut down and cloud data deletion would begin. That is 16 days from announcement to deletion, and 9 of them allow normal work. (If you're facing a notice like that right now, our guide on &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/ai-tool-shutdown-what-to-do" rel="noopener noreferrer"&gt;what to do when an AI tool shuts down&lt;/a&gt; covers the whole sequence, from reading the notice to the deletion date. This post is only about the rehearsal.)&lt;/p&gt;

&lt;p&gt;The comments that stuck with me weren't about files. One user listed what they'd miss: "cloud webhooks + workspace/session API + isolated VMs". None of that is something you download. It made me wonder how much of my own agent's workspace I could actually pack up, and how much I only thought I could.&lt;/p&gt;

&lt;p&gt;So I wrote a script and ran it. Then I ran it again, and again. It took four runs before the result meant anything.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpnopg13euudsutkt2jzd.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpnopg13euudsutkt2jzd.jpg" alt="an agent workspace is a stack" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What the script does
&lt;/h2&gt;

&lt;p&gt;It's deliberately boring. Standard library only, because a drill you can't run on a fresh machine isn't much of a drill. It takes two arguments: a copy of the workspace, and a small JSON file listing things the workspace depends on that don't live in the folder.&lt;/p&gt;

&lt;p&gt;Five steps:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Inventory&lt;/strong&gt;. Walk the folder, hash every file with SHA-256, and sort it into a layer (config, scripts, content, assets) by simple path rules. Hold back anything that looks like a secret.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Export&lt;/strong&gt;. Write one &lt;code&gt;.tar.gz&lt;/code&gt; per layer, so an incomplete layer is obvious and can be handed to a different person.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Restore&lt;/strong&gt;. Extract every archive into a brand-new temporary directory created with &lt;code&gt;mkdtemp&lt;/code&gt;. Never into a shared folder.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Verify&lt;/strong&gt;. Re-hash every restored file and compare.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Report gaps&lt;/strong&gt;. Differences, held-back secrets, symlinks that point outside the folder, and anything from the "outside" list that isn't present.&lt;br&gt;
The "outside" file is the honest part. A script can only check what it's told exists. For my workspace I listed five things: conversation history from a cloud engine that ran part of this project earlier today, local session transcripts, the shared skills the agent uses, the agent's memory, and scheduled jobs.&lt;br&gt;
&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[
 {"name": "conversation history from the cloud engine (before 09:19 today)", "layer": "transcripts", "path": ""},
 {"name": "local session transcripts", "layer": "transcripts", "path": "~/.karma/sessions"},
 {"name": "shared skills the symlinks point to", "layer": "config", "path": "~/.karma/skills"},
 {"name": "avatar memory", "layer": "memory", "path": "~/.karma/memory"},
 {"name": "scheduled jobs", "layer": "integrations", "path": "~/.karma/cron"}
]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before running anything, I copied the workspace into a fresh directory under &lt;code&gt;/tmp&lt;/code&gt; with &lt;code&gt;rsync -a&lt;/code&gt;, leaving out temporary folders. The script never writes inside its source. I'd learned that one earlier the same day: during a quick manual drill this morning I unpacked an archive straight into &lt;code&gt;/tmp&lt;/code&gt; instead of a new folder. If anything had already been there, the comparison would have proved nothing.&lt;/p&gt;

&lt;p&gt;Here is the final version, all 151 lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;#!/usr/bin/env python3
"""Exit drill for an AI agent workspace (D07 Dev.to piece).

Usage:  python3 -I d07p_exit_drill.py &amp;lt;workspace_copy&amp;gt; &amp;lt;outside.json&amp;gt;

Inventory -&amp;gt; export one archive per layer -&amp;gt; restore into a fresh empty dir
-&amp;gt; verify every file by sha256 -&amp;gt; print a gap report. Standard library only.
Run it on a COPY of the workspace; it never writes inside the source.
"""
import hashlib, json, os, re, sys, tarfile, tempfile, time

LAYERS = [  # first match wins
    ("config",  lambda p: os.path.basename(p) in ("CLAUDE.md", "settings.json", "settings.local.json")
                          or p.startswith(".claude/")),
    ("scripts", lambda p: p.endswith((".py", ".sh"))),
    ("content", lambda p: p.endswith((".md", ".txt", ".html", ".docx", ".pdf"))),
    ("assets",  lambda p: p.endswith((".png", ".svg", ".jpg", ".jpeg", ".zip"))),
    ("other",   lambda p: True),
]
# Case-sensitive on purpose: with re.I, "AKIA..." matches random base64 in embedded images.
SECRET = re.compile(rb"sk-[A-Za-z0-9]{20,}|AKIA[0-9A-Z]{16}|-----BEGIN [A-Z ]*PRIVATE KEY-----|"
                    rb"(?i:(api[_-]?key|secret|token)\s*[:=]\s*['\"][A-Za-z0-9_\-]{16,})")
SKIP_DIRS = {".git", "__pycache__"}


def sha(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1 &amp;lt;&amp;lt; 20), b""):
            h.update(chunk)
    return h.hexdigest()


def inventory(src):
    files, secrets, links = {}, [], []
    real_src = os.path.realpath(src)
    for root, dirs, names in os.walk(src):
        dirs[:] = [d for d in dirs if d not in SKIP_DIRS]
        # os.walk lists symlinked directories under dirs and never descends into them,
        # so they would vanish from the inventory without a word. Record them here.
        for d in dirs:
            full = os.path.join(root, d)
            if os.path.islink(full):
                target = os.path.realpath(full)
                if not target.startswith(real_src + os.sep):
                    links.append((os.path.relpath(full, src), os.readlink(full)))
        for n in names:
            full = os.path.join(root, n)
            rel = os.path.relpath(full, src)
            if os.path.islink(full):
                if not os.path.realpath(full).startswith(real_src + os.sep):
                    links.append((rel, os.readlink(full)))
                continue
            with open(full, "rb") as f:
                head = f.read(2 &amp;lt;&amp;lt; 20)
            if SECRET.search(head):
                secrets.append(rel)  # never archived
                continue
            layer = next(name for name, rule in LAYERS if rule(rel))
            files[rel] = (layer, sha(full), "file")
    return files, secrets, links


def export(src, files, outdir):
    archives = {}
    for layer, _ in LAYERS:
        members = [r for r, (l, _, _) in files.items() if l == layer]
        if not members:
            continue
        path = os.path.join(outdir, f"{layer}.tar.gz")
        with tarfile.open(path, "w:gz") as t:
            for rel in sorted(members):
                t.add(os.path.join(src, rel), arcname=rel, recursive=False)
        archives[layer] = (path, len(members))
    return archives


def restore(archives, dest):
    # tarfile's filter="data" only exists on newer Pythons (macOS ships 3.9),
    # so refuse absolute paths and ../ ourselves before extracting.
    root = os.path.realpath(dest)
    for layer, (path, _) in archives.items():
        with tarfile.open(path) as t:
            for m in t.getmembers():
                target = os.path.realpath(os.path.join(root, m.name))
                if not target.startswith(root + os.sep):
                    raise SystemExit(f"unsafe path in {layer}: {m.name}")
            t.extractall(root)


def verify(files, dest):
    diffs = []
    for rel, (_, digest, kind) in files.items():
        p = os.path.join(dest, rel)
        if not os.path.exists(p):
            diffs.append((rel, "missing"))
        elif sha(p) != digest:
            diffs.append((rel, "content differs"))
    return diffs


def outside(spec):
    """Things the workspace depends on that do not live in the folder."""
    report = []
    for item in json.load(open(spec)):
        where = os.path.expanduser(item.get("path", ""))
        present = bool(where) and os.path.exists(where) and (
            not os.path.isdir(where) or any(os.scandir(where)))
        report.append((item["name"], item["layer"], "found locally" if present else "empty or missing here"))
    return report


def main(src, spec):
    t = {}
    work = tempfile.mkdtemp(prefix="exit-drill-")          # fresh, empty, ours
    out, dest = os.path.join(work, "archives"), os.path.join(work, "restored")
    os.makedirs(out); os.makedirs(dest)

    s = time.perf_counter(); files, secrets, links = inventory(src); t["inventory"] = time.perf_counter() - s
    s = time.perf_counter(); archives = export(src, files, out); t["export"] = time.perf_counter() - s
    s = time.perf_counter(); restore(archives, dest); t["restore"] = time.perf_counter() - s
    s = time.perf_counter(); diffs = verify(files, dest); t["verify"] = time.perf_counter() - s
    ext = outside(spec)

    size = sum(os.path.getsize(os.path.join(src, r)) for r, v in files.items() if v[2] == "file")
    packed = sum(os.path.getsize(p) for p, _ in archives.values())
    print(f"workdir   {work}")
    print(f"files     {len(files)}  ({size/1e6:.1f} MB raw, {packed/1e6:.1f} MB packed)")
    for layer, (p, n) in archives.items():
        print(f"  {layer:8} {n:4} files  {os.path.getsize(p)/1e6:7.2f} MB")
    print("timing    " + "  ".join(f"{k} {v:.2f}s" for k, v in t.items()))
    print(f"diffs     {len(diffs)}")
    for rel, why in diffs[:10]:
        print(f"  - {rel}: {why}")
    print(f"secrets   {len(secrets)} file(s) held back from the archive")
    for rel in secrets:
        print(f"  - {rel}")
    print(f"links     {len(links)} symlink(s) point outside the folder (archive would hold dangling links)")
    for rel, target in links[:3]:
        print(f"  - {rel} -&amp;gt; {target}")
    if len(links) &amp;gt; 3:
        print(f"  ... and {len(links) - 3} more")
    print("outside the folder:")
    for name, layer, status in ext:
        print(f"  - [{layer}] {name}: {status}")
    gaps = len(diffs) + len(secrets) + len(links) + sum(s != "found locally" for *_, s in ext)
    print(f"GAPS      {gaps}")


if __name__ == "__main__":
    main(*sys.argv[1:3])
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it with &lt;code&gt;python3 -I&lt;/code&gt;, which stops Python from importing anything sitting next to the script or in the current directory. That matters when you're pointing a script at a folder full of files you didn't write. (It's also the wrong setting for the task you run afterwards, as I found out below.)&lt;/p&gt;

&lt;h2&gt;
  
  
  Run 1: it crashed
&lt;/h2&gt;

&lt;p&gt;The first version restored archives with &lt;code&gt;t.extractall(dest, filter="data")&lt;/code&gt;. That filter refuses absolute paths and &lt;code&gt;../&lt;/code&gt;tricks, which is exactly what you want when unpacking an archive. It arrived with &lt;a href="https://peps.python.org/pep-0706/" rel="noopener noreferrer"&gt;PEP 706&lt;/a&gt; for Python 3.12 and was backported to some older patch releases.&lt;/p&gt;

&lt;p&gt;The Mac's system Python is 3.9.6. It doesn't have it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TypeError: extractall() got an unexpected keyword argument 'filter'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Small thing, but it's the kind of small thing a real migration hits on the new machine, at the worst moment. A drill that only works on the laptop where you wrote it hasn't tested anything. The fix was ten lines that check every member's resolved path stays inside the destination before extracting.&lt;/p&gt;

&lt;p&gt;A guard you've never seen fire is a hope, so later I tested it. In a throwaway folder I built a one-file archive whose only member was named &lt;code&gt;../escaped.txt&lt;/code&gt;, the classic way an archive writes outside the folder you meant to unpack into. Then I handed it to the &lt;code&gt;restore&lt;/code&gt; function from the script. It stopped with &lt;code&gt;unsafe path in evil: ../escaped.txt&lt;/code&gt;, the destination folder stayed empty, and no &lt;code&gt;escaped.txt&lt;/code&gt; appeared next to it. The check runs before &lt;code&gt;extractall&lt;/code&gt;, so nothing is written at all when one bad member is found. That's stricter than skipping the bad member, and I prefer it here. An archive with a path like that in it is not an archive I want to half-trust.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run 2: four "secrets" and a silent hole
&lt;/h2&gt;

&lt;p&gt;The second run finished. It reported zero differences, which felt good for about a minute.&lt;/p&gt;

&lt;p&gt;Then I read the rest. It had held back four files as secrets. All four were preview HTML pages, the ones I generate every day with images embedded as base64. My regex had &lt;code&gt;re.I&lt;/code&gt; on it, so the AWS key pattern &lt;code&gt;AKIA&lt;/code&gt; plus sixteen uppercase characters became case-insensitive, and somewhere in 3 MB of base64 there were sequences like &lt;code&gt;Akia5mE7JtVMcCymlWul&lt;/code&gt;. The fix was to make the key patterns case-sensitive and keep &lt;code&gt;(?i:...)&lt;/code&gt; only around the &lt;code&gt;api_key = "..."&lt;/code&gt;style pattern.&lt;/p&gt;

&lt;p&gt;I reran both versions of the pattern against the 16:30 copy to see how big the difference really is. Seventeen files in that copy carry embedded base64 images. The old, case-insensitive pattern flagged five files. Four were preview pages. The fifth was this article, because by then the draft quoted &lt;code&gt;Akia5mE7JtVMcCymlWul&lt;/code&gt; as an example, and that string is exactly what the old pattern was wrongly looking for. The corrected pattern flagged none. A scanner that holds back your own write-up of its bug is a good reminder that false positives aren't harmless. Every file held back is a file that silently doesn't move.&lt;/p&gt;

&lt;p&gt;The worse problem didn't produce any output at all. The config layer had two files. That couldn't be right, because the agent has dozens of skill folders under &lt;code&gt;.claude/skills&lt;/code&gt;. They weren't in the inventory, they weren't in the archive, and nothing said so.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://docs.python.org/3/library/os.html" rel="noopener noreferrer"&gt;Python docs for os.walk&lt;/a&gt; explain it in one line: &lt;code&gt;dirnames&lt;/code&gt; includes symlinks to directories, and with the default &lt;code&gt;followlinks=False&lt;/code&gt; the walk never goes into them. Every one of those skill folders is a symlink. They point to &lt;code&gt;~/.karma/skills&lt;/code&gt;, outside the workspace. My drill had skipped all 62 of them and reported success.&lt;/p&gt;

&lt;p&gt;If I'd stopped at run 2, I would have told you this workspace moves cleanly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Run 4: zero diffs, 65 gaps
&lt;/h2&gt;

&lt;p&gt;Run 3 added symlink detection and fixed the regex. Run 4 corrected one entry in the outside list, which I'd pointed at an empty folder instead of where the symlinks actually lead. Here is what it printed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;files     557  (161.1 MB raw, 133.6 MB packed)
  config      2 files     0.01 MB
  scripts    61 files     0.27 MB
  content   132 files    39.56 MB
  assets    362 files    93.78 MB
timing    inventory 3.03s  export 3.99s  restore 0.82s  verify 0.09s
diffs     0
secrets   0 file(s) held back from the archive
links     62 symlink(s) point outside the folder (archive would hold dangling links)
outside the folder:
  - [transcripts] conversation history from the cloud engine (before 09:19 today): empty or missing here
  - [transcripts] local session transcripts: empty or missing here
  - [config] shared skills the symlinks point to: found locally
  - [memory] avatar memory: found locally
  - [integrations] scheduled jobs: empty or missing here
GAPS      65
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0n464hx5f5xjodnbkwoj.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0n464hx5f5xjodnbkwoj.jpg" alt="what the final run measured" width="800" height="430"&gt;&lt;/a&gt;&lt;br&gt;
Wall clock was 7.99 seconds (local disk to local disk, no network, Apple M4 Mac mini, 24 GB). Every file that was in the folder came back byte for byte. By the measure most people use, that's a perfect result.&lt;/p&gt;

&lt;p&gt;The gap count tells a different story, so here is what the 65 actually are:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flaggt64wpr0v6tu1ba2f.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flaggt64wpr0v6tu1ba2f.png" alt=" " width="800" height="419"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;So 62 of the 65 come from one design decision: skills are shared across avatars and linked in rather than copied. That's a sensible choice on one machine. It's a migration trap the moment you move. One is a genuine loss, the cloud conversation. One is my own ignorance about where a thing is stored, which a drill is supposed to surface. And one, scheduled jobs, turned out to be a blind spot rather than a false alarm: the agent's folder really is empty, but the machine's own crontab holds a daily job for another project, in a place my outside list never looked. A gap report you have to interpret is still more honest than one that hides its blind spots.&lt;/p&gt;
&lt;h2&gt;
  
  
  Two ways to pack a symlink, and why the wrong one looks fine
&lt;/h2&gt;

&lt;p&gt;Later that afternoon I made a fresh copy of the workspace and tried the obvious fix for the 62 folders: let &lt;code&gt;tar&lt;/code&gt; deal with them. macOS ships bsdtar 3.5.3. By default it stores a symbolic link as a link. With &lt;code&gt;-L&lt;/code&gt; (or its synonym &lt;code&gt;-h&lt;/code&gt;), it follows every link and archives whatever the link points to. Python's &lt;code&gt;tarfile&lt;/code&gt; has the same switch, &lt;code&gt;dereference&lt;/code&gt;, and it also defaults to off.&lt;/p&gt;

&lt;p&gt;So I packed just the skills folder both ways:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;tar -czf  links.tgz -C ws .claude/skills    1,496 bytes    60 entries
tar -czhf deref.tgz -C ws .claude/skills  213,671 bytes   170 entries
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first archive is tiny because it holds nothing but 59 pointers and the folder they sit in. The second holds the skills themselves.&lt;/p&gt;

&lt;p&gt;Here's the trap. I unpacked &lt;code&gt;links.tgz&lt;/code&gt; into a new empty folder and checked one of the links. It resolved. &lt;code&gt;SKILL.md&lt;/code&gt; was right there. On this machine every absolute path in that archive points at something that exists, so the restore looks perfect. Run the same check on a clean laptop and you'd get 59 links to nothing. A drill that restores on the same machine it packed on will never show you this, which is why my script now flags links instead of trusting them.&lt;/p&gt;

&lt;p&gt;Dereferencing isn't automatically the right answer either. It cost about 210 KB here, which is nothing. But a copied skill stops being the shared skill. If someone improves the original next week, the copy in your archive won't know. Whether the skills folder belongs to the workspace or to the machine is a decision, and both answers are reasonable. What matters is making it on purpose and writing it down.&lt;/p&gt;

&lt;p&gt;One more thing showed up in that copy. At 14:28 the workspace had 62 linked skills. At 16:30 it had 59. I don't know which three went or why. It was a normal afternoon of work on the same machine. That's the quiet argument for drilling more than once: the thing you're measuring keeps moving.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does byte-for-byte mean the same file?
&lt;/h2&gt;

&lt;p&gt;A matching SHA-256 tells you the contents survived. It doesn't tell you about anything else. So on the 16:30 copy I compared what the hash ignores.&lt;/p&gt;

&lt;p&gt;Across all 642 restored files, the permission bits matched the source in every case, and so did the modification times, to the second. None of the files carried extended attributes, the macOS metadata that can mark a download as quarantined. For this workspace, at least, the archive carried more than just contents.&lt;/p&gt;

&lt;p&gt;Then I looked at what the files need in order to work. The answer wasn't in any of them. None of the Python scripts read an environment variable, which I'd half expected to be the weak spot. Instead, eight of them call &lt;code&gt;qlmanage&lt;/code&gt;, a macOS Quick Look tool, to turn SVG charts into PNGs. Thirteen hard-code font paths under &lt;code&gt;/System/Library/Fonts&lt;/code&gt;. Seven point at a PingFang font file inside a system asset folder whose name contains a long hash.&lt;/p&gt;

&lt;p&gt;Every one of those files restored byte for byte. On a machine without &lt;code&gt;qlmanage&lt;/code&gt;, or a Mac that stores that font somewhere else, the figure pipeline would stop. Nothing in the gap report would mention it, because nothing is missing from the folder. The dependency is the machine.&lt;/p&gt;

&lt;p&gt;This is the case for the last step of the drill, running one real task after the restore. A file count can't find a font that isn't installed. Rendering one chart on the target machine can.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running one real task on the restored copy
&lt;/h2&gt;

&lt;p&gt;Everything so far compares files. The drill's last step is different: take the restored copy and do one ordinary piece of work with it. For this workspace the most ordinary task is drawing a chart, so I went into the restored folder from the third 16:30 run and asked it to render one of the figures in this series.&lt;/p&gt;

&lt;p&gt;The first attempt failed in under a tenth of a second:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ModuleNotFoundError: No module named 'd07_figs'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing was missing. The module was sitting right next to the script. The problem was my own advice. I'd been running everything with &lt;code&gt;python3 -I&lt;/code&gt;, which keeps the script's directory and the current directory off Python's import path. That's the right call when you point a tool at a folder full of files you didn't write. It's the wrong call for your own pipeline, which expects to import its neighbours. The drill script and the task you run afterwards need different settings, and I'd used the same one for both.&lt;/p&gt;

&lt;p&gt;Without &lt;code&gt;-I&lt;/code&gt;, the chart rendered in 0.42 seconds. I compared the PNG with the one in the original workspace using &lt;code&gt;cmp&lt;/code&gt;, and they were identical, byte for byte. On this machine, the restored workspace can do its job.&lt;/p&gt;

&lt;p&gt;Then I took away the one thing the earlier section worried about. I ran the same command with &lt;code&gt;PATH&lt;/code&gt; pointing at an empty folder, so &lt;code&gt;qlmanage&lt;/code&gt; couldn't be found. It didn't print &lt;code&gt;FAILED&lt;/code&gt; and carry on, as I'd guessed from reading the code. It crashed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;FileNotFoundError: [Errno 2] No such file or directory: 'qlmanage'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;FAILED&lt;/code&gt; branch only runs when &lt;code&gt;qlmanage&lt;/code&gt; exists but produces no image. When the tool is missing entirely, &lt;code&gt;subprocess.run&lt;/code&gt; raises before the script gets that far. I'd read my own code and predicted the wrong failure. That's a small embarrassment and a fair summary of why the drill ends with running something rather than reasoning about it.&lt;/p&gt;

&lt;p&gt;I can't test a Linux machine from here, so I won't claim to know exactly how the pipeline behaves on one. What the empty-&lt;code&gt;PATH&lt;/code&gt; run shows is the shape of it: the files arrive intact, and the first real task stops on a tool the folder never contained.&lt;/p&gt;

&lt;h2&gt;
  
  
  How much the timing moves
&lt;/h2&gt;

&lt;p&gt;The 16:30 copy was bigger than the one I'd drilled at lunchtime: 642 files, 188.6 MB raw and 155.9 MB packed, mostly the drafts and charts written in between. I ran the final script on it three times in a row.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;run   inventory  export  restore  verify   wall clock
1     3.57s      4.63s   0.87s    0.10s    9.24s
2     3.38s      4.66s   0.89s    0.10s    9.11s
3     3.45s      4.64s   0.88s    0.09s    9.14s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Wall clock varied by 0.13 seconds across the three runs. Inventory moved the most, because it reads and hashes every file and depends on what else the disk is doing. Export was steady, because it's mostly compression. Restore and verify were steady too.&lt;/p&gt;

&lt;p&gt;The point of writing these down is the comparison, not the absolute number. All of it is local disk to local disk. A real move adds a network. As a rough guide, and this is arithmetic on assumptions rather than a measurement: at an assumed 100 Mbit/s, 155.9 MB of archives takes about 12.5 seconds to send (155.9 × 8 ÷ 100). At an assumed 10 Mbit/s upload, it's a little over two minutes. A vendor's own export tool could be much slower than either, and I have no number for that.&lt;/p&gt;

&lt;p&gt;None of those figures comes close to the time it would take to rebuild 59 skills, a font setup and a conversation history nobody saved. The clock isn't the bottleneck. The gaps are.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd tell anyone running their own
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4nv4w8dzwgtkku3osgpn.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4nv4w8dzwgtkku3osgpn.jpg" alt="four runs, three surprises" width="800" height="300"&gt;&lt;/a&gt;&lt;br&gt;
Run it on a different machine, or at least a different Python, from the one you wrote it on. Your migration target won't have your shell history.&lt;/p&gt;

&lt;p&gt;Treat "zero diffs" as the start of the report, not the end. Comparing what's in the archive against what was in the folder can only find problems inside the folder. The expensive problems are the ones outside it: symlinks, conversation history in &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/where-do-your-ai-conversations-live" rel="noopener noreferrer"&gt;some cloud engine&lt;/a&gt;, settings in a vendor's UI.&lt;/p&gt;

&lt;p&gt;Look at your links before you pack anything. &lt;code&gt;find . -type l -exec readlink {} \;&lt;/code&gt; lists every symlink and where it points. I hadn't run it once on this workspace before today, and it would have shown me the skills problem before I wrote a line of the drill. If most of the targets sit in one shared folder, as mine did, decide whether that folder is part of the workspace or part of the machine, and write the answer down.&lt;/p&gt;

&lt;p&gt;Write the outside list by hand and keep it in the repo. I have to admit the irony here. When I went to commit mine, &lt;code&gt;git rev-parse&lt;/code&gt; told me this workspace isn't a git repository at all. Its scripts, drafts and charts have no history except the files themselves, and the only record of the four runs above is the output I saved by hand. For a drill that's an inconvenience. For a migration it's a bigger question, because there's no remote copy to compare against and no log of what changed between 14:28 and 16:30. Putting the workspace under version control is the cheapest improvement this drill turned up, and it isn't one the script could have suggested.&lt;/p&gt;

&lt;p&gt;So: write the outside list by hand and keep it in version control. It's the part of the drill that forces you to name what your agents depend on. Mine was wrong in run 3 and the script couldn't have known.&lt;/p&gt;

&lt;p&gt;Make the doubtful lines visible instead of tuning them away. I nearly dismissed the scheduled-jobs line as noise. The job it should have pointed me to was sitting in the system crontab the whole time.&lt;/p&gt;

&lt;p&gt;Time it. My 7.99 seconds says nothing about your setup over a network, or about a vendor's export tool. It does give me a baseline to compare against next time, which is the whole point of a drill.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does and doesn't cover
&lt;/h2&gt;

&lt;p&gt;A drill like this addresses data-exit risk: it tells you, before anyone sends you a date, which parts of your workspace would come with you. It doesn't address software-continuity risk. Conductor's local app keeps working for now and winds down in early 2027, and having every file on your own disk doesn't change that. It doesn't address model-supplier risk either, since an agent that calls an outside model still depends on that provider. And it certainly doesn't fix the 62 links. It only made me look at them.&lt;/p&gt;

&lt;p&gt;Running agents on hardware you own, which is what KarmaBox sells, moves the deletion date into your hands. It doesn't move the skills folder into the workspace. That's still a decision someone has to make, and this drill is how I found out it needed making.&lt;/p&gt;

&lt;p&gt;If Conductor's 16-day email had landed in your inbox this morning, which line of your gap report would you not have known to write?&lt;/p&gt;

&lt;p&gt;Sources&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://techcrunch.com/2026/11/04/meta-acquires-ai-coding-startup-conductor/" rel="noopener noreferrer"&gt;TechCrunch&lt;/a&gt;, Meta acquires AI coding startup Conductor, 4 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://news.ycombinator.com/item?id=50369503" rel="noopener noreferrer"&gt;Hacker News&lt;/a&gt;, Conductor is joining Meta (discussion quoting Conductor's announcement dates), 4 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://docs.python.org/3/library/os.html" rel="noopener noreferrer"&gt;Python Software Foundation&lt;/a&gt;, os — Miscellaneous operating system interfaces (os.walk), accessed 5 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://docs.python.org/3/library/tarfile.html" rel="noopener noreferrer"&gt;Python Software Foundation&lt;/a&gt;, tarfile — Read and write tar archive files (dereference parameter), accessed 5 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=tar&amp;amp;sektion=1" rel="noopener noreferrer"&gt;FreeBSD Project&lt;/a&gt;, tar(1) manual page (bsdtar; -L, --dereference), accessed 5 November 2026. &lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://peps.python.org/pep-0706/" rel="noopener noreferrer"&gt;Python Software Foundation&lt;/a&gt;, PEP 706 – Filter for tarfile.extractall, created 9 February 2023. &lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>devops</category>
      <category>productivity</category>
    </item>
    <item>
      <title>Designing a real AI kill switch: capability absence vs middleware gates</title>
      <dc:creator>KarmaBox</dc:creator>
      <pubDate>Wed, 04 Nov 2026 06:09:20 +0000</pubDate>
      <link>https://dev.to/karmabox/designing-a-real-ai-kill-switch-capability-absence-vs-middleware-gates-14pg</link>
      <guid>https://dev.to/karmabox/designing-a-real-ai-kill-switch-capability-absence-vs-middleware-gates-14pg</guid>
      <description>&lt;p&gt;"Which guardrail framework should I use?" is the wrong question for a kill switch. The best interceptor in the world still intercepts something that exists.&lt;/p&gt;

&lt;p&gt;Quick disclosure before the code: I work on KarmaBox, where agent permissions are enforced exactly the way this article recommends, so I'm arguing my own book. Three things keep this checkable: the pattern below is ~40 lines of dependency-free Python you can run and attack yourself; every news fact carries its outlet and date; and the last section lists what this pattern does not solve — including the part my employer hasn't solved either.&lt;/p&gt;

&lt;p&gt;Why this is worth your evening this particular week: on November 3, Google announced that schools disabling its AI features lose Google Search entirely, because — per its education VP — "Search and AI are deeply integrated" (&lt;a href="https://techcrunch.com/2026/11/03/google-will-let-schools-disable-its-ai-tools-but-not-without-sacrificing-search/" rel="noopener noreferrer"&gt;TechCrunch, November 3, 2026&lt;/a&gt;). Set aside the politics. As engineers, read it as an architecture confession: they built the off switch into a layer where flipping it takes down the building. This article is about not building that.&lt;/p&gt;

&lt;h2&gt;
  
  
  The definition, before the code
&lt;/h2&gt;

&lt;p&gt;A real kill switch removes a capability. It does not hide it, intercept it, or instruct the model to decline it — it makes the capability unreachable, for every caller, through every path. If "off" is implemented anywhere the capability can still be invoked, what you have is a gate: a thing that says no while the wiring behind it stays live.&lt;/p&gt;

&lt;p&gt;The distinction matters because gates and absences fail in opposite directions. A gate that crashes, gets misconfigured, or gets argued past fails open — the capability runs. An absence cannot fail &lt;strong&gt;open&lt;/strong&gt;, because there is nothing on the other side of the switch to run. At 2 a.m., that's the whole difference.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two architectures for "off"
&lt;/h2&gt;

&lt;p&gt;Most agent stacks implement "off" as middleware: the tool stays registered, and a policy layer intercepts calls.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# Architecture A: middleware gate
agent → tool_call("send_email") → policy middleware → [allow? deny?] → send_email()

# Architecture B: registry absence
agent → tools_for(role)  →  send_email was never in the list
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In Architecture A, &lt;code&gt;send_email&lt;/code&gt; exists in the agent's tool list. The model can see it, plan around it, and attempt it; a middleware decides at call time. In Architecture B, the tool surface is assembled per role from a registry, and a capability that is off was simply never assembled in. The model cannot call what it cannot see, and neither can a prompt injection riding in on a document.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0a23l1ti83t718z1bvay.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0a23l1ti83t718z1bvay.jpg" alt="two ways to build &amp;quot;off&amp;quot;" width="800" height="370"&gt;&lt;/a&gt;&lt;br&gt;
The failure-mode comparison, side by side:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc7mnjzgiqnuteulh8zgh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc7mnjzgiqnuteulh8zgh.png" alt=" " width="800" height="544"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;One row deserves emphasis because this week made it concrete: &lt;strong&gt;the audit story&lt;/strong&gt;. Google's announcement also claims schools can toggle each service individually — but two months from now, the only way a district can verify what was actually off is whatever the console shows. A switch you can't independently audit is a promise with a UI. Registries make the audit cheap: the assembled list is the ground truth.&lt;/p&gt;

&lt;p&gt;The pattern in ~40 lines&lt;/p&gt;

&lt;p&gt;Dependency-free, three permission tiers: &lt;code&gt;direct&lt;/code&gt; (runs), &lt;code&gt;approval&lt;/code&gt; (runs after a human approves), &lt;code&gt;absent&lt;/code&gt; (not assembled at all).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;# registry.py — capability registry with three tiers
from dataclasses import dataclass
from typing import Callable

TIERS = ("direct", "approval", "absent")

@dataclass(frozen=True)
class Capability:
    name: str
    fn: Callable
    tier: str  # one of TIERS

class Registry:
    def __init__(self):
        self._caps: dict[str, Capability] = {}

    def register(self, name, fn, tier):
        if tier not in TIERS:
            raise ValueError(f"unknown tier: {tier}")
        self._caps[name] = Capability(name, fn, tier)

    def tools_for(self, role_policy: dict[str, str]) -&amp;gt; dict[str, Capability]:
        """Assemble the tool surface for one role.
        A capability whose tier is 'absent' is NOT in the result.
        There is no gate to bypass — it simply isn't mounted."""
        surface = {}
        for name, tier in role_policy.items():
            cap = self._caps.get(name)
            if cap is None or tier == "absent":
                continue  # never mounted
            surface[name] = Capability(name, cap.fn, tier)
        return surface

    def audit(self, role_policy: dict[str, str]) -&amp;gt; list[tuple[str, str]]:
        """Export the effective capability list — this is the audit artifact."""
        mounted = self.tools_for(role_policy)
        return sorted(
            (name, mounted[name].tier if name in mounted else "absent")
            for name in role_policy
        )

def dispatch(surface, name, approver=None, **kwargs):
    cap = surface.get(name)
    if cap is None:
        raise LookupError(f"{name}: not a capability of this role")  # fail closed
    if cap.tier == "approval":
        if approver is None or not approver(name, kwargs):
            raise PermissionError(f"{name}: approval required, none given")
    return cap.fn(**kwargs)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notes on the load-bearing lines. &lt;code&gt;tools_for&lt;/code&gt; is where "off" happens: an &lt;code&gt;absent&lt;/code&gt; capability is skipped at assembly, so the model's tool list — the thing you serialize into the API call — never contains it. &lt;code&gt;dispatch&lt;/code&gt; fails &lt;strong&gt;closed&lt;/strong&gt; on anything outside the surface: an unknown or absent tool raises, it doesn't fall through to a default handler. And &lt;code&gt;audit&lt;/code&gt; exists because the Google story's quiet lesson is that consoles and reality drift; your auditor gets the assembled list, not a screenshot of settings.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three tests, and what it means when each fails
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;def test_absent_is_unreachable(registry):
    surface = registry.tools_for({"send_email": "absent"})
    assert "send_email" not in surface
    try:
        dispatch(surface, "send_email", to="x@y.z")
        assert False, "dispatch must raise"
    except LookupError:
        pass
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;If this fails:&lt;/strong&gt; your "off" is a gate, not an absence — somewhere a fallback path still resolves the tool. This is the test that separates the two architectures; everything else is decoration.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fba4kr42mi3ds6nhmzo7b.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fba4kr42mi3ds6nhmzo7b.jpg" alt="failure direction is the whole design" width="800" height="350"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;def test_approval_blocks_without_human(registry):
    surface = registry.tools_for({"post_public": "approval"})
    try:
        dispatch(surface, "post_public", body="hi")  # no approver passed
        assert False, "must require approval"
    except PermissionError:
        pass
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;If this fails:&lt;/strong&gt; your approval tier silently degraded into the direct tier — the exact drift the middleware column of the table predicts. Expect to find a "temporary" default-approver someone added during an incident.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;def test_audit_matches_surface(registry, role_policy):
    audit = dict(registry.audit(role_policy))
    surface = registry.tools_for(role_policy)
    for name, tier in audit.items():
        assert (name in surface) == (tier != "absent")
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;If this fails:&lt;/strong&gt; your audit artifact lies. That's worse than having no audit, because people will trust it. Fix this before fixing anything it reports.&lt;/p&gt;

&lt;h2&gt;
  
  
  The audit endpoint
&lt;/h2&gt;

&lt;p&gt;Expose &lt;code&gt;registry.audit(role_policy)&lt;/code&gt; as an internal endpoint or a CLI command and snapshot it on every deploy. Go one step further and make the snapshot a CI artifact: a job that diffs the assembled surface against the last release and fails the build on any unreviewed widening — a capability moving from &lt;code&gt;absent&lt;/code&gt; to anything, or a new tool appearing in a role. Narrowing can auto-pass; widening needs a human's name on it. Now "what could the agent do on March 12" is a &lt;code&gt;git log&lt;/code&gt; question, not an archaeology project. When a vendor tells you a capability "can be disabled," this is the artifact to ask them for — the effective capability list, not the settings screenshot. (For schools asking Google that question right now: per the November 3 reporting, the answer for Search-AI is that no such list will ever show it off while Search is on. The wiring is the policy.)&lt;/p&gt;

&lt;p&gt;One more direction worth knowing about, published the same day: Anthropic released FlashProve, open-source research toward cryptographic proof of which model produced an output — fingerprints published per model, no weights revealed, verifiable locally (&lt;a href="https://techcrunch.com/2026/11/03/anthropic-is-working-on-a-system-to-prove-ai-models-identity/" rel="noopener noreferrer"&gt;TechCrunch, November 3, 2026&lt;/a&gt;). Registry audit tells you which capabilities were mounted; provable inference points at the next layer — proving which model sat behind them. Research status, no deployments; but that's the end state audits are crawling toward.&lt;/p&gt;

&lt;h2&gt;
  
  
  Migrating an existing stack: the strangler path
&lt;/h2&gt;

&lt;p&gt;Nobody gets to rewrite their agent platform this sprint, so here is the incremental path we'd actually recommend — the classic strangler pattern, applied to tool surfaces. Four stages, each shippable on its own.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stage one: inventory the gates.&lt;/strong&gt; Grep for wherever tool calls get intercepted today — middleware, decorators, the if-statement graveyard inside your dispatcher. Each interception point is a capability that believes it has an off switch. List them with their current default behavior on error: you will find at least one that fails open, and finding it is the first win of the migration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stage two: wrap, don't replace&lt;/strong&gt;. Introduce the registry in front of the existing gates. The registry assembles the tool surface; the old gates stay exactly where they are, now guarding capabilities that were already filtered upstream. Nothing can break, because nothing was removed — you've added a second, stricter filter.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stage three: dual-run with a mismatch alarm.&lt;/strong&gt; For a few weeks, log every case where the old gate blocked something the registry had mounted, or — the interesting direction — where the registry excluded something the gate would have allowed. Each mismatch is either a policy bug or an undocumented dependency. We'd treat the second kind exactly like the accounting-filters discovery in this week's sysadmin threads: evidence of load-bearing coupling nobody wrote down.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stage four: flip per role, then delete&lt;/strong&gt;. Move one role at a time to registry-only enforcement, lowest-risk role first. Delete the gates last, not first — dead code that can't be reached through the registry is annoying; a gate deleted before the registry covered it is an incident.&lt;/p&gt;

&lt;p&gt;Two objections that always come up, answered honestly. Latency: effectively none — assembly happens once at session start, not per call; tools_for is a dictionary filter. Operational cost: yes, changing a capability now requires a deploy instead of a config flip. That is not a drawback to minimize; it is the entire point. A capability change that produces a diff, a reviewer, and a rollback path is the audit trail — the config flip that skips all three is how the middleware column of the failure table happens.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this pattern does not solve
&lt;/h2&gt;

&lt;p&gt;Honesty section, including against my employer. The registry pattern does not solve &lt;strong&gt;judgment&lt;/strong&gt;: deciding which capability belongs in which tier is a human call, and a wrong call with perfect enforcement is still wrong. It does not solve &lt;strong&gt;data already sent&lt;/strong&gt;: a capability you unmount today did whatever it did last quarter. It does not solve &lt;strong&gt;indirect effects&lt;/strong&gt;: a &lt;code&gt;direct-tier&lt;/code&gt; tool can still be misused within its scope (a search tool can exfiltrate by querying; the registry won't see it). And it does not give you &lt;strong&gt;cryptographic verification&lt;/strong&gt; — my audit endpoint is logs you have to trust; the FlashProve direction is the eventual fix, and nobody ships it yet, including &lt;a href="https://www.karmaboxai.com/en/concept/factory/resources/private-ai-capability-paradox" rel="noopener noreferrer"&gt;the product I work on&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The question worth taking back to your own stack: for each tool your agent can currently see — if the policy service died at 2 a.m., which way does it fail?&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://techcrunch.com/2026/11/03/google-will-let-schools-disable-its-ai-tools-but-not-without-sacrificing-search/" rel="noopener noreferrer"&gt;TechCrunch&lt;/a&gt; (Aisha Malik), Google will let schools disable its AI tools, but not without sacrificing Search, November 3, 2026&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://techcrunch.com/2026/11/03/anthropic-is-working-on-a-system-to-prove-ai-models-identity/" rel="noopener noreferrer"&gt;TechCrunch&lt;/a&gt; (Russell Brandom), Anthropic is working on a system to prove AI models' identity, November 3, 2026 FlashProve is research with published code, no deployment.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Code in this article: illustrative, dependency-free, written for this post — run it, break it, tell me where.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>architecture</category>
      <category>security</category>
      <category>python</category>
    </item>
  </channel>
</rss>
