The latest articles on DEV Community by ab371 (@ab371). https://dev.to/ab371 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3260199%2F91f75887-c76a-406d-9daf-501c97f5c7a3.png https://dev.to/ab371 en ab371 Mon, 12 Oct 2026 02:14:13 +0000 https://dev.to/ab371/the-runtime-is-the-benchmark-endurance-metrics-for-local-agents-3kb6 https://dev.to/ab371/the-runtime-is-the-benchmark-endurance-metrics-for-local-agents-3kb6 <h2> What I Benchmarked </h2> <p>I tested whether the runtime around the model holds up: retrieval endurance, memory scaling, sandbox containment, context behavior. I used a Gemtek NUC M6 Ultra mini-PC on Pop!_OS under zRAM swap. I wanted to benchmark sessions that run for days.</p> <h2> Models Tested </h2> <p>I chose the models I actually run locally. Same runtime, same tests, three price points: a local 1.2B triage model, a local 8B-class model, and a cloud workhorse. I ran the work with agent sessions and verified every number myself, at least three ways, before it made this post. The tier setup turned out to be its own finding (#5).</p> <h2> Findings </h2> <ol> <li><p>Memory systems have a hang threshold. I figured out, it's a file count and not bytes. Near ~1,500 files per store, my context assembly stalled. I partitioned into 80 bounded stores: 2.6ms/file, linear, largest store 38% under the hang line — 3.1× better than the monolith.</p></li> <li><p>I look at containment as a score instead of a bit. "Did the sandbox kill the fork bomb?" — a yes/no answer doesn't tell the full story. So, I tried scoring blast radius: MB of memory pressure, PID-seconds survived, host RSS delta under zRAM.</p></li> <li><p>After some trial and error, I pinpointed that facts sitting mid-context were measurably less likely to be used than facts at the head or tail — the U-shaped recall curve, confirmed on my own documents. I now write every internal document conclusions-first, receipts-last. It helps because I have very dense documents that I'm processing so agents can get enough info to understand the context before searching deeper into the docs. </p></li> <li><p>I run 10-day sessions and log latency drift, memory growth, and retrieval decay nightly — 30+ days continuous so far. My agents don't fail at t=0; they rot.</p></li> <li><p>Only ~10% of my agent tasks needed the top tier. A ~30× cost spread across tiers, no measurable quality loss on the other 90% — as long as the runtime routes instead of defaulting. I measure cost per completed task instead of cost per token.</p></li> </ol> <h2> My Benchmark </h2> <p><a href="https://www.kaggle.com/code/antoniobedford/the-runtime-is-the-benchmark" rel="noopener noreferrer">The Runtime IS the Benchmark</a> — minimal notebook</p> <p>Everything above re-runs from it. </p> devchallenge kagglechallenge ai machinelearning