What I Benchmarked
I tested whether the runtime around the model holds up: retrieval endurance, memory scaling, sandbox containment, context behavior. I used a Gemtek NUC M6 Ultra mini-PC on Pop!_OS under zRAM swap. I wanted to benchmark sessions that run for days.
Models Tested
I chose the models I actually run locally. Same runtime, same tests, three price points: a local 1.2B triage model, a local 8B-class model, and a cloud workhorse. I ran the work with agent sessions and verified every number myself, at least three ways, before it made this post. The tier setup turned out to be its own finding (#5).
Findings
Memory systems have a hang threshold. I figured out, it's a file count and not bytes. Near ~1,500 files per store, my context assembly stalled. I partitioned into 80 bounded stores: 2.6ms/file, linear, largest store 38% under the hang line — 3.1× better than the monolith.
I look at containment as a score instead of a bit. "Did the sandbox kill the fork bomb?" — a yes/no answer doesn't tell the full story. So, I tried scoring blast radius: MB of memory pressure, PID-seconds survived, host RSS delta under zRAM.
After some trial and error, I pinpointed that facts sitting mid-context were measurably less likely to be used than facts at the head or tail — the U-shaped recall curve, confirmed on my own documents. I now write every internal document conclusions-first, receipts-last. It helps because I have very dense documents that I'm processing so agents can get enough info to understand the context before searching deeper into the docs.
I run 10-day sessions and log latency drift, memory growth, and retrieval decay nightly — 30+ days continuous so far. My agents don't fail at t=0; they rot.
Only ~10% of my agent tasks needed the top tier. A ~30× cost spread across tiers, no measurable quality loss on the other 90% — as long as the runtime routes instead of defaulting. I measure cost per completed task instead of cost per token.
My Benchmark
The Runtime IS the Benchmark — minimal notebook
Everything above re-runs from it.
Top comments (1)
tr.ee/dev-to