Computer-use agents pay full frontier inference prices to work out routines their user has already performed hundreds of times, and a paper posted August 6 argues the reason is a design choice nobody examined: an agent's memory records what the user said, not what the user did. Nossa Iyamu's proposed fix is a compiler with no model in it, which turns passively captured screen activity into a context block 86 times smaller than the raw capture in 68 milliseconds - and an agent reading that block answers questions about the day with 98.4 percent accuracy against an independent oracle, versus 66 to 80 percent for a language-model summary of the same material.
Key facts
- The compiler is deterministic and model-free, so identical input produces byte-identical output that can be cached and audited.
- Evaluated on one professional's corpus of 128,756 activity frames across 51 active days.
- A compiled routine can be replayed with the model out of the loop entirely, demonstrated live at zero model tokens on a guard-matched hit.
- Primary source: Activity Frames (arXiv 2608.05784), submitted August 6, 2026, released under CC BY 4.0.
The framing is the contribution. Everything currently filed under agent memory is a record of conversation: what you asked, what the agent replied, maybe some extracted facts about your preferences. Meanwhile the actual evidence of how you work - which applications you open in what order, which sites you visit to accomplish which task, how long each step takes - is sitting in a screen-capture stream that nothing reads. Iyamu's argument is that the second stream is the more useful one for delegation, because it describes the routine rather than the request.
The pipeline segments that stream into what the paper calls activity frames: bounded episodes carrying the application, the site, the timing, the input volume, and evidence pointers back to the raw rows they were derived from. No model participates. That constraint is the point, not an optimization. As the paper puts it, the output is "byte-identical, cacheable, and mechanically auditable." Ask a language model to summarize your day and you get a fluent paragraph that will be slightly different tomorrow, with no way to check any specific claim in it against the source. Compile it with fixed rules and you get a structure where every line points back at the rows that produced it, and rerunning the compiler produces exactly the same bytes.
The accuracy comparison is where the argument gets teeth. The compiled block wins by roughly twenty to thirty points over an LLM summary of the same capture, and the paper notes that a mid-tier model reading the compiled block matches a frontier model doing the same. That is a familiar shape in this field, and a persistently underrated one: fixing the representation buys more than upgrading the model. The reported figure carries a Wilson 95 percent confidence interval of 91.7 to 99.7 percent, which is the correct way to report an accuracy from a sample this size and is more statistical honesty than most agent papers offer.
The compiler has a second use the paper takes seriously. Read against passive human activity rather than agent rollouts, it produces two numbers that cost models for agents assume but, the author says, nobody has measured: how much overhead an agent incurs re-deriving a routine, and how often routines actually recur. The reported routine overhead ratio - a modeled upper bound - lands between 60 and 343 times, with delegable recurrence at 9.0 percent in-sample and 7.7 percent out-of-sample, implying a realistic ceiling near 8 percent of fleet tokens. Translated: the fraction of an agent's work that is genuinely repeat business is smaller than the delegation pitch implies, but the waste on each instance of it is enormous.
The replay result closes the loop. Because the compiled representation is deterministic, it can emit a replay script that reruns a routine with no model call at all - demonstrated, the paper says, live at zero model tokens on a guard-matched hit. That is the difference between an agent that remembers and an agent that has learned a macro.
The caveats are large and stated plainly. This is n equals one: a single professional's capture over 51 days. The overhead ratio is a modeled upper bound, not a measured cost saving. And the whole approach assumes continuous screen capture on the user's machine, which is a privacy posture many people and most employers will refuse outright - the paper's own emphasis on local capture and evidence pointers reads as an answer to that objection, but it is an answer, not an exemption. Schema, compiler, and evaluation harness are open.
What makes it worth reading anyway is the direction it pushes. The field's answer to "what should an agent remember" has mostly been more context and better retrieval. This is an argument that the answer is a different data type entirely: not a transcript of the conversation, but a compiled, checkable record of the work.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)