The latest articles on DEV Community by belcore (@belcore). https://dev.to/belcore https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4234291%2Fb08c59de-7fcf-4a57-bd51-376a88ed6fb3.png https://dev.to/belcore en belcore Mon, 09 Nov 2026 01:42:55 +0000 https://dev.to/belcore/token-savings-depend-on-what-you-count-three-numbers-from-the-same-benchmark-runs-2pa1 https://dev.to/belcore/token-savings-depend-on-what-you-count-three-numbers-from-the-same-benchmark-runs-2pa1 <p><em>Disclosure: I'm affiliated with Belcore, a memory and context layer for LLM apps. This is a measurement post. The numbers, their limits, and one result that did not hold up are all below.</em></p> <h2> Why "tokens saved" is slippery </h2> <p>Chat and agent setups usually keep context by resending history on every call, so input tokens per call grow with the session. A memory layer replaces that with a selected context. How large the saving looks depends on what you compare, so here are the same runs counted three ways.</p> <h2> Setup </h2> <ul> <li>Dataset: LongMemEval_S, 500 questions, about 109k tokens of prior conversation each (roughly 500 turns).</li> <li>Answer model: gpt-5. Fixed seed, frozen harness.</li> <li>Tokenizer for context counts: o200k_base.</li> <li>Runs: August 2026.</li> </ul> <h2> Three numbers </h2> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>What is counted</th> <th>Tokens per call</th> </tr> </thead> <tbody> <tr> <td>Full transcript, uncut</td> <td>109,079</td> </tr> <tr> <td>Assembled memory context</td> <td>6,671</td> </tr> <tr> <td>Billed input per call (27-question subset, includes prompt and question)</td> <td>9,908</td> </tr> </tbody> </table></div> <ul> <li>Context tokens: <strong>-93.9%</strong>.</li> <li>Billed input on that subset: about <strong>-91%</strong>. The subset was hand-picked, so treat this one as indicative only.</li> </ul> <h2> What this does not show </h2> <ul> <li>Nothing here says anything about answer quality. Accuracy is a separate metric and I'm not quoting one in this post.</li> <li>Cost per successful task, including retries and fixes, is the number that matters for a real bill. I haven't measured it yet.</li> <li>LongMemEval_S is a public benchmark, not production traffic.</li> </ul> <h2> A result that did not hold up </h2> <p>We tried a multi-pass retrieval step that re-checks and re-retrieves before answering. On 27 questions hand-picked for missing evidence it went from 16 to 20 correct. On a random sample of 103 the attributable effect was +2, inside a ±5.2 noise bar, with one case where extra evidence turned a correct aggregation answer wrong. It also cost about 2.9x the input tokens and 2.6x the latency on the subset. We are not shipping it as an improvement.</p> <p>Two more limits: in the wrong answers we audited, 38% involved a gold label that was wrong or defensible either way, and n = 499 cannot resolve small effects.</p> <h2> Try it </h2> <p>There is a free measurement demo on the site: <a href="https://belcore.xyz/?utm_source=devto&amp;utm_campaign=token-counting" rel="noopener noreferrer">belcore.xyz</a></p> <p>Questions about the method are welcome, especially the ones that make the numbers look worse.</p> ai llm agents programming