DEV Community: Codecora Dev

Uteke 0.18: memory that knows when

Anaz S. Aji — Tue, 15 Sep 2026 02:31:12 +0000

Uteke 0.18.0 lets a memory carry the date it was true, gives rooms a real lifecycle, and fixes a recall default that silently filtered almost every result.

Uteke 0.18.0 shipped on September 14, 2026. Uteke is a local-first memory engine for AI agents: a single Rust binary, no API keys, no cloud, Apache 2.0. It has 246 stars on GitHub as of this writing. The theme of this release is plumbing you can operate: when a memory was true, where it lives, and what a recall call owes you back.

The problem: every memory borrows the clock it arrived on

Until 0.18, a memory's timestamp was the moment it entered the store. That sounds fine until you import a month of notes in one batch, or migrate from another tool, or write down today a decision that was made in March. Time-travel recall (recall --at), recency boosts, and audit chains all read that ingest time, so a backfilled import looked brand new and an old decision looked older than it was. Import order quietly became part of your recall results.

Date anchors: write down when it was true

The headline feature is an explicit timestamp at write time:

uteke remember "Ship window moved to March 15" --timestamp "2026-03-15T09:00:00Z"

The same field exists on batch import, so a folder of backfilled notes can carry its real dates in one pass. Time-travel recall, temporal boosts, and audit chains now read the anchor instead of the ingest clock. A memory written today about March behaves like a memory from March, wherever it lands in your import queue.

One honesty note: the LMEVAL_DATE_ANCHOR environment variable mentioned in the changelog is benchmark-harness-only. It never affects the binary you run.

Rooms grow up: rename, update, move

Rooms, Uteke's per-project memory partitions, now have lifecycle operations on every surface: uteke room rename, uteke room update, and uteke room move-memory, with matching HTTP routes (POST /room/rename, /room/update, /room/memory/move) and MCP tools. A rename rewrites the registry and every room reference in a single transaction, so a half-renamed state is impossible. Rooms can also carry a description now (schema v19, additive, backward-compatible exports).

For individual memories there is uteke update <id>: edit content, tags, importance, pinned state, or type in place. The HTTP PUT /memory route and the MCP uteke_update tool could always do this. The CLI could not. Now it can.

The default threshold that filtered almost everything

This is the fix worth explaining in full. Since 0.16.0, the default recall strategy has been fusion (weighted RRF), and its scores are rank-based: an RRF contribution plus salience and recency boosts, typically between 0.0 and 0.2. The CLI's default min_score was still 0.3, a value from the cosine era, when scores clustered above 0.5. HTTP and MCP already defaulted to 0.0.

The result: a fresh install running plain uteke recall got empty output. We measured it on the embeddinggemma-q4 embedder. All 20 paraphrase probes scored between 0.169 and 0.186 at rank 1, and every one fell below a threshold built for a different scoring scale.

0.18 lowers the default to 0.0, and the docs now state the score scale per strategy. Thresholds still exist for people who want them: [recall] min_score in config, --min on the CLI, --strict (0.5), or min_score over HTTP. The tool should filter when you ask it to, not as a side effect of history.

A contract for recall payloads

Version 0.18 also pins the recall payload contract with conformance tests across all three surfaces: CLI, HTTP, and MCP. A recall hit carries its full payload, never a hit-count stub. If a future optimization breaks that promise, a test fails instead of a benchmark drifting.

Benchmarks: same numbers, a clearer home

Retrieval behavior is unchanged in 0.18.0. We revalidated at the published config: recall@5 on the non-abstention split is 0.9457, with identical per-question rankings to 0.17.0. What changed is where the numbers live. The benchmarks/ directory is now the single source of truth, with an index in benchmarks/README.md and committed raw artifacts for LongMemEval. A stale results stub that still carried pre-embedding figures is gone, and a mislabeled baseline is fixed: the 0.854 and 0.885 full-500 figures published in August came from the vector-only run, and the results file now says so on every aggregate row.

Upgrade

UTEKE_VERSION=v0.18.0 curl -fsSL https://raw.githubusercontent.com/codecoradev/uteke/main/install.sh | sh

Already running Uteke? uteke upgrade replaces the CLI only. After upgrading, check that uteke, uteke-serve, and uteke-mcp all report 0.18.0, since the server and MCP binaries come from the release bundle.

Recall could already show its work after 0.17. Now it can also tell you when, and move where you tell it. Full changelog and release notes: github.com/codecoradev/uteke.

This article is crossposted with blog.codecora.dev as the canonical source.

Half a Million Tokens of Context Still Needs an Index

Anaz S. Aji — Thu, 10 Sep 2026 02:41:44 +0000

A thread on r/LocalLLaMA ran Qwen 3.8 27B with a 524,000-token context window on two RTX 3090s this week, holding 60 to 88 tokens per second. Two consumer GPUs, half a million tokens of context, one box. Five years ago that sentence would have been science fiction.

Every milestone like this revives the same argument: long context kills RAG. If the model can read everything, why build an index? The "RAG Is Simpler Than You Think" discussion on Hacker News (515 points, 218 comments, August 26) pulled the debate back to earth. The interesting question was never retrieval versus context. It is which layer should carry which cost.

What half a million tokens cost per query

Transformer serving is linear in context. Prefill reads the whole prompt, the KV cache keeps keys and values for every token resident in VRAM, and each generated token re-reads all of it. Double the window and you double the memory and the per-token work, on every query, even when the answer needed three paragraphs.

That is the deal the r/LocalLLaMA setup signed up for. Dual 3090s, one query stream, half a million tokens resident. It works, and it is heavy by design.

Retrieval moves the work off the hot path

A retrieval pipeline does the opposite. Build the index once. Per query, fetch the few chunks that matter and read only those. Cost per query scales with what you retrieve, not with what you own.

That is also why the "simpler than you think" argument landed. For lookup-shaped questions, and most agent workloads are lookup-shaped, a small index plus a small read beats carrying half a million tokens through every turn.

The layer nobody budgets for: embedding storage

If retrieval is the plan, the index becomes the artifact you ship, mirror, and back up. The storage math is unforgiving.

A 768-dimension f32 embedding is 3,072 bytes. A million chunks of corpus is about 3.07 GB before you write a single replica. On a server you shrug. On a phone, a robot, or an offline box, you are suddenly choosing between your index and your photos.

vecq is our answer to that layer: training-free scalar quantization (Lloyd-Max centroids, configurable 4/5/6-bit width, default 5-bit) with search built in. Numbers from docs/BENCHMARK.md, measured on real EmbeddingGemma vectors on an aarch64 dev box:

mode	bytes/vector	recall@10	ms/query
5-bit (default)	642	0.979	3.21
4-bit	514	0.958	0.89
4-bit + residual	1,028	0.984	1.76

At the default width, that million-chunk index drops from 3.07 GB to about 642 MB, 4.78x smaller, with recall 1.6 points from full f32. When you need near-perfect first hits, residual mode holds recall@1 at 0.990 and remains the fastest high-recall scan. There is a third lever as well: Matryoshka-trained models such as EmbeddingGemma degrade gracefully when truncated, so you can quantize only the leading 256 dimensions of a 768-dim model before quantization ever starts.

Build is fast too. A 2k-vector index constructs in 75 ms against 893 ms for HNSW in our benchmark, because there is no graph to build. You quantize, persist a single file, and mmap it.

Trade-offs, honestly

vecq is brute force. There is no graph and no navigational approximation, so scan time grows with corpus size while HNSW stays roughly flat. At 2k vectors the 5-bit scan costs 3.21 ms per query against HNSW's 0.23. That gap is architectural, and it is why vecq targets edge, mobile, and single-box indexes instead of billion-vector server fleets. If you are serving a data center, use a server engine.

Long context genuinely wins some rounds too. One controlled test from a European bank, reported in a March 2026 comparison, put long context 34% more accurate on simple single-document queries. When every query really does need every page of one document, stuffing wins. The failure mode is treating a 524k window as the default container for everything. Cost per query scales with the window you carry, and most workloads are not shaped like that.

Reproduce the numbers

git clone https://github.com/codecoradev/vecq
cd vecq && git checkout develop
cargo run --release -p vecq-bench --bin widths   # full 4/5/6-bit + residual matrix
cargo run --release -p vecq-bench --bin real     # recall and latency vs exact cosine

Or skip the benchmark and use the crate:

cargo add vecq-core

If this is your first pass on vecq, start with Meet vecq, the lean vector layer. Repo: github.com/codecoradev/vecq. Crate: vecq-core on crates.io.

This article is crossposted with blog.codecora.dev as the canonical source.

Uteke in Production: Lessons from 150 GitHub Stars

Anaz S. Aji — Tue, 08 Sep 2026 09:16:55 +0000

Editor's note (Sep 8, 2026): the lessons below were written when Uteke passed 150 stars. It has since crossed 200.

Uteke in Production: Lessons from 150 GitHub Stars

Uteke crossed 150 GitHub stars in under two months. That is not viral growth, but it is enough to generate real feedback from real users running Uteke in production. Here are the patterns we have observed and the lessons we have learned.

How People Use Uteke

The most common use case is adding memory to AI agent frameworks: LangChain, CrewAI, and custom agent setups. Developers want their agents to persist knowledge across sessions without setting up a full vector database.

Self-hosted AI assistants with persistent conversation memory
Documentation chatbots for internal knowledge bases
Multi-agent systems using rooms for coordination
Research tools that accumulate findings over time

The Single Binary Advantage

Users consistently cite the single-binary deployment as a key reason for choosing Uteke over alternatives. No Docker, no Python environment, no external services. Download, run, done. This matters especially for developers embedding Uteke into their own applications.

Common Pain Points

The most frequent issue requests are around embedding model flexibility and backup/restore. Users want to bring their own embedding models instead of being locked into all-minilm, and they want straightforward database migration between versions.

Another pattern: users running Uteke on low-resource hardware (Raspberry Pi, small VPS) hit memory limits with large HNSW indices. We added configuration options for index parameters to help tune memory usage.

What Worked

Small API surface: Keeping the REST API to five core endpoints (remember, recall, search, list, forget) made Uteke easy to adopt. Users can integrate in under an hour.

Honest documentation: Being upfront about limitations (no GPU support, single-node only, no clustering) set the right expectations and reduced disappointed issue reports.

Fast releases: Publishing fixes within hours of bug reports built trust. Users see the project as maintained and responsive.

What We Would Do Differently

We would have added the Room system earlier. The flat namespace worked for single-agent use cases, but multi-agent coordination requests started appearing in the first week. Shipping rooms sooner would have reduced the 'works great, but...' feedback.

We also underestimated the demand for language-specific client libraries. The REST API is straightforward, but developers want SDKs in Python, Go, and TypeScript. Building these earlier would have lowered the integration barrier.

Looking Ahead

The next priorities are plugin-based embedding model support, improved backup tooling, and client libraries for the most requested languages. The core architecture (SQLite, HNSW, single binary) stays the same. These additions extend the surface area without changing what makes Uteke work.

Star us on GitHub if you are using Uteke in your project.

This article is crossposted with blog.codecora.dev as the canonical source.

Your Local LLM Isn't Dumb. It's Amnesic.

Anaz S. Aji — Tue, 08 Sep 2026 09:16:48 +0000

You downloaded a model everyone praised. You ran it locally. And within ten minutes you thought: this thing is dumb.

A detailed technical writeup that hit the Hacker News front page this week (417 points, "Why your local LLM feels dumber than it is") makes a case that's equal parts uncomfortable and liberating: your local implementation sucks, but everyone else's does too. Quantization, mixed GPU generations, inference stack mismatches: every local setup silently degrades the benchmark numbers the lab published.

So part of the fix is honest stack-tuning. But that's only half the story.

The other half: your model forgets everything

Even a perfectly configured local LLM has a disability the benchmarks never measure: it starts every single session from zero.

No memory of yesterday's debugging session. No recall of the architecture decision you explained last week. Every conversation begins with you re-uploading, re-pasting, re-explaining, burning tokens to reconstruct context the model already "learned" once.

That's not a dumb model. That's an amnesic one. And amnesia looks like dumbness:

It "doesn't understand your codebase" → because it re-reads it cold, every time
It "forgets your conventions" → because it literally does
It "gives inconsistent answers" → because each session is a stranger

Proof the models are good enough

The same week, another front-page story: someone gave Qwen 3.8 27B a reverse-engineering job and it finished in 30 minutes. Work you'd normally assume needs a frontier API model.

A 27B model, running locally, doing frontier-grade work. The capability is there. What's missing between "amazing in the demo" and "frustrating in daily use" is usually context continuity, not raw intelligence.

Fix 1: Tune the stack (the article's point)

From the Level1Techs writeup, the parts you control:

Match quantization to hardware. A heavily quantized GGUF on mismatched silicon is not the model the lab benchmarked.
Use a sane inference runtime, not whatever shipped default with a one-click installer.
Stop judging models by two casual test prompts. The gap between lab conditions and your homelab is implementation, not intelligence.

Fix 2: Give the model memory (our point)

This is the fix almost nobody applies, and it compounds. Uteke is an open-source, Apache-2.0 memory engine built exactly for this: a single Rust binary, fully offline, SQLite + embedded HNSW vectors + on-device embeddings, with semantic recall in ~45ms. No API key, no Docker, no cloud. It also speaks MCP, so Claude Code and other MCP agents can use it as their persistent memory server.

The difference in practice:

Without memory: every session, your agent re-derives your project structure, your conventions, last week's decisions, in tokens you pay for and wait on.
With memory: the agent recalls that context in milliseconds and starts working.

It's the difference between hiring a brilliant contractor with amnesia every morning, and one who remembers your project.

The takeaway

If your local LLM feels dumb, you have two levers, and most people only pull one:

Fix the stack: quantization, runtime, realistic expectations.
Fix the memory: stop letting every session start from zero.

The models caught up. The stack is tunable. Memory is the remaining gap, and it's a solved problem.

Try it: Uteke on GitHub, one binary, zero dependencies, ~45ms recall, works with any MCP-capable agent.

This article is crossposted with blog.codecora.dev as the canonical source.

Uteke 0.17: memory that shows its work

Anaz S. Aji — Mon, 07 Sep 2026 09:45:38 +0000

Version 0.17 of Uteke shipped today, and the theme is explainability. Your local memory store should be able to show you why it returned what it returned, and what happened to the facts it replaced. Here is a quick tour of the release.

Explain mode for recall

Recall now has an explain mode. Alongside the results you get the full scoring breakdown: the vector rank, the FTS rank, how RRF fuses the two lists, and which boosts got applied along the way. The whole breakdown adds roughly 50ms on top of a normal query. So when a search result surprises you, you can check the actual scoring path instead of guessing.

Supersede ledger with undo

When a newer fact replaces an older one, 0.17 records the replacement in a supersede ledger. The history stays queryable, and mistakes stay recoverable: uteke contradictions undo reverses a supersede you regret. Anyone auditing the system gets a full trail of how each fact changed over time.

Provenance and trust tiers

Every entry now carries provenance metadata: a SHA-256 hash of its source plus a trust tier. Entries from shaky sources now rank differently from entries with verified origins, so the retrieval layer can weigh reliability instead of treating all memories as equal.

Contradiction benchmark

We stress tested the contradiction handling on 40 topics. After conflicting updates were ingested, winner@1 accuracy went from 0.850 to 1.000, and stale@5 dropped from 1.000 to 0.000 once supersede took effect. The benchmark script lives in the repo at benchmarks/longmemeval/contradiction_segment.py, so you can reproduce the numbers on your own machine.

Namespace management and pagination

Namespaces got proper management commands, and list endpoints now return pagination metadata. Building tooling on top of the API no longer means guessing how many pages are left.

Try it

Install in one line:

curl -sSL codecora.dev/uteke/install | sh

Source code is at github.com/codecoradev/uteke. The full release write-up, with more detail on everything above, is on the CodeCora blog: Uteke 0.17: memory that shows its work.

Agent memory should be a file. We built it.

Anaz S. Aji — Sun, 06 Sep 2026 02:35:22 +0000

Last Sunday, a Hacker News post argued that agent memory belongs in plain files. 190 points and 94 comments in two days. For a topic as unglamorous as "save notes to a text file," that is a lot of heat.

We agreed with the thread, because we had already shipped the thing it asked for. Uteke is a semantic memory engine for AI agents: one Rust binary, one SQLite file on your machine, recall in about 45 milliseconds, fully offline.

What the thread got right

Cal Paterson's argument: most agent memory systems are broken. Some lock you into a vendor. Some are absurdly heavy: pgvector, a graph database, and a separate LLM whose only job is deciding what deserves to be remembered. His fix is memory as data: markdown pages in a zip, plus an optional SQLite file for search.

Files are inspectable. You can cat a memory, diff two of them, put the directory in git. No vendor can deprecate your zip file.

Where plain files stop

Store "we deploy v2.1 to staging after the payment freeze" on a Tuesday. Three weeks later you ask your agent "when are we shipping?" Grep finds nothing, because the words "ship" and "release" never appear in the file. Grep does not do meaning. At five memories that is a curiosity. At five hundred, roughly two weeks of real agent work, it is the difference between memory and an archive.

So we built the file that answers questions.

What Uteke does

curl -sSL codecora.dev/uteke/install | sh
uteke remember "Deploy v2.1 to staging at 3pm"
uteke recall "when do we deploy?"

That third command is the one files cannot do. Recall runs hybrid search: SQLite FTS5 for keywords, vector similarity for meaning, fused into one ranked list. A local embedding model (about 188MB) downloads once. After that, nothing leaves your machine.

Numbers we publish:

98.2% recall_any@5 on LongMemEval-S (500 questions, zero-config fusion default, public harness in the repo)
42ms P50 / 50ms P95 recall at 10K memories, and latency stays flat from 100 to 10K

Uteke is Apache-2.0. The repo sits at 237 stars.

If you nodded along with that thread, those three commands are the whole evaluation. Your agent's memory stays a file you own. The file just answers questions now.

The full story is on the blog: Agent memory should be a file. We built it.

This article is crossposted with blog.codecora.dev as the canonical source.

A Small Transformer Trained in 1.5 Hours Beat Many LLMs on ARC

Anaz S. Aji — Wed, 02 Sep 2026 06:56:56 +0000

Mithil Vakde trained a small transformer from scratch in 1.5 hours on a rented RTX 5090. The compute cost came to about 67 cents. The model scored 44% on the ARC-AGI-1 public eval, one point behind TRM's 45% and in the same band that left DeepSeek R1, o3-mini, and Gemini 2.5 Pro behind when the TRM paper ran the comparison (arXiv 2510.04871).

This is the third post in his ARC series. The previous result went viral and drew public scrutiny from researchers including Lucas Beyer, Jeremy Howard, and Rohan Anil. The new one raises the score, cuts the cost, and answers every objection in a long appendix.

What the model does

Each puzzle becomes a sequence of tokenized grids. The transformer trains on those sequences from scratch at test time, using both the train puzzles and the eval puzzle inputs, with the eval labels hidden. Test-time training is not a loophole here. ARC is a meta-learning benchmark built around skill acquisition, and Chollet's original paper frames the whole thing as a measure of how efficiently a system gains new skills (arXiv 1911.01547).

The architecture got a 2026 refresh: 8 layers, SwiGLU, RMSNorm, the NorMuon optimizer, 3D RoPE, and per-task embeddings. Inference augments each test input with color and dihedral permutations, inverts the augmentations, and submits the two most common outputs.

The biggest change is the loss function. Previous versions trained on input and output tokens. This one trains on output tokens only, which makes the approach supervised. Score went from 40% to 44%. Vakde admits he does not fully understand why, and the training loss got worse even as scores improved. His guess is finite model capacity. The ablations back him up on what carries the result: drop the 3D RoPE and the score falls to about 24%, drop the per-task embeddings and it falls to about 24%. Representations, not compute, are doing the work.

The lineage it joins

None of these systems pretrain. They learn from the benchmark's roughly thousand examples:

System	Parameters	ARC-AGI-1	Source
CompressARC	76K	20%	arXiv 2512.06104
HRM	27M	40%	arXiv 2506.21734; score from 2510.04871
TRM	7M (see caveats)	45%	arXiv 2510.04871
Vakde's model	small (1.5 hrs training)	44%	author's blog

In about a year, from-scratch systems went from solving a fifth of the eval to nearly half of it, and the cost per result collapsed from research-cluster budgets to spare change on a rented GPU.

Vakde also measured what happens when you strip his advantages. Restricting training data to ARC-1 plus ConceptARC still scores about 40%. A CompressARC-style unsupervised variant drops to about 18%. He thinks 65% is reachable within the current framework, and a union of solved tasks across his own runs already reached 55%. One request to contributors: do not add more training data.

Where the skeptics have a point

The 67-cent figure deserves a careful read. It counts lifetime compute: training from initialization plus inference on all tasks, which works because every task is trained at once. Dividing that total by task count would amortize it unfairly against models charged per task, and Vakde now compares only against TRM, HRM, and CompressARC for exactly this reason. LLM cost figures, on the other side, leave out pretraining entirely, so no single cost axis treats every approach the same.

Parameter claims need the same scrutiny. He argues TRM's famous "7M parameters" leaves out O(100M+) trained embedding weights, and that "7M active weights" would be the honest label. Same caution for HRM.

The score itself stays humble. 44% means 56% of puzzles still fail, and 7% on ARC-2 is early days. OpenAI's o1 reached 75% on ARC-1 through massive post-training (ARC Prize 2024 report), and frontier systems still lead the private leaderboard. This result is about efficiency per training dollar, not about beating GPT-class models across the board. Public eval scores for LLMs are also polluted, since the answers circulate on the internet and end up in training data.

And this is one researcher's unreviewed work. The previous round got picked apart in public, and while the appendix answers the objections convincingly, independent replication is still thin.

Efficiency without the training bill

The pattern matters more to us than the puzzle. Small-model efficiency used to arrive with an invoice: distillation runs, quantization-aware training pipelines, weeks of compute before the first benchmark.

Vakde's result says the invoice is optional on the reasoning side: train from scratch at test time, in 1.5 hours, for 67 cents. Our own work says the same on the storage side. vecq, our training-free vector quantization library, gets its 4-bit and 5-bit codes from Lloyd-Max centroids that are fixed constants. There is no training pass to pay for at all.

In our benchmarks, 4-bit codes hit recall@10 of 0.958 at 5.98x compression, scanning at 0.89 ms per query on ARM, and the current release defaults to 5-bit at 4.79x compression with 0.979 recall@10. The same quantization math scales all the way up to an 87GB model on a CPU. Inside our memory engine, the pipeline reached 98.2% recall against ChromaDB and Mem0 without any learned quantizer. The brute-force scan does give back some speed to HNSW, and why we pick that trade is an architectural choice for edge hardware, not an accident.

What to watch

The interesting shift is the price of admission for credible efficiency results: no pretraining run, one GPU, under two hours, spare change. Chollet's skill-acquisition framing finally has a small-model track with real momentum behind it, and Vakde argues the organizers should go further and ban offline pretraining to make ARC a true sample-efficiency test. His own result is the existence proof that such a rule is survivable.

The author believes 65% is reachable and that hand-written GPU kernels could cut his costs another 10x. Either would have sounded like trolling a year ago.

References

Mithil Vakde, "44% on ARC-AGI-1 in 67 cents"
"Less is More: Recursive Reasoning with Tiny Networks"
"Hierarchical Reasoning Model"
"ARC-AGI Without Pretraining"
Francois Chollet, "On the Measure of Intelligence"
"ARC Prize 2024: Technical Report"

This article is crossposted with blog.codecora.dev as the canonical source.

Why bigger context windows won't kill memory systems

Anaz S. Aji — Wed, 02 Sep 2026 03:57:24 +0000

A claim keeps making the rounds: context windows are at a million tokens and climbing, so external memory is a stopgap that better models will make obsolete. If that's true, we picked a strange moment to publish a memory benchmark. We weren't worried, but we did want to know what the evidence says. So we read the papers.

What a million tokens does to information

The seminal result is Liu et al's "Lost in the Middle" (arXiv 2307.03172). Tested on multi-document QA and key-value retrieval, models performed best when relevant information sat at the beginning or end of the context, and degraded hard when it sat in the middle.

For a while you could hope that was an engineering wart. Then a 2026 paper derived the same U-shaped retrieval profile from a theoretical model of attention dynamics (arXiv 2605.09213). The position bias falls out of the math. Bigger windows inherit it. They don't remove it.

Practice matches theory. In a white-box study of coding agents, pass rate fell from 8 of 10 runs in a clean 10,991-character context to 3 of 10 runs at 299,140 characters, and padding the context with irrelevant material hurt as much as relevant code (arXiv 2607.17937). Half the drop had nothing to do with relevance.

Long-horizon search shows the same shape. Across four flagship models and three benchmarks, researchers documented "premature termination": models give up or answer with falling confidence long before they approach the window limit (arXiv 2606.29718).

The economics nobody mentions

Everything inside the window is re-paid on every call, in every session. A fact from last March costs the same as something the user said ten seconds ago, every single time you send it.

Retrieval inverts that curve. Store once, then pay for a small set of relevant records when they're needed.

There's a real trade-off inside memory design itself. Keep raw records and you preserve detail but burn budget. Consolidate and you cover more ground per token but risk dropping a query-critical fact (arXiv 2607.17545). Both strategies argue for the same thing: a small, deliberately chosen working set, not a firehose.

Memory is not a small context window

LongMemEval (arXiv 2410.10813) breaks long-term memory into five measurable abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Look at that list. None of them mean "hold more tokens at once."

Knowledge updates are the cleanest example. When a user changes deploy targets, the old value has to be overwritten. In a million-token window, the stale fact and the current one both sit there, and the model has to guess which wins. A memory system that tracks updates resolves it by design.

The field is still moving. LongMemEval-V2 landed in May 2026 and pushes evaluation toward whether an agent becomes an experienced colleague in its environment. Nobody builds benchmarks for problems they consider solved.

When big context wins

Credit where it's due. Reasoning over one large document you already have, one-shot analysis that needs the whole thing in view at once, needle-style lookups inside a single snapshot: if the world fits in the window and everything in it is current, fill it. Memory systems earn their keep when history is long and facts change across many sessions, which describes most real assistants after their first week.

Our numbers, reproducible

We build a memory engine (Uteke) and we test it the boring way. On LongMemEval-S with 500 questions, it scores 98.2% recall_any@5. Then we re-ran the entire suite on a different CPU architecture: 107 of 108 questions produced identical rankings, and the one that moved was an adjacent-rank near-tie we traced to float summation order.

The harness is in the repo. Run your own, and treat numbers you cannot reproduce, including ours, as marketing.

Verdict

Bigger windows raise the ceiling on working memory. Persistence, salience, and overwrite stay separate jobs, and the field is now measuring them directly. Context and memory do different jobs. The agents people keep coming back to are the ones that stopped starting from zero.

References

Lost in the Middle: How Language Models Use Long Contexts (arXiv 2307.03172)
Kinetic theory for Transformers and the lost-in-the-middle phenomenon (arXiv 2605.09213)
When and How Context Rot Appears in Coding Agents (arXiv 2607.17937)
Diagnosing and Mitigating Context Rot in Long-horizon Search (arXiv 2606.29718)
Retain or Consolidate? Budget-Dependent Operator Selection for Language Agent Memory (arXiv 2607.17545)
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (arXiv 2410.10813)
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues (arXiv 2605.12493)

Full write-up lives on our blog, canonical source for this post.

We re-ran our benchmark on a different CPU architecture. 107 of 108 matched.

Anaz S. Aji — Tue, 01 Sep 2026 14:06:41 +0000

We publish benchmarks for Uteke, our open-source memory engine. The headline this week: 98.2% recall on LongMemEval-S. 500 questions, zero config, no LLM in the retrieval path.

Then we asked the question every benchmark should answer: does the number survive on different hardware?

The published run came from cloud x86 infrastructure. The re-run happened on a 4-core ARM desktop. Same v0.16.0 binary, same public harness, 108-question subset.

We compared per question, ranking against ranking.

107 of 108 identical.

The one difference is the interesting part. Both runs retrieved the same top-10 sessions. One session sat at rank 5 in the published run and rank 6 in the re-run. Our retrieval fuses two rankings with Reciprocal Rank Fusion, which is floating-point math all the way down. Cross a CPU architecture boundary and near-ties can round differently. That is the whole story: one adjacent-rank flip, and the subset aggregate moved from 100.0% to 99.4%.

If you compare memory systems for a living, the field guide we wrote alongside this matters more than our numbers. Check what the figure is:

recall_any@K means at least one gold session lands in the top K. recall_all@K means all of them. Vendors mostly publish the first family. Our own run scores 98.2% on the first and 88.0% on the second, from the same data. Ten honest points apart.
The cutoff matters. Recall@15 is easier than Recall@5.
Sample size matters. 100 questions is a smoke test, 500 is a validation run.
Version and date matter. Benchmarks are snapshots of moving software.
The harness being public matters most. A table is marketing. A harness you can clone is a benchmark.

The full post has the comparison chart with per-metric disclosures, the full field table (Mnemosyne, MemPalace, agentmemory, Supermemory, each with its metric labeled), and what we deliberately do not claim:

Read the full breakdown: https://blog.codecora.dev/we-published-our-benchmark-then-we-tried-to-break-it/

Don't trust our benchmark. Run your own. The harness and reproduction results are in the repo.

Cross-posted from the CodeCora blog, where we write about local-first AI infrastructure.

An 87GB model ran on a CPU at 8 tok/s. Quantization did that.

Anaz S. Aji — Tue, 01 Sep 2026 06:44:17 +0000

Someone on r/LocalLLaMA benchmarked Qwen3.8-Flash-Next in llama.cpp from CPU-only all the way to 96GB of VRAM. The model file is 87.2 GiB. On a plain CPU it decodes at 8.34 tok/s. With the full 96GB, 109 tok/s.

The number that looks like a party trick is the CPU one. It is a compression story. The weights are 4-bit. At 16-bit the file would be around 4x bigger, north of 300GB, and no consumer setup loads that, GPU or not.

Two more findings from the benchmark worth stealing:

At 245K context, the 96GB advantage over 24GB shrinks from 2.80x to 1.45x. Long context flattens everything.
Dropping the model's 27.2 GiB embedding table onto the GPU cut decode from 108.5 to 1.95 tok/s. RAM placement was 55.6x faster. Where a tensor lives matters as much as how much VRAM you have.

Why this matters here: embeddings hit the same wall at smaller scale. Every f32 embedding at 768 dims costs 3,072 bytes. A 100k-vector agent memory is about 300MB. Fine on a server, real money on a phone.

We built vecq for that. Training-free vector quantization in Rust (Apache-2.0, crate: vecq-core). No calibration data, deterministic results across platforms. On real EmbeddingGemma vectors: 4.78x smaller at default width (642 bytes/vector) with recall@10 of 0.979, and the index builds 14x faster than HNSW.

Honest trade: search is brute force, about 14x slower than HNSW at default width. Made for on-device indexes in the thousands of vectors, not million-vector servers. The file format is versioned and readers accept older versions, so indexes keep loading across releases.

Full breakdown with the benchmark tables is on the blog:

👉 https://blog.codecora.dev/quantization-cpu-to-96gb-and-embeddings/

Repo: https://github.com/codecoradev/vecq
Crate: https://crates.io/crates/vecq-core