MemT-1.0: How Baking Memory Into the Model Itself Cuts KV Cache Storage by 66x

Every time a large language model processes a long conversation, it pays a tax. The key-value (KV) cache — the data structure that stores intermediate attention states for every past token — grows linearly with context length. At 128,000 tokens, a single request from a model like Llama 3 70B can consume roughly 40 GB of GPU memory just for the cache, competing directly with model weights for scarce high-bandwidth memory. Scale that to a million tokens and the numbers become impractical for most deployments.

The standard responses to this problem — quantizing the cache, compressing it with grouped-query attention, or offloading it to CPU — are engineering patches on a structural issue. A new paper from October 2026, MemT-1.0: Towards Memory-Native Evolution, proposes a different answer: make memory an intrinsic part of the model's computation rather than an external appendage.

The Three-Way Memory Problem

Foundation models currently handle memory in one of three ways, each with a distinct cost profile.

Parametric memory bakes knowledge into model weights during training. It is fast at inference time but expensive to update — you cannot cheaply teach a model what happened last Tuesday without fine-tuning.

Retrieval-augmented generation (RAG) keeps knowledge outside the model in a vector store and fetches relevant chunks at query time. It is flexible and updatable, but it adds retrieval latency, requires a separate indexing pipeline, and the model must still fit the retrieved context into its attention window.

KV cache / long context simply extends the attention window and stores every past token's key-value pair. It is the most faithful representation of history but the most expensive: storage scales linearly, and re-transmitting the full context at every decoding step creates a bandwidth bottleneck that grows with sequence length.

MemT-1.0 targets a fourth option: a continuously updated internal state that accumulates information across interactions without growing unboundedly.

The Architecture: Gated DeltaNet-2 and Mixture-of-Storage

MemT-1.0 builds on Metis, a prior memory-native backbone, and introduces two core mechanisms.

Gated DeltaNet-2 performs stable token-level memory updates within each inference turn. Rather than appending new tokens to an ever-growing KV cache, the layer updates a fixed-size recurrent state using a gated delta rule. The key property is stability: the update mechanism is designed to avoid the state explosion or forgetting problems that plagued earlier linear-attention architectures. Because state updates live inside the layer mathematics, the model stops paying the bandwidth penalty of re-transmitting the full past at every decoding step.

Mixture-of-Storage scales native memory capacity beyond what a single recurrent state can hold. The mechanism allocates different types of storage — think of it as a learned routing between fast, compact states and slower, higher-capacity memory slots — allowing the model to retain information over longer interaction histories without reverting to full attention KV entries.

The training pipeline runs in two stages: continued pre-training followed by post-training, using approximately 34 billion tokens of training data. The team released two model variants:

• MemT-1.0-Nano: 36B total parameters, 3B active, 10M dynamic memory. Backbone: Qwen3.6-35B-A3B.
• MemT-1.0-Flash: 288B total parameters, 17B active, 336M dynamic memory. Backbone: DeepSeek-V4-Flash.

What the Numbers Show

The headline result is the storage comparison at 128K tokens of history. MemT-1.0-Nano achieves approximately 1.86x end-to-end inference speedup and a 66x reduction in persistent storage relative to a full-context KV cache. Time-to-first-token speedups reach 2.44x at 128K tokens and 4.19x at 256K tokens — gains that compound as context length grows.

On eleven memory benchmarks spanning MemBench, PersonaMem, LoCoMo, LongMemEval-S, MemoryAgentBench, and BEAM-1M, MemT-1.0-Flash averages 50.6 and Nano averages 47.0 — reported as 12.5 and 8.9 points above the strongest parametric baselines respectively. Both models remain competitive with retrieval-augmented methods while largely preserving general capabilities on MMLU-Pro, GPQA, IFEval, and LongBench (within one to two points of their teacher backbones).

The agentic results are notable: by internalizing retrieved procedural experience, MemT-1.0 gains 7.6 points on SWE-bench Verified and 8.3 points on SWE-bench Multilingual over the respective backbones without experience. For agents that need to accumulate task-specific knowledge across multi-step workflows, this is a meaningful signal.

Why This Matters for Practitioners

The practical implication is straightforward: if you are serving long-context workloads at scale, the KV cache is a cost center. Techniques like prefix caching and MLA compression have already moved the needle, but they are optimizations on top of the same structural design. MemT-1.0 changes the design itself.

For persistent assistants and personalized agents — systems that need to remember user preferences, past interactions, and accumulated procedural knowledge — native memory offers something RAG cannot easily provide: the memory is part of the model's computation, not a lookup that must be explicitly triggered and correctly retrieved. The model does not need to decide whether to consult an external store; the relevant history is already encoded in its internal state.

There are open questions worth tracking. The paper does not fully specify how the model handles memory isolation across users or tasks — a critical concern for multi-tenant deployments. The recurrent state can accumulate errors over long interaction chains, and the mechanisms for safe forgetting or state correction are not detailed in the abstract. The code is released under a PolyForm noncommercial license (the paper under CC-BY-NC-SA), so commercial deployment requires a separate license inquiry with the MemTensor team.

The Broader Trend

MemT-1.0 arrives alongside a cluster of related work: KronLSTM from JKU Linz, RPMem for cross-session parametric memory, and ARM (Attention with Routed-Memory). The pattern is consistent — the field is moving toward architectures where memory is a first-class citizen of the model rather than an afterthought managed by the serving infrastructure.

The 66x storage reduction is a structural cost advantage, not a tuning tweak. With public weights available on Hugging Face and SGLang session persistence support, every team running long-context inference now has a concrete benchmark target to evaluate against their current stack.

Whether MemT-1.0 becomes a production standard or a research stepping stone, it makes the case clearly: the KV cache is not the only way to give a model a past.

---

Primary source: MemT-1.0: Towards Memory-Native Evolution (arXiv:2610.26825, October 2026). Supporting coverage: AI Weekly, Hugging Face Papers, cctest.ai.