A sparse addressing layer can expand the effective hidden state of recurrent models by orders of magnitude while keeping FLOPs constant, overturning the long‑standing trade‑off between capacity and compute. Sparse Delta Memory (SDM) demonstrates that this leap is achievable with a lightweight modification to gated linear RNNs.
Linear RNNs such as Gated DeltaNet rely on dense key‑value outer products, so their state size grows only linearly with the amount of computation per token. Prior work has shown that boosting the hidden dimension improves long‑context recall, but each extra unit incurs additional FLOPs, making scaling prohibitively expensive for many deployment scenarios.
SDM achieves a three‑order‑of‑magnitude increase in memory state size while maintaining the same compute budget. “To address the limitations of dense state updates, we introduce Sparse Delta Memory (SDM), a novel architecture based on the observation that the GDN update rule can be sparsified. This enables a three-order-of-magnitude increase in memory state size while maintaining the same compute budget, as shown in Fig. 1” [1]. By keeping parameters and FLOPs identical to the dense GDN baseline, the larger sparse memory directly translates into higher performance on downstream tasks.
When the enlarged sparse memory is held to an iso‑FLOP constraint, models exhibit markedly better in‑context learning and long‑context retrieval. “We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks” [1]. The gains appear without any architectural overhaul beyond the sparse read/write layer, suggesting that capacity can be multiplied rather than simply padded.
The result implies that developers can replace standard gated linear RNNs with SDM to obtain a virtual memory boost of up to 1,000× at no extra computational cost, redefining how we think about efficient scaling for long‑context applications. Re‑evaluating existing in‑context learning and retrieval benchmarks under this iso‑FLOP regime should now become the default sanity check for any new recurrent architecture.
Top comments (0)