LFM3’s small backbone still has a growing KV cache

At LFM3’s maximum configured context length, the keys and values in its full-attention layers alone work out to 3 GiB for one sequence with a two-byte cache dtype. That is a tensor-payload calculation, before the weights and the rest of the runtime enter the picture.

Small model. Separate memory bill. 🧠

Liquid AI’s LFM3 release puts several modalities on a shared small backbone. The part I wanted to unpack was its long context: what exactly keeps growing when a hybrid model mixes convolutions, local attention and global attention? [1]

Computed from the pinned text-model configuration. This is a payload estimate, not measured device memory.

Follow the layers, rather than the parameter label

The pinned LiquidAI/LFM3-1.7B configuration contains 24 layers: 13 convolution layers, five sliding-attention layers and six full-attention layers. It sets max_position_embeddings to 262144. Its attention has 16 query heads but only four key/value heads, each with dimension 128. [2]

That last distinction matters. The stored keys and values use the KV-head count. Multiplying by the query-head count would overcount this component by a factor of four. ⚠️

I traced the attention implementation too: k_proj and v_proj produce num_key_value_heads × head_dim, and those tensors enter the cache before attention consumes them. The model creates a DynamicCache from its configuration when caching is enabled. [3]

For full-attention layers, the simple payload equation is:

bytes = batch × tokens × full-attention layers × KV heads × head dimension × 2 × bytes per element

The first 2 counts keys and values. The final factor is the cache dtype assumption. A two-byte cache can represent BF16 or FP16; it does not mean the weights have the same precision.

With batch one, that gives 12,288 bytes per cached token across these six layers. I checked the arithmetic independently by serializing a small array of 16-bit elements and scaling its byte count. ✅

A calculator that reads the actual configuration

This example needs Python’s standard library and an internet connection. It fetches one pinned configuration file. It does not download weights or execute remote model code.

import json
import urllib.request
from collections import Counter
revision = "40ebe0aa6662c9d40389c2f5d565b6a2c6aebacd"
url = f"https://huggingface.co/LiquidAI/LFM3-1.7B/resolve/{revision}/config.json"
with urllib.request.urlopen(url, timeout=30) as response:
    config = json.load(response)
layers = Counter(config["layer_types"])
assert layers == {"conv": 13, "full_attention": 6, "sliding_attention": 5}
bytes_per_token = (
    2 * layers["full_attention"] * config["num_key_value_heads"]
    * config["head_dim"] * 2
)
for tokens in (8192, 65536, config["max_position_embeddings"]):
    print(f"{tokens} tokens: {tokens * bytes_per_token / 2**20:.0f} MiB")
print("Scope: full-attention K/V tensors only; batch=1, 2 bytes/element")

Output:

8192 tokens: 96 MiB
65536 tokens: 768 MiB
262144 tokens: 3072 MiB
Scope: full-attention K/V tensors only; batch=1, 2 bytes/element

🧮 Here, MiB means bytes divided by 2**20. The final row is exactly 3 GiB, using 2**30 bytes per GiB. The script pins the repository with a public commit identifier so its inputs stay reproducible.

What this changes about deployment

A hybrid architecture does not make every layer retain the whole history. The convolution and sliding-attention parts have different state behavior; I have deliberately left their memory out of this calculation. The six global layers are enough to show why weight size cannot answer the context-budget question.

If a serving setup keeps two equally long sequences resident, this full-attention payload doubles. If it uses a four-byte cache dtype, this component doubles again. Quantized caches, offloading and other cache implementations change the assumptions; check the engine actually being deployed.

💡 Keep weight precision, cache precision, retained sequence length and active batch size as separate settings. A compact weight file is useful information. It is not a total-memory measurement.

This calculation also says nothing about prompt-processing speed or answer quality at long context. Activations, temporary attention buffers, convolution state, sliding-layer state and allocator overhead still need their own accounting. I did not load LFM3 weights or measure its inference here.

References and visual credit

[1] Liquid AI: Build specialized multimodal models with LFM3.

[2] Pinned text-model configuration.

[3] Pinned attention and model implementation. The checkpoint uses the LFM Open License; this article redistributes no weights.

Original chart; values calculated from the cited configuration under the stated assumptions.

AI disclosure

AI was used to draft the article.