If you are picking a model to deploy, architecture shows up as memory, latency, and context length. A neural network takes numbers in, passes them through layers of units that combine inputs with learned weights, and produces numbers out. Those weights are the parameters. From there, the deploy question is how cost grows with sequence length, how many parameters you pay for on each token, and whether the model is general or built for one job.
What does a transformer cost as context grows?
The transformer was introduced in 2017, when Ashish Vaswani and colleagues described a model built entirely on attention, with no recurrence or convolutions. It trained faster and translated better than the recurrent models of the time. Attention compares every token with every other token, so compute and memory climb quickly with context length. FlashAttention makes exact attention cheaper on GPUs. I would still measure the long-context case, because that is where a pure attention stack gets heavy.
State-space models keep a fixed-size running state as they sweep the sequence, so cost grows linearly with length. Mamba, from Albert Gu and Tri Dao, is a selective state-space model. The authors report five times the inference throughput of transformers, and a 3 billion parameter model that matches transformers twice its size on language modeling. Quality across tasks is still an open question, because attention's global view is useful. That glossary entry calls today's frontier largely hybrid, mixing attention with state-space-style layers, which is the same direction Liquid AI is pushing: keep the strengths, cut the scaling cost.
What changes in Liquid's hybrid?
Maxime Labonne, Head of Post-Training at Liquid AI, said they wanted an on-device LLM that is fast and accurate, for hardware such as a phone or a wearable. He described current LLMs as accurate, but not very fast. LFM one was released in 2024. LFM two, released in 2025, is open source. He said they had open sourced 17 models since July, in about three months.
The 350 million, 700 million, and 1.2 billion parameter LFM two models each use six attention layers and ten short convolution layers, so convolution outnumbers attention. He says inference is much faster, and memory stays much lower at long context, because the convolution layers keep the KV cache from exploding, while quality stays at the level of a pure transformer.
He frames it as three metrics that usually trade off against each other: memory use, inference speed, and quality. I would score any candidate on those three numbers: memory, inference speed, and quality. A win on speed that misses your quality bar is still a failed deploy.
theoretical gains might not be realized in practice. And it's not because you have an architecture that is supposed to run very fast because the algorithm, the math says so, that it's going to fully realize once you lower the architecture on the target hardware. That was a lesson that we learned with the first generation of LFM models.
Maxime Labonne, Head of Post-Training at Liquid AI, on Chain of Thought ep 43
What they did next was optimize on target hardware from the start, with a Samsung phone as the example, and use more than 100 pretraining evaluations while choosing the architecture. I would borrow the measurement habit. Time the model on the hardware you will ship, and keep a quality bar beside the speed number.
Where do parameters and a foundation model fit?
Model parameters are a rough guide to capability and cost. More of them can store more knowledge and patterns. Among models of similar design, they generally take more memory and computing power, which raises the cost and latency of each answer. A mixture-of-experts model uses only some of its parameters for each token. Quantization shrinks the memory each parameter takes. On the prompts OpenAI's InstructGPT research tested, people preferred a 1.3 billion parameter model trained to follow instructions over the 175 billion parameter GPT-3.
Labonne argues that knowledge depends on the number of parameters. In his account, distilling knowledge into a 1 billion parameter model will never make it as smart as a 3 billion parameter model. My reading is that if the task needs that stored knowledge, you work around the misses or move up in size.
A foundation model is the general side of the choice: one large model trained on broad data, then adapted to many tasks through prompting, retrieval, or fine-tuning. A specialized model is aimed at one job. Liquid's Nanos are small fine-tuned versions of their models, including a data extraction model that takes text and returns a JSON object you specify. Labonne says function calling on even the 350 million parameter model has gotten good feedback, that results depend on the task, and that this will not replace cloud model quality for complex workflows. He treats the small model as one task a multi-agent system can call.
I would use a hybrid or a state-space design when context length makes attention's memory and latency unacceptable and quality still clears your bar. I would use a larger parameter count, often a foundation model, when the job needs broad knowledge or a complex workflow. I would use a specialized small model when the job is one structured task and you have checked the misses.
What should you benchmark before you commit?
Run the candidates on your own hardware. Same prompts, same context lengths. Log peak memory, answer latency, and a task score you defined. Include a long context, because that is where a transformer's cache grows and where Labonne says convolution layers hold memory down. He said a 1 billion parameter model would get 1% at best on some frontier evaluations, which is why his team writes narrower tests for the skills they care about. I would do the same: score the task you will ship, not a leaderboard built for a much larger model.
# Sketch only. Compare measurements you already collected.
# Not a Liquid AI tool, and it calls no vendor SDK.
def fits(run, ram_gb, latency_ms, min_quality):
return (
run["peak_ram_gb"] <= ram_gb
and run["latency_ms"] <= latency_ms
and run["task_score"] >= min_quality
)
def prefer(transformer, hybrid, limits, long_context):
hybrid_ok = fits(hybrid, **limits)
transformer_ok = fits(transformer, **limits)
if long_context and hybrid_ok:
return "hybrid"
if transformer_ok:
return "transformer"
if hybrid_ok:
return "hybrid"
return "change_budget_or_context"
Checklist
- Write down context length, RAM, and latency before you compare models.
- Benchmark a transformer against a hybrid or state-space model on the same device, with a long prompt included.
- Record peak memory, answer latency, and a task score you chose.
- Treat parameter count as a knowledge ceiling. Distillation does not turn a 1 billion parameter model into a 3 billion parameter model.
- Use a specialized small model for one structured task, and a foundation model when the job is broad or the workflow is complex.
More on this topic, with the related episodes, is on Chain of Thought. It draws on this episode.
Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.
Drafted with AI assistance from the episode transcripts.
Top comments (0)