The most interesting FreeToken benchmark is easy to summarize:
Qwen3.6-35B-A3B
RTX 4060 Laptop
8 GB VRAM
39.3 tok/s
Then it gets more unusual:
DeepSeek-V4-Flash
284B parameters
single RTX 5090
22-25 tok/s
FreeToken is an open-source serving engine designed specifically for Mixture-of-Experts models on personal hardware.
The important idea is not "VRAM no longer matters."
It is that VRAM is only one resource in the machine.
FreeToken treats the GPU, CPU, host RAM, and interconnect as one heterogeneous inference system. ⚡
Why MoE changes the problem
A dense model effectively needs all relevant weights for every token.
A Mixture-of-Experts model has many expert blocks but activates only a subset for each token.
FreeToken's paper gives DeepSeek-V4-Flash as an example.
The model contains 284B total parameters.
Only 13B participate in a single token.
That makes the computation much more manageable on a consumer GPU.
But the rest of the model does not disappear.
The full expert pool is still huge.
The hard problem becomes:
where do experts live?
which experts stay in VRAM?
what happens when one is missing?
how do we hide transfer latency?
FreeToken is built around those questions.
The expert pool lives in host memory
At a high level, FreeToken uses a two-level expert-memory hierarchy.
The full routed-expert pool stays in CPU-accessible host RAM.
Non-expert weights remain on the GPU.
Remaining VRAM becomes a shared expert cache across MoE layers.
When an expert is already cached, the GPU can execute it immediately.
When it misses, the engine has two possible routes.
route A:
RAM -> PCIe -> VRAM -> GPU execute
route B:
RAM -> CPU execute directly
Neither route is universally better.
Bandwidth-adaptive CPU/GPU co-execution
This is one of the most interesting parts of the design.
PCIe transfer consumes link bandwidth.
Direct CPU expert execution consumes host-memory bandwidth.
A laptop and desktop can have completely different ratios between those resources.
So FreeToken measures the machine instead of relying on a fixed expert-placement policy.
Its q* decode policy divides cache misses between GPU cache fills and direct CPU execution based on the actual host-memory and PCIe bandwidth.
The goal is to keep both resources productive.
That is a more sophisticated strategy than static CPU offloading.
Prefill requires a different approach
MoE sparsity helps decode because each token touches a small subset of experts.
A long prompt is different.
Thousands of prompt tokens can collectively route through almost every expert.
That makes the effective working set much denser.
FreeToken handles this with full-layer double buffering.
While the GPU computes layer l, the experts for layer l+1 stream over PCIe.
GPU: compute layer N
PCIe: load layer N+1
The two buffers then swap.
This hides transfer behind useful computation.
Semantic-aware caching for agents
The project is explicitly designed around agent workloads too.
Agent contexts are not simple append-only conversations.
Tool calls modify context.
Thinking blocks may be removed.
Prompts get edited.
Traditional recurrent-state or KV reuse can become invalid.
FreeToken creates semantic anchor checkpoints around boundaries such as tool calls and thinking segments.
When the context changes, it can resume from a surviving anchor and re-prefill only the suffix instead of starting over.
That is a very practical optimization for long-running coding agents.
Elastic VRAM management
Personal hardware is not a datacenter.
The GPU shares VRAM with:
- the desktop
- browsers
- creative apps
- games
- other local workloads
At the same time, an agent's KV cache grows as context grows.
FreeToken can resize and rebuild the GPU expert cache under a new memory budget without restarting the engine or reloading the host-resident expert pool.
The memory split is allowed to move with the workload.
Published performance
The paper reports:
- Qwen3.6-35B-A3B at 39.3 tok/s on an 8 GB RTX 4060 Laptop
- Qwen3.6-35B-A3B at 77-83 tok/s on RTX 5090
- DeepSeek-V4-Flash 284B at 22-25 tok/s on RTX 5090
- GLM-5.2 753B served on one RTX PRO 6000 workstation GPU
Across five consumer systems, the paper reports a 1.3-2.1x decode-throughput improvement over the tested edge-serving baselines.
Those numbers are project/paper benchmarks, so I would treat them as published benchmark results rather than a promise for every machine.
RAM is still part of the requirement
A giant model still has to live somewhere.
The project's FAQ explains that MoE expert weights live in host RAM.
For example, Qwen3.6-35B-A3B in BF16 needs roughly 70 GB of free system RAM for experts.
Quantized checkpoints reduce that requirement substantially.
So the correct interpretation of the 8 GB RTX 4060 result is:
8 GB GPU is enough for the GPU side of the serving strategy
not:
the entire 35B model fits in 8 GB total memory
That distinction matters.
Current hardware support
The current install docs list:
- Linux x86_64 for the Python install
- NVIDIA GPU
- Ampere / RTX 30-series or newer
- driver r580+
- CUDA 13
- Python 3.10+
FreeToken also ships a desktop app for Windows and Linux.
The maintainers list macOS, AMD, and aarch64/DGX Spark on the roadmap.
Quick start
uv pip install "freetoken[accel]"
Then:
ft serve --model ~/models/Qwen3.6-35B-A3B
The server exposes OpenAI-compatible APIs:
/v1/chat/completions
/v1/responses
/v1/models
and Anthropic-compatible endpoints:
/v1/messages
Local coding agents
FreeToken also has a useful launcher:
ft launch claude
ft launch codex
ft launch dsh
ft launch hermes
ft launch openclaw
ft launch opencode
It configures the agent to use the local server.
That makes FreeToken more than a local-chat runtime.
It can serve as the local inference backend for coding and tool-calling agents.
My takeaway
For dense models, local inference is often summarized as:
model size <= VRAM
MoE gives us a different equation:
small active path
+ large host-resident expert pool
+ smart caching
+ CPU/GPU co-execution
+ enough bandwidth
FreeToken is an attempt to optimize that entire equation.
And the published results suggest that the practical ceiling for local open-weight models can be much higher than VRAM alone would imply.
Top comments (1)
The RAM side of this is the part I wish got more attention. The "8 GB RTX 4060" headline is doing a lot of work: a BF16 Qwen3.6-35B-A3B needs roughly 70 GB of host RAM for the expert pool, so the real system requirement is closer to "a workstation-class memory footprint" than "any gaming laptop." Quantized checkpoints change that math, but it feels like the bottleneck people will actually hit first is host-memory bandwidth, not PCIe.
The bandwidth-adaptive q* split is the most interesting piece to me. Most offloading setups I've poked at are static — you decide at load time what lives on the GPU and eat every miss. Measuring the actual PCIe/host-memory ratio per machine and splitting cache misses between cache fill and direct CPU execution is the kind of design that only pays off once you deploy across heterogeneous boxes, which is exactly where an engine like this lives.
The semantic anchor checkpoints for agent workloads are what I'd want to benchmark in anger, though. Agent contexts get edited constantly — tool outputs truncated, thinking blocks dropped, retries that rewind the transcript — and every invalidation currently costs a full re-prefill on long runs. Resuming from a surviving anchor and re-prefilling only the suffix is a real win there. Did the paper say anything about what happens when the boundary itself gets edited away, i.e. when a tool call the anchor was anchored to gets trimmed?