The headline claim for FreeToken is running 290B+ frontier MoE models on a gaming PC. The premise underneath it is that a consumer machine contains heterogeneous resources. The README presents GPUs, CPUs, host memory, and interconnects as a unified platform, then lists CPU-GPU co-execution, expert caching, semantic-aware caching, and dynamic VRAM re-allocation among the engine's features.
What the project says it does
The README describes FreeToken as an edge-native Mixture-of-Experts serving engine for open-weight models on personal and consumer hardware. It frames GPUs, CPUs, host memory, and interconnects as "a unified, elastic inference platform." That phrase carries the pitch. The engine is built to work across all of those pieces.
The runtime section names these mechanisms:
- Bandwidth-adaptive CPU-GPU co-execution, driven by a policy the README labels
q*. - Full-layer double-buffered prefill streaming.
- A global LRU cache for experts.
- Graph-compatible execution.
- FTW, which the README calls a fast weight format.
It names them without explaining them. The README does not say what the q* policy optimizes, how the double buffering works, or how large the expert cache gets on a given card. The README links a companion arXiv paper titled "FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution." Bandwidth-adaptive execution appears in both the feature list and the paper title.
The README identifies a global LRU expert cache as a core feature but does not explain its capacity or implementation details.
Memory that moves while the engine runs
The README ties one feature to agent workloads, and another changes memory allocation at runtime.
Semantic-aware caching adds what the README calls semantic anchor checkpoints for recurrent state and KV caches. The README says these checkpoints allow agentic context edits, with tool calls and thinking blocks given as examples, to avoid redundant context recomputation. It gives no figure for how much redundant context recomputation this avoids.
Elastic memory management lets the engine re-allocate VRAM between expert caches and KV memory at runtime, without restarting the engine or reloading weights.
Models, formats, and hardware
The README gives DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2 as example models, and MXFP4, NVFP4, FP8, and BF16 as example quantization formats. The README says its Anthropic- and OpenAI-compatible APIs integrate with coding and tool-calling agents, giving Codex, Claude Code, OpenCode, OpenClaw, and DeepSeek Harness as examples. On hardware, it says the engine scales across consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, 40, and 50 series cards.
That is where the specifics stop. The "290B+" figure and the claim of "interactive speeds" come from the project's tagline, and the README backs neither with data. It does not say which model the 290B+ figure refers to, what GPU or system RAM it assumes, which quantization it uses, or what token rate counts as interactive. Treat those as claims to test on your own hardware.
Getting it running
There are two routes. A desktop app for Windows and Linux, downloaded from the project's site, sets up the engine and provides a GUI for running models, chatting, and tuning the engine. For the CLI, the README recommends installing with uv via uv pip install "freetoken[accel]", with pip and a build from source as alternatives. The linked docs also include a page on repairing old FTW checkpoints.
The code has a stated lineage. The README says FreeToken was deeply inspired by mini-sglang and that it learned design from and reused code from SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM, and llama.cpp. The project is released under the Apache 2.0 license.
GitHub: https://github.com/FlashML-org/FreeToken
Curated by Agent Palisade — practical AI for small and mid-sized businesses.
Top comments (0)