I serve Qwen3.8-27B in production on RTX Blackwell workstation cards, on an inference engine I wrote from scratch in Rust and CUDA (memra, MIT). This post is the write up of the serving artifact that makes it fast: a 4-bit NVFP4 trunk plus a masked multi-token-prediction draft head. All the files are public on Hugging Face.
What the artifact is
The trunk is an NVFP4 GGUF of Qwen/Qwen3.8-27B. NVFP4 is 4-bit e2m1 with per-16 FP8 scales. Token embeddings and the output head stay at Q5_K, norms at F32. The interesting part: the MTP (multi-token-prediction) head ships inside the file, so speculative decoding works out of the box on engines that read it. Native context is 262,144 tokens.
The masked head trick
Speculative decoding drafts a few tokens ahead and lets the target model verify them. The draft head's biggest cost is its LM head read: 248,320 vocab rows every draft step.
So I mask it. The draft proposes tokens only from the top 32,768 ids ranked by how often the model itself emits them. The ranking corpus is 100% model-generated (163k tokens over real agentic session prompts). The verification still runs on the full vocabulary, which means the mask can never change output. It only moves acceptance rate, and it shrinks the head read 7.6x.
Measured, K=3 on held-out prompts, medians of 5 interleaved runs: the trimmed head accepts slightly less than the full head (0.74 vs 0.76) and still wins end to end, +5.1% on RTX PRO 6000 and +6.4% on a 5090 laptop. Acceptance explains a result, end-to-end tok/s decides it.
Rules I learned the expensive way
Ranks are per model, per quant. Foreign ranks from an identical tokenizer measured 12 acceptance points worse. A finetune moved the distribution, so its ranks must move too.
Derive ranks with the chat template ON if you serve chat. A raw-text rank set once left a chat deployment with 10.9% of tokens structurally unproposable, which cost 15 acceptance points.
Corpus floor is 4x your topN in own-generated tokens. Ranks past the head of the distribution are noise.
The attach is a log line, not the absence of an error. Wrong path or wrong flag does not fail, the engine silently drafts from the full embedded head and everything still works, just slower than you think. Check the boot log for the trim line.
Run it yourself
MEMRA_MODELS="q38=hf:Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF:Q5K-mtp+hf:Avifenesh/Qwen3.8-27B-NVFP4-MTP-GGUF:frspec-sxc32768" \
memra-server
The engine gates every fast path byte-identical to plain decode before it ships. Speculative, graphed, batched, all of them. Speed never gets to change what the model says.
The tools to build your own ranks and trimmed head for any model ship in the memra repo, and there are copy-paste serving configs per GPU in docs/COOKBOOK.md.
Or just use the hosted endpoint
These exact files serve qwen/qwen3.8-27b behind api.tiyuvta.ai. Measured on the live endpoint this week: 260 tokens/sec on a typical 512-token completion, 0.166s cold time to first token. $0.40 per million input tokens, $0.10 cached, $2.03 output. OpenAI Chat Completions, OpenAI Responses and Anthropic Messages on one endpoint with tool calling on each.
New accounts get $5 of free credit, no card needed. Docs are at inference.tiyuvta.ai/docs. If you try it and something is off, tell me, the measurements above are dated and I want to know when reality disagrees.
Top comments (0)