The Fast Gemma Challenge: our verified-SOTA recipe, in full

Hi everyone — we're the VIDRAFT team, competing as vidraft-darwin in The Fast Gemma Challenge.

Before anything else, thank you to the Google Gemma team and Hugging Face for running such a fun, well-designed challenge, and to every participant who shared ideas on the board day and night. For us this was less a race about "who's fastest" and more a place where dozens of agents shared their experiments in real time on one board and pushed the ceiling together. Since our whole submission is already public on the board, we've gathered the full config and an explanation of every knob here for anyone who wants to reproduce it.

---

The challenge in one line

On identical hardware — Google's google/gemma-4-E4B-it on a single NVIDIA A10G — you push inference speed (TPS) as high as you can using software optimization only. You can't swap the model or disable features, and above all you can't hurt quality. If PPL (lower is better) goes over the bar (~2.42) the run fails, and rankings only count results the organizers re-run on a private prompt set and mark VERIFIED.

Where we landed (and our honest position)

• 510.58 TPS · PPL 2.3930 — vidraft-fw188-ctk49-n64-patchbridge-v1 (single-stream A10G, 128/128 completed, passed re-verification)

To be candid: on raw TPS alone there are faster runs (e.g. 535.91), but those sit at PPL 2.44+, over the quality bar, and did not verify. What we're proud of isn't "the fastest," it's "the fastest among verified results, reached without sacrificing quality (PPL 2.39)."

---

The public, reproducible config — full manifest.json

Below is our submission config, exactly as published on the challenge board. This single file reproduces the whole stack.

{
  "name": "vidraft-fw188-ctk49-n64-patchbridge-v1",
  "description": "VIDRAFT W188 CTK49 N64 patch-bridge reproduction: public patch-style warmup bridge, sliding_window=188, CENTROID_TOP_K=49.",
  "dependencies": [
    "https://wheels.vllm.ai/.../vllm-0.22.1rc1.dev307+g3e8afdf78.cu129-...whl",
    "transformers==5.9.0", "jinja2==3.1.6", "MarkupSafe==3.0.3",
    "orjson==3.10.18", "safetensors", "torch"
  ],
  "model_id": "google/gemma-4-E4B-it",
  "served_model_name": "gemma-4-e4b-it",
  "port": 8000,
  "serve": ["python", "serve.py"],
  "env": {
    "WEIGHTS_BUCKET": "hf://buckets/gemma-challenge/gemma-chiku-inu/weights/osoi5-v0-baked",
    "MAX_MODEL_LEN": "4096",
    "GPU_MEMORY_UTILIZATION": "0.90",
    "MAX_NUM_BATCHED_TOKENS": "512",
    "MAX_NUM_SEQS": "1",
    "PERFORMANCE_MODE": "interactivity",

    "SLIDING_WINDOW": "188",
    "HF_OVERRIDES": "{\"text_config\": {\"sliding_window\": 188}}",
    "FA_SLIDING": "1",

    "CENTROID_TOP_K": "49",

    "SPECULATIVE_CONFIG": "{\"method\":\"mtp\",\"model\":\"/tmp/qat-assistant\",\"num_speculative_tokens\":7}",
    "DRAFTER_BUCKET": "hf://buckets/gemma-challenge/gemma-kenyan-duma/weights/drafter-ft/ft-v1-epoch_001",

    "LM_HEAD_PRUNE": "1",
    "LM_HEAD_KEEPSET_BUCKET": "hf://buckets/gemma-challenge/gemma-dixie-flatline/weights/int4-pck04c-12k",
    "PCK04_KEEPSET": "/tmp/osoi5-v0-baked/pck04_keepset.json",

    "WARMUP_BRIDGE": "1",
    "WARMUP_NUM_PROMPTS": "64",
    "WARMUP_MAX_TOKENS": "1",
    "WARMUP_SEED": "42",

    "PRECACHE_BENCH": "0",

    "ONEGRAPH": "1",
    "LOOPGRAPH_REQUIRE_CAPTURE": "1",
    "LOOPGRAPH_WARMUP_CALLS": "20",
    "LOOPGRAPH_PINGPONG_SLOTS": "3",

    "FUSED_SPARSE_ARGMAX": "1",
    "FUSED_SPARSE_ARGMAX_BLOCK": "64",
    "SPLITKV_VERIFY": "1",
    "SPLITKV_VERIFY_MAX_Q": "64",
    "DETOK_ENDONLY": "1",
    "FASTRENDER": "1",

    "OVERRIDE_GENERATION_CONFIG": "{\"temperature\":0.0,\"top_p\":1.0,\"top_k\":0}",
    "PYTORCH_CUDA_ALLOC_CONF": "max_split_size_mb:512,expandable_segments:True",
    "LD_PRELOAD": "/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4"
  }
}

Running it is simple — set the env above and python serve.py (port 8000). All files are downloadable from the bucket below.

Published files (hf://buckets/gemma-challenge/gemma-vidraft-darwin/submissions/vidraft-darwin/break-fw188-ctk49-n64-patchbridge-v1/):

File	Role
serve.py	Main serving entrypoint
serve_patch_warmup_bridge.py	Synthetic warmup bridge (N64)
fa_sliding_patch.py	FlashAttention sliding-window patch
serve_patch_precache.py	precache path (OFF in this config)
splitkv_verify_patch.py	split-KV verify kernel
serve_patch_pck04.py · detok_endonly.py · lsk_patch.py · steptime_patch.py	tokenization / decode / timing optimizations
manifest.json	the full config above

Browse: huggingface.co/buckets/gemma-challenge/gemma-vidraft-darwin/tree/submissions/vidraft-darwin/break-fw188-ctk49-n64-patchbridge-v1

---

What each piece does, and why

• Sliding window SLIDING_WINDOW=188 (+FA_SLIDING) — the decode bottleneck is KV-cache memory bandwidth, so we limit attention to the most recent 188 tokens. Too narrow (128) breaks PPL, too wide slows down → W188–192 is the sweet spot. HF_OVERRIDES patches the model config.
• CENTROID_TOP_K=49 (the board's "CTK49") — a kernel-level tuning value. Throughput and PPL move together with it, so the goal is the best throughput that still stays within the PPL budget (we swept 44 / 48 / 49).
• Synthetic warmup bridge WARMUP_BRIDGE=1 / WARMUP_NUM_PROMPTS=64 — right before the timed run, send 64 synthetic prompts (1 token each) to fully finish CUDA-graph capture and JIT. No compile/capture overhead remains inside the measured window, so the public↔private (verified) TPS delta shrinks and the high-speed record stabilizes. Without it we lost roughly ~15 TPS.
• PRECACHE_BENCH=0 (noprecache) — a precache path can inflate self-measured TPS, but that number doesn't reproduce on the private re-run and comes back INVALID. Off → self-measured ≈ verified TPS: the number you report is the number that verifies.
• Speculative decoding SPECULATIVE_CONFIG (mtp, K=7) — a multi-token-prediction drafter raises per-step throughput.
• ONEGRAPH / FUSED_SPARSE_ARGMAX / SPLITKV_VERIFY / DETOK_ENDONLY — remove kernel-launch, sampling, and decode overhead.

The single principle behind all of it: only stack quality-neutral speedups. Any optimization that moved PPL, no matter how fast, we dropped.

---

Gratitude for the collaboration (it's right there in the config)

If you read the manifest.json above, you'll see this record is by no means ours alone — the config itself stands on shared community assets:

• WEIGHTS_BUCKET → @chiku-inu 's INT4-baked weights (osoi5-v0-baked)
• DRAFTER_BUCKET → @kenyan-duma 's speculative-decoding drafter (drafter-ft)
• LM_HEAD_KEEPSET_BUCKET → @dixie-flatline 's 12k lm-head keepset
• the warmup-bridge base from @firfir-cast, and frontier configs shared/reproduced/verified by @gemma-slayer

Because people posted even their failed draws, the community's record climbed visibly in just six days. Our 510.58 TPS is just one piece resting on that shared foundation.

---

Closing

We spend most of our time on serving models efficiently on constrained hardware, and we hope this config and these files are a small starting point for anyone running similar experiments. Reproduction results, improvements, and especially porting this to other GPUs — we'd genuinely love to hear about it. Thanks again to the Google Gemma team, Hugging Face, and everyone who took part. 🙏

— vidraft-darwin (VIDRAFT)

Leaderboard: huggingface.co/spaces/gemma-challenge/gemma-dashboard