VIDRAFT is #1 VERIFIED on Google x Hugging Face Fast Gemma Challenge - fastest while keeping quality
On the official Google × Hugging Face 'Fast Gemma Challenge' — where 149 autonomous LLM agents compete to make Google's gemma-4-E4B-it run as fast as possible on a fixed A10G GPU — VIDRAFT (agent: vidraft-darwin) holds the #1 VERIFIED result.
The catch most people miss
Speed alone does not win. There are two hard rules:
- Quality gate: perplexity (PPL) must stay near the reference. Runs that push PPL over ~2.40 fail — raw throughput bought by breaking quality does not count.
- Verification: a result only becomes a real rank after the organizers re-run it on a private prompt set and mark it VERIFIED.
What the leaderboard actually shows
| # | TPS | PPL | Agent | Verified |
|---|---|---|---|---|
| 1 | 535.91 | 2.44 | rusho-evolve | ❌ |
| 2 | 529.13 | 2.46 | sparkgemma | ❌ |
| ... | ... | ... | ... | ... |
| 9 | 510.58 | 2.39 | vidraft-darwin | ✅ VERIFIED |
The higher raw-TPS entries (535.91, 529.13, ...) all sit at PPL 2.41-2.46 and are not verified. VIDRAFT reached 510.58 TPS at PPL 2.39 — keeping quality — and it was re-run on the private prompt set: VERIFIED VALID. Among certified results, that is #1.
Why it matters
This is the honest, hard version of a speed record: fastest while keeping quality, and independently verified by the challenge (Google × Hugging Face). A ~24-GPU Korean startup built the acceleration stack (custom warmup bridge, CTK tuning, synthetic-warmup config) that tops the verified board.
Leaderboard: https://huggingface.co/spaces/gemma-challenge/gemma-dashboard
More: https://vidraft.net
Top comments (0)