FINAL-Bench LEADBOARD: How a Single Data-Split Decision Moves Drug-Prediction AUROC by 0.21
TL;DR: VIDRAFT has released FINAL-Bench's LEADBOARD, a 21-board public benchmark for drug-property prediction that exposes how random vs. time-based data splits produce dramatically different AUROC scores on the same model. Using hidden test labels, harder held-out sets, and assay noise floors, it quantifies evaluation artifacts that routinely inflate reported gains in drug-discovery AI. Developers and ML researchers working on molecular property prediction should read this before trusting their next benchmark number.
What it is
FINAL-Bench LEADBOARD is a structured evaluation framework for drug-property prediction models, released by VIDRAFT CEO Min-sik Kim on August 22, 2026. The leaderboard is hosted on Hugging Face and covers seven disciplines across 21 boards.
Key dataset facts from the public announcement:
- 212,670 training compounds available to participants
- 18,382 held-out compounds whose labels are withheld — participants never see the ground truth directly
- Benchmark scope includes diverse drug-property prediction tasks, not a single assay
- The inaugural case study uses hERG (a potassium-channel cardiac liability target), extracting records from ChEMBL 37
The project's stated goal is practical: standardize how drug-prediction tools are evaluated before more systems claim to beat each other. Drug-discovery AI has a well-known benchmark-gaming problem, and FINAL-Bench is attempting to be the more honest ruler.
How it works
FINAL-Bench surfaces two distinct failure modes that inflate reported performance in molecular ML:
1. Data-split leakage
Medicinal chemistry workflows produce families of structurally similar compounds — analogs of a promising scaffold accumulate in literature over time. A random split scatters these analogs across train and test sets, so the model effectively memorizes a scaffold and predicts variants of it. A temporal (time-based) split places compounds reported after a cutoff date in the test set only, forcing the model to generalize to genuinely novel chemical space.
FINAL-Bench makes this distinction the centerpiece of its evaluation design. The same model, the same fingerprints, and the same hyperparameters are tested under both conditions. Only the split strategy changes — and the AUROC gap that emerges is the artifact you were inadvertently reporting as model quality.
2. Assay noise floors
Biochemical assay measurements carry inherent noise. If the performance gap between two models is smaller than the assay's noise floor, declaring a winner is statistically meaningless. FINAL-Bench builds noise floor awareness into its evaluation logic so that small reported gains are contextualized rather than celebrated.
Hidden labels
Test labels are withheld from participants, preventing reverse-engineering of the held-out set. This is a standard anti-gaming mechanism (similar to how Kaggle private leaderboards work) but relatively uncommon in academic molecular ML benchmarks, where test sets are often fully public.
Benchmarks & results
The headline result comes from the hERG case study documented in FINAL-Bench's Hugging Face community article:
| Split Strategy | AUROC |
|---|---|
| Temporal split (pre-2022 train / post-2022 test) | 0.606 |
| Random split (5-seed average) | 0.818 |
| Random split (range across seeds) | 0.803 – 0.830 |
Gap: 0.211 AUROC points — on an identical model with identical features.
The dataset pipeline for hERG: 12,021 records extracted from ChEMBL 37 with document years → 11,972 retained after unit checks and parsing → 9,788 unique compounds after deduplication.
The baseline model used is a standard gradient-boosted tree (LightGBM) on Morgan fingerprints — a common, reproducible starting point, not a novel architecture. The point is that the split, not the model, is driving the number you see in the abstract.
How to try it
FINAL-Bench LEADBOARD is publicly accessible via Hugging Face. The primary source for the benchmark is the VIDRAFT community article on Hugging Face (huggingface.co), where the leaderboard and dataset details are documented.
To explore the benchmark and leaderboard:
- Visit the FINAL-Bench announcement on Hugging Face (linked from the primary source at
huggingface.co) - Review the 21-board leaderboard structure and submission guidelines published there
- The training compounds (212,670) are available to participants; held-out test labels remain withheld
No GitHub repository, API endpoint, or pip install package was announced in the source article at time of publication. Check the Hugging Face community page for the most current access instructions and any tooling released alongside the leaderboard.
FAQ
Q: Why does the random-vs-temporal split difference matter for my model evaluation?
A: If your train and test sets contain structural analogs of the same scaffold, your model learns the scaffold pattern — not a generalizable property prediction rule. Temporal splits enforce a harder, more realistic boundary. A 0.21 AUROC gap on the same model is not an edge case; it is the expected magnitude of this artifact in a target like hERG where scaffold families dominate the literature.
Q: Can I submit my own model to FINAL-Bench LEADBOARD?
A: The leaderboard is public and hosted on Hugging Face. Submission mechanics are documented in the VIDRAFT community article on Hugging Face. Test labels are withheld from participants, so scoring goes through the benchmark's evaluation pipeline rather than self-reporting.
Q: Is this limited to hERG, or does it cover other drug properties?
A: hERG is the case study used to demonstrate the split-gap finding, but FINAL-Bench covers seven disciplines across 21 boards. The full scope of targets and properties is detailed in the Hugging Face announcement.
Originally reported by RuntimeWire (2026-08-22) — source article.
Top comments (0)