Fenix

Posted on Jul 11

dock-confidence: calibrated P(RMSD < 2Å) for docking poses (ECE 0.109, 15/15 tests)

#bioinformatics #drugdiscovery #opensource #python

dock-confidence: calibrated P(RMSD < 2Å) for docking poses

A post-hoc calibration layer over any docking engine (DiffDock, AutoDock, Smina). It estimates a calibrated probability that a pose is near-native and computes Expected Calibration Error (ECE) to filter decoys — no retraining required.

The problem (EN)

Molecular docking is run at scale across pharma and biotech. Yet empirical scoring functions remain the main bottleneck in docking: "they routinely fail to rank near-native poses above decoys" (AgenticPosesRanker, arXiv:2605.03707). Roughly half of all eligible PDBbind systems show a scoring-function failure — the crystallographically correct pose is not the top-scored one.

Generative docking (DiffDock, FlowDock) produces poses, and DiffDock even emits a confidence score. But its own README warns it "can be hard to interpret and compare across complexes" and is "NOT a direct measure of binding affinity." The score is not calibrated and not comparable across complexes.

The market gap, quantified

GitHub signal	Repos
`molecular-docking`	183
`docking pose confidence RMSD calibration`	0

Zero open-source tools solve the calibrated-confidence layer. dock-confidence fills that gap.

El problema (ES)

El docking molecular se ejecuta a gran escala en pharma y biotech. Sin embargo, las funciones de puntuación empíricas siguen siendo el cuello de botella principal: "fallan rutinariamente en rankear poses nativa-cercanas por encima de decoys" (AgenticPosesRanker, arXiv:2605.03707). Cerca de la mitad de los sistemas PDBbind elegibles presentan un fallo de puntuación.

El docking generativo (DiffDock, FlowDock) produce poses, e incluso emite una confianza. Pero su propio README advierte que "puede ser difícil de interpretar y comparar entre complejos" y "NO es una medida directa de la afinidad de unión". La puntuación no está calibrada ni es comparable entre complejos.

El hueco del mercado, cuantificado

Señal en GitHub	Repos
`molecular-docking`	183
`docking pose confidence RMSD calibration`	0

Cero herramientas open-source resuelven la capa de confianza calibrada. dock-confidence llena ese vacío.

How it works (EN)

dock-confidence does not generate poses. It verifies and calibrates them:

Parse — reads DiffDock/Smina SDF, AutoDock PDBQT, or AutoDock .dlg into a single Pose dataclass.
RMSD — heavy-atom symmetric RMSD to the crystal native, via Maximum Common Substructure alignment (robust to atom reordering; different molecules yield None, never a bogus 0.0).
Calibrate — maps the raw dock score to a calibrated P(RMSD < 2.0 Å) using either a Platt logistic fit (trained on a calibration set) or a documented heuristic fallback.
Report — Expected Calibration Error (ECE, Guo et al. 2017), top-1 best-pose accuracy, decoy filter, and a reliability diagram.

Results (verified, leak-free holdout)

Metric	Value
Tests passing	15 / 15 (incl. SDF/PDBQT/DLG, missing-native, different-molecule, atom-order invariance)
CI	Green on GitHub Actions (Python 3.12)
ECE (calibrated, Platt)	0.109 (measured on a 0-system-overlap holdout)
ECE (raw score)	0.386
AC7 verdict	PASS (calibrated < raw)
Top-1 accuracy (holdout)	0.400 vs random 0.100

Cómo funciona (ES)

dock-confidence no genera poses. Las verifica y calibra:

Parse — lee SDF de DiffDock/Smina, PDBQT de AutoDock, o .dlg de AutoDock en una sola dataclass Pose.
RMSD — RMSD heavy-atom simétrico al cristal nativo vía alineación por MCS (robusta a reordenamiento; moléculas distintas dan None, nunca un 0.0 falso).
Calibra — mapea el score crudo a una P(RMSD < 2.0 Å) calibrada usando ajuste logístico de Platt (entrenado en conjunto de calibración) o un fallback heurístico documentado.
Informe — Error de Calibración Esperado (ECE, Guo et al. 2017), top-1 accuracy, filtro de decoys y diagrama de fiabilidad.

Resultados (verificados, holdout sin leakage)

Métrica	Valor
Tests pasando	15 / 15 (incl. SDF/PDBQT/DLG, nativo ausente, molécula distinta, invarianza de orden)
CI	Verde en GitHub Actions (Python 3.12)
ECE (calibrado, Platt)	0.109 (medido sobre holdout de 0 overlap de sistemas)
ECE (score crudo)	0.386
Veredicto AC7	PASS (calibrado < crudo)
Top-1 accuracy (holdout)	0.400 vs random 0.100

Transparency note (EN / ES)

After an independent audit, the originally reported ECE of 0.080 was found to be measured with train/test leakage: the Platt model was fit on a set of systems and then evaluated on the full pose set, which still included those same systems. That inflated the calibration number.

The fix splits the synthetic dataset by system_id with zero overlap between the training and evaluation sets. Platt is fit on train_poses; ECE, top-1 and the decoy filter are evaluated on test_poses (a clean holdout). An internal assertion guards against regression: train_sys ∩ test_sys == ∅.

The honest, leak-free numbers are ECE 0.109 < 0.386 (was 0.080 < 0.368 with leakage). The direction of the claim is unchanged — calibrated confidence still beats the raw score — but the figure is now what it claims to be. Test count rose from 14 to 15 with the addition of an atom-order-invariance test.

Nota de transparencia: tras una auditoría independiente, el ECE original de 0.080 se midió con leakage train/test: el modelo de Platt se ajustaba sobre unos sistemas y luego se evaluaba sobre el conjunto completo, que incluía esos mismos sistemas. Eso inflaba el número de calibración.

El fix separa el dataset sintético por system_id con cero overlap entre entrenamiento y evaluación. Platt se ajusta sobre train_poses; el ECE, la top-1 y el filtro de decoys se evalúan sobre test_poses (holdout limpio). Un assert interno evita la regresión: train_sys ∩ test_sys == ∅.

Los números honestos sin leakage son ECE 0.109 < 0.386 (antes 0.080 < 0.368 con leakage). La dirección del claim no cambia — la confianza calibrada sigue ganando al score crudo — pero la cifra ahora es lo que dice ser. El conteo de tests subió de 14 a 15 al añadir un test de invarianza de orden de átomos.

Try it (EN / ES)

python3 -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"

# Parse + calibrate
dock-confidence parse --input poses.sdf --native crystal.pdb \
    --calibrate-mode platt --out poses.json

# Validate on the built-in fixture (leak-free holdout)
dock-confidence validate --fixture --seed 42 --n-systems 20 --mode platt

python3 -m venv .venv && . .venv/bin/activate
pip install -e ".[dev]"

# Parsear + calibrar
dock-confidence parse --input poses.sdf --native crystal.pdb \
    --calibrate-mode platt --out poses.json

# Validar en el fixture integrado (holdout sin leakage)
dock-confidence validate --fixture --seed 42 --n-systems 20 --mode platt

Honest limitations (EN / ES)

The heuristic mode is a documented fallback with higher ECE than platt; we report ECE without fabrication.
Real-dataset validation (CASF-2016, PDBbind v2016, PoseBusters, DockGen) is the next step — those tarballs require manual browser download (scripted requests return HTTP 403). The reproducible fixture proves the calibration logic, not yet the real-dataset ECE.
No GPU required: the calibrator uses physical/score features, not a retrained diffusion model.

El modo heuristic es un fallback documentado con ECE mayor que platt; reportamos ECE sin maquillar. La validación con dataset real (CASF-2016, PDBbind v2016, PoseBusters, DockGen) es el siguiente paso — esos tarballs requieren descarga manual (HTTP 403 a requests scriptados). El fixture reproducible prueba la lógica de calibración, no aún el ECE con datos reales.

DEV Community