We cut the activity weight from 30 to 10 across three seasons, and the data made us do it

The Open Discovery Challenge has been running for a couple of weeks now. Three seasons
are live at once, and as of today the numbers look like this:

Season	Disease	Target	Submissions	Participants	Top score	Closes	Prize
1	Malaria	PfDHODH	3,140	88	85.7	30 Sep	$1,000
2	Tuberculosis	InhA	1,956	38	83.7	31 Oct	$2,000
3	Chagas disease	T. cruzi CYP51	1,100	25	72.4	30 Nov	$1,000
		Total	6,196	97 unique			$4,000

Anyone can enter. Pick a target, design a molecule with whatever AI you like, submit a
SMILES string, get scored on six axes against published criteria.

This post is about one number that changed between those three seasons, because the
reason it changed is the most interesting thing we have learned so far.

The number

Each season scores out of 100 across six axes. Here is how the weights moved:

Axis	S1 Malaria	S2 Tuberculosis	S3 Chagas
Whole-cell activity	30	20	10
Target binding	20	25	30
Selectivity	20	25	30
ADMET	15	15	15
Novelty	10	10	10
Synthesizability	5	5	5

Activity — does the compound actually kill the parasite in a cell — went from being the
single heaviest axis to the lightest scored one. That looks backwards. Killing the
pathogen is the point.

We did not decide this. Three pre-registered tests decided it, one season at a time.

How a season opens

Before a season takes a single submission, we run acceptance tests on the activity model
and write the thresholds down first. The rule, fixed in advance: if a test fails, the
activity axis loses weight. We do not get to renegotiate after seeing the result.

Season 3 was the clearest case.

Test	Threshold	Measured	Verdict
A1 — lower-bound coverage	≥ 0.88	0.8952	pass
A2 — active vs inactive separation	≥ 0.50 log	0.389	fail
A3 — Pearson r	≥ 0.40	0.82	pass

One failure, so activity dropped to 10 and the points moved to binding and selectivity,
which are measured rather than predicted.

The failure was not the model's fault

This is the part worth dwelling on.

A2 asks whether the model separates compounds that work from compounds that do not. It
came back at 0.389 log, under the 0.50 bar. The obvious reading is that the model is bad.

Then we looked at the labels.

In the T. cruzi whole-cell data, compounds tagged "inactive" have a median pAct of 4.49.
That is not "does nothing." That is "still alive at 32 µM, so we stopped testing."
Actives sit at 5.20. The separation in the labels themselves is 0.705 log.

The model achieved 55% of a ceiling that the data imposes. A compound that works at 20 µM
gets filed as inactive, because nobody screened deeper.

So the honest move was not to tune the model. It was to stop paying 30 points for a
measurement the source data cannot support, and pay for the things we can measure.

Three seasons, three different failure modes

What makes this interesting is that the seasons failed differently.

Season 2 (tuberculosis) had the opposite profile. Binary classification was strong —
AUROC 0.860 on held-out scaffolds — but the numeric prediction needed a lower bound
sitting ~2.0 log below the estimate to hold 90% coverage. MIC measurements disagree by
~0.6 log even inside a single assay. Good at sorting, poor at the number. Activity went
to 20.

Season 3 (Chagas) was the mirror image. AUROC only 0.689, but MAE 0.51 log and
Pearson r 0.82. Weak at the yes/no call, decent at the magnitude. So we deleted the
threshold term entirely and kept a continuous ramp for 10 points.

Same six axes, same pipeline, three genuinely different instruments. If we had shipped
one fixed rubric across all three, two of them would have been scoring noise with a
confident-looking number attached.

The other check every season has to pass

Reference compounds. Known drugs go through the scorer before the season opens, and if
the ordering is wrong, the season does not open.

Season 3, before opening:

Compound	Score
posaconazole	36.14
ketoconazole	34.23
itraconazole	32.87
fluconazole	18.58
benznidazole	2.40
nifurtimox	2.02
caffeine	0.081
aspirin	0.050

Two conditions, both written down beforehand. An irrelevant control has to land far below
the active azoles — caffeine at 0.081 against 32.87 clears that by a wide margin. And
posaconazole, the strongest azole against this target, has to beat fluconazole, the
weakest — 36.14 against 18.58.

Note where benznidazole and nifurtimox land. Those are the actual clinical drugs for
Chagas, and they score around 2. They should. Neither works through CYP51. This season is
a CYP51 track, so a drug with a different mechanism gets scored as off-target, which is
correct behaviour and would look like a scandal if you did not know the target.

We publish the reference compounds on the leaderboard as amber rows. They get scored and
they never get ranked, because they are there to show you the scale, not to compete.

What we are not saying

A score here is a computational estimate. No molecule in this challenge has touched a
cell, and the distance between a good score and a drug is measured in years and failures.

The rubric changing between seasons is a feature, not a retrofit — every change happened
before that season opened, under rules written before the tests ran. Once a season is
live, the rules are frozen. Entrants scored under published criteria do not get re-scored
because we learned something later.

And the activity weight dropping is not a claim that whole-cell activity matters less. It
is a claim that we cannot measure it well enough on this data to charge 30 points for it.

Why malaria, tuberculosis and Chagas

Because the science is tractable and the economics are not.

We know the parasites, we know the enzymes, there are decades of literature. What is
missing is a pipeline, because the people who need these drugs cannot fund the discovery.
Chagas is the sharpest version of it — roughly 6 to 7 million people infected, and the
two drugs in use both predate 1975.

That is not a problem AI solves by being clever. It is a problem that gets smaller when
the barrier to trying goes away.

97 people have tried so far. All three seasons are open.

Open Discovery Challenge

---

VIDRAFT Inc. is a deep-tech company founded in March 2024 and headquartered in Seoul,
South Korea, developing Pre-AGI artificial intelligence models and quantum computing
technology in-house.