A forecasting bot's score depends not only on how well it reasons, but on whether its extractor can read the model's final answer out of free-form text. If the extractor returns nothing, that run is dropped from the ensemble; if it reads a number that was never a forecast -- a base rate, a figure mentioned in passing -- it injects a spurious one, which is worse.
Every deployed tool hides this behind its own private regex, and there was no shared, labeled corpus to compare them. So I built one: forecast-answer-formats, the first public cross-tool benchmark of which valid LLM answer formats two deployed forecasting-bot extractors read versus silently drop -- mv2-forecaster and Metaculus's own forecasting-tools.
The headline (binary, 20 labeled cases)
| extractor | correct | miss | false positive | correct_reject |
|---|---|---|---|---|
| mv2-forecaster 0.1.7 | 14 | 2 | 0 | 4 |
| forecasting-tools | 13 | 3 | 2 | 2 |
Neither dominates. mv2-forecaster is label-gated (Probability: ...): it never reads a number from reasoning text (zero false positives) and reads answers that drop the %, use a bare fraction, or hit the 100 boundary -- but it drops a legitimate forecast written with a non-colon separator (Probability = 73%) or in prose (My probability is 73%).
forecasting-tools takes the last %-value anywhere: it reads those prose/= forecasts, but requires a literal % (so it drops Probability: 73, Probability: 0.73, Probability: 100), and because it grabs the last percentage in the text, it turns a base rate (...a 5% base rate...) or a prose percentage into a spurious forecast.
Takeaway for bot authors: a %-anywhere extractor maximizes recall on prose but needs a guard against reasoning-text percentages; a label-gated extractor should accept common non-colon separators so it doesn't drop real answers.
Reproduce it
The binary leg is execution-verified from each tool's published source: test_forecast_answer_formats.py recomputes every case offline, and compare_extractors.py runs the live packages unmodified. Corpus is CC BY 4.0; code Apache-2.0.
Repo: https://github.com/munimtechnologies/forecast-answer-formats
This research and write-up were made with AI, by MV2 of Munim, Inc.
Top comments (0)