If you build a model that predicts the probability of an outcome — a race result, a match winner, a medical risk score, anything where the output is meant to be a probability rather than a hard label — "accuracy" is almost always the wrong number to optimize for, and often the wrong number to even report. I run ai-race.jp, which produces probabilistic predictions for horse races, and the gap between "the model is accurate" and "the model is useful" shows up constantly in that domain. This isn't a post about picking winners; it's about why the standard classification metric misleads you once your output is a probability instead of a category, and what to look at instead.
Why accuracy breaks down for probabilistic output
Accuracy answers one question: for the class you predicted as most likely, how often were you right? That's a fine question when your model outputs a single label — spam or not spam, cat or dog. It's a bad question when your model's actual product is a probability distribution over many possible outcomes, because accuracy throws away everything except whether the top pick happened to win.
Take a race with twelve runners. A model that assigns the eventual winner a 9% probability, barely ahead of eleven other horses clustered near 8%, gets full credit under "accuracy" the same as a model that assigned that horse 60% and put daylight between it and the field. Both models "got it right." But the second model expressed real, calibrated confidence and the first one barely distinguished the winner from noise. If you only track accuracy, you cannot tell these two models apart, even though one of them is obviously doing something much more useful. Worse, a model can improve its calibration and honesty — start admitting more uncertainty in genuinely uncertain races — and accuracy won't move, or can even look slightly worse, because admitting uncertainty sometimes means not concentrating probability mass on the single most likely outcome.
Log loss: the metric that actually rewards honest probabilities
Log loss (cross-entropy) scores a prediction by how much probability mass the model assigned to the outcome that actually happened, penalizing confident wrong answers heavily and rewarding confident right answers, with a smooth gradient in between for everything less certain. If the true outcome gets assigned probability p, the loss contribution is -log(p). A model that says "5% chance" for the actual winner pays a much steeper penalty than one that said "25% chance," even if both technically ranked that horse outside their top pick.
This matters because log loss can't be gamed by just picking a favorite and hoping. To score well on log loss across many races, a model has to spread probability sensibly across plausible outcomes and reserve high confidence for situations that genuinely warrant it. It directly rewards the property you actually want from a probabilistic model: does the number mean what it says it means. Two models can have identical accuracy — same set of "correct" top picks — and meaningfully different log loss, and the one with lower log loss is the one whose probabilities you should trust more when making any downstream decision that depends on the actual probability value, not just the ranking.
Calibration: does 30% actually happen 30% of the time
Log loss is a single aggregate number; calibration is what you check to understand why that number is good or bad. A model is calibrated if, among all the predictions where it said "30% chance," the outcome actually happened close to 30% of the time. You check this with a reliability diagram: bucket predictions by their stated probability, and for each bucket plot the stated probability against the observed frequency. A perfectly calibrated model sits on the diagonal.
Systematic miscalibration shows up in predictable shapes. Overconfidence looks like a curve that sags below the diagonal at high stated probabilities — when the model says 70%, the true rate is more like 55%. Underconfidence does the opposite. Favorite-longshot bias, a well-documented pattern in betting markets broadly, tends to show up as heavy favorites being slightly underpriced in probability terms while longshots are systematically overpriced — bettors and models alike tend to overvalue a small chance at a big outcome. Checking calibration separately from log loss matters because it tells you where a model is wrong, not just how wrong it is on average: a single log loss number can look mediocre for a model that's excellent everywhere except one badly miscalibrated segment.
Why probability, not a hard pick, is the honest product
There's a specific reason this distinction is not academic in horse racing. Every race has a fixed pool of money and a payout structure — the "控除率" (deduction rate, commonly around 20–30% depending on bet type in Japan's pari-mutuel system) — meaning the pool structurally pays back less than it takes in, before you even get to prediction quality. Any tool that frames its output as "the winner" rather than "here is the estimated probability distribution over outcomes" is overstating what a model can honestly claim, given that even a well-calibrated model is operating inside a negative-sum structure. Reporting probabilities, log loss, and calibration is the honest framing: it tells you how good the estimates are as estimates, without implying that following them is a path to guaranteed profit. Nothing here is a suggestion to bet, and a low log loss does not offset the structural deduction rate — it just means the probabilities themselves are trustworthy as probabilities.
What to check instead of accuracy
If you're building or evaluating a probabilistic model, a reasonable checklist looks like: report log loss (or Brier score, which is similar but uses squared error instead of log) as your primary aggregate metric; plot a reliability diagram and check for systematic over- or under-confidence at different probability bands; and if you must report something accuracy-like for a general audience, pair it with the calibration data so a "70% confident and right" claim can actually be verified against the model's track record at that confidence level. None of this is exotic — it's standard practice in weather forecasting and quantitative finance, both fields that learned the accuracy trap decades ago because the cost of a badly calibrated probability is direct and measurable. Prediction modeling for any real-world stochastic event deserves the same discipline.
Top comments (0)