How We Cut Target Leakage from 92% to 0.1% in Court Outcome Prediction (and Built a Triple-A MCP Server)

Elio Coppola — Fri, 11 Sep 2026 00:39:12 +0000

When building AI for court outcome prediction, there is a massive hidden trap that invalidates most benchmarks: target leakage.

In Dutch court rulings, roughly 92% of raw texts contain the actual outcome verbatim (dictum or conclusion sentences like "the court dismisses the claim"). If you feed raw text to a model, it does not learn legal logic. It simply learns to read the answer back to you.

Here is how we solved this across 609,715 cases, built a traceable LightGBM model, and exposed it as a zero-dependency open-core MCP server.

1. The Pre-Training Cut: 92% to 0.1% Leakage

Before training any classifier, we implemented a strict sanitization step:

The dictum, summary lines, and outcome-announcing phrases are stripped from the text.
We continuously measure residual outcome markers.
Result: Leakage dropped from 92% to 0.1% (around 1 in 1,000 texts).

Only on this sanitized dataset did we train.

2. Why LightGBM Instead of an LLM

We intentionally picked LightGBM over deep neural networks or fine-tuned LLMs:

Fast and cheap: Sub-10ms inference without GPUs.
Traceable: Clear tree structures and feature importance.
Deterministic calibration: If confidence drops below 55%, the model does not guess. It returns "insufficient certainty".

3. Benchmark on 609,715 Cases (Out-of-Fold)

Evaluated through 5-fold cross-validation, strictly measured out-of-fold:

Overall Accuracy: 78.2% (against a 43.7% majority baseline)
Macro-F1: 77.1%
Per-Class F1:
- Dismissed: 0.827
- Partly granted: 0.726
- Granted: 0.761

Domain Breakdown:

Criminal Law (n=105,151): 82.6% accuracy, 0.804 macro-F1 (strongest performance)
Administrative Law (n=316,273): 81.0% accuracy, 0.685 macro-F1 (high accuracy, but government victory is the majority class)
Civil Law (n=188,177): 71.0% accuracy, 0.656 macro-F1 (most complex due to factual nuances)

4. Model Context Protocol (MCP) Interface

To make this accessible to AI assistants (Claude, Cursor, autonomous agents), we wrapped the pipeline into an MCP server.

Zero external dependencies: Single Python file using only standard library (sys, json, urllib).
Audited on Glama: Triple-A rating (5/5 on coherence and completeness).
3 Tools:
1. rechtspraak_cijfers (keyless): Benchmark statistics and baseline metrics.
2. lekkage_check (keyless): Paste any legal text to test for outcome leakage before and after the cut.
3. voorspel_uitkomst (key required): Outcome risk classification.

DEV Community: Elio Coppola

How We Cut Target Leakage from 92% to 0.1% in Court Outcome Prediction (and Built a Triple-A MCP Server)

1. The Pre-Training Cut: 92% to 0.1% Leakage

2. Why LightGBM Instead of an LLM

3. Benchmark on 609,715 Cases (Out-of-Fold)

4. Model Context Protocol (MCP) Interface

Links