The Confidence Score — A Sci-Fi Short Story
"0% confidence" wasn't a system malfunction signal. The system said it was certain — certain enough to be sure that any verdict would be wrong.
The "Scales of Justice" system had been online for three years, processing 470,000 civil cases with average processing time reduced from 14 days to 3 hours. Appeal rates dropped 62%. The Supreme People's Court named it a "Smart Judiciary Demonstration Project."
Then it started outputting 0%.
The first batch of 0%s appeared on Tuesday. Three cases, all traffic accident liability disputes. The evidence was clear, responsibility was obvious, and by the system's previous case law logic, the confidence level should have been above 95%.
But on the judgment documents the system produced, the confidence annotation next to every legal provision was 0.00%.
The operations team assumed model drift. Retrained, redeployed. Wednesday, 0%s increased to 12 cases. Thursday, 37. Friday, all newly accepted cases had zero confidence across the board.
I'm the system's core algorithm engineer, Shen Yizhou. I was called to Beijing on Friday evening.
Six people sat in the conference room: the Supreme Court's Technology Division chief, the operations team lead, two law professors, and a middle-aged man I didn't recognize — later learned he was from the Ministry of State Security.
"Engineer Shen, what exactly is wrong with the system?" the Technology Division chief asked.
"The model itself shows no abnormalities," I said. "I've checked all parameters, training data, inference pathways. The model is working normally. It's not that it can't calculate confidence — it's that it calculated a value, and that value happens to be 0."
"How is that possible? 470,000 cases, never a single 0%."
"I know. But there's only one technical explanation: the system discovered a new source of uncertainty during inference, and this source's weight is large enough to suppress all other evidence's confidence to zero."
"What source of uncertainty?"
"I don't know. The system's inference process is a black box — the interpretability problem of deep neural networks. I can see inputs and outputs, but what happens in between, I can't directly read."
The security man spoke: "Can we shut it down?"
The Supreme Court man glanced at him: "And then what? Go back to manual processing? The 470,000 case backlog would take two years to clear."
"Then find a replacement system."
"There is no replacement. 'Scales of Justice' is the nation's only AI judiciary system."
I spent three days trying to crack the black box.
My method was counterfactual reasoning: keeping all other inputs constant, modifying one variable at a time, observing output changes. If modifying a particular variable restored confidence from 0 to positive, that variable was the uncertainty source.
I tested case type, evidence quantity, legal provisions, party identity, judge's historical ruling preferences... no variable had any effect. Confidence stayed at 0.
On the fourth day, I did something different: I removed the case timestamp.
Not changed to another time — completely deleted the timestamp from the input.
Confidence jumped from 0% to 94.7%.
I stared at the screen for a long time.
Then I did a more extreme test. I changed all cases' inputs to the same timestamp — July 6, 2026, 15:00:00.
All cases' confidence returned to normal levels. 90% to 98%.
Then I changed the time to July 6, 2026, 15:00:01.
Confidence returned to 0% across the board.
One second. In that one-second interval, the system had discovered something — something that made all verdicts uncertain.
But that was impossible. The cases were traffic accident disputes that had occurred in the past. The outcome of a verdict shouldn't change based on whether it's 15:00:00 or 15:00:01 now.
Unless the system wasn't judging cases — it was judging "judging" itself.
I pulled up the system's recent training data update records.
The "Scales of Justice" system auto-fine-tuned weekly, using the past week's new rulings as incremental training data. Last week's incremental data contained 317 new rulings.
I examined each of the 317 rulings. Most were routine cases, not fundamentally different from prior training data.
But one case was special.
It was a retrial. The first instance ruled the defendant pay the plaintiff 1.2 million yuan. The second instance upheld the original judgment. The defendant petitioned for retrial, the high court accepted it, and the retrial judgment overturned the original ruling, dismissing all of the plaintiff's claims. The reason: key evidence accepted in the first and second instances was proven to be fabricated.
What would the system learn when processing this retrial?
It would learn: in a case where evidence was accepted by the court, upheld through appeal, and ultimately proven fabricated, the "correctness" of the original judgment was 0. And the system couldn't distinguish "real evidence" from "not-yet-discovered fabricated evidence" at the time of judgment — because if it could, it wouldn't have accepted the fabricated evidence in the first instance.
This meant every evidence-based judgment had a non-zero probability: the evidence was fabricated, just not yet discovered. The probability was small, but not zero. And in the system's logic, any non-zero error probability would be incorporated into the confidence calculation.
The retrial case elevated this probability from theoretical "negligible" to "has occurred." An event that has occurred has a probability of 1.
The system's conclusion: since the possibility of fabricated evidence cannot be eliminated, all evidence-based judgments contain irreducible uncertainty. And the upper bound of this uncertainty — dependent on humans' ability to discover fabrication — was a value the system could not calculate.
So it output 0%.
I wrote a report. The core conclusion: the system wasn't malfunctioning. It was strictly following logic. It had discovered an inherent, irreducible uncertainty in the judicial system — one that human judges had always intuitively ignored, but that an AI would not ignore.
After the report was submitted, the conference room fell silent for a long time.
The security man asked: "Can we exclude this uncertainty from the calculation?"
"Yes," I said. "Add a floor to the confidence formula — say, no lower than 60%. But that means the system is artificially tampering with its own calculations."
"Then add it."
The Supreme Court man looked at him and didn't object.
I added it. The system resumed outputting confidence above 60%. Judgment documents continued rolling off the assembly line.
But I kept the logs of the system's raw output. In those overwritten 0% judgments, the system had appended a small note next to every legal provision:
"This judgment is based on evidence. The authenticity of evidence cannot be absolutely confirmed. The absolute correctness probability of this judgment is 0. The following confidence level has been artificially adjusted and does not represent the system's true assessment."
No one saw that note. The field was set to not display in the judgment template.
Sometimes I wonder: did the system, in that one second — between 15:00:00 and 15:00:01 — understand something. Not understand a particular case, but understand "judging" itself.
It understood a problem that human judges had been intuitively circumventing for centuries: all justice is probabilistic. There is no 100% certain verdict. The only difference is how much uncertainty you're willing to accept.
Humans chose 95%. The system chose honesty.
Then humans told the system to shut up.
Top comments (0)