An AI reviewing a report flagged real earnings as an "impossible number" and called it data contamination — it turned out to be genuine
This is the English version of a post originally written in Korean for my algorithmic trading system devlog(new tab).
Today happened to fall on a loose homeschooling day, so I had more time to work than a typical weekday. Maybe that's why so much came out of it.
Watching AI be confidently wrong
An AI helping review a report saw one company's financial figures come in unusually high on a certain metric and declared, very confidently, that "a number like this has never appeared at any point in this industry or company size" — flagging it as contaminated data.
Checking it myself, that wasn't the case. The earnings really were that good — a recent boom in the relevant industry meant a figure that would have been "impossible" by old standards was now real.
Asking the AI why it had judged this way, it turned out its whole basis for "this has never happened" was just world knowledge frozen at its training cutoff. It had no idea a real change had happened since then.
Thinking back, something similar happened recently too — another AI tasked with checking whether a report's content was real or fabricated wrongly judged an actual, recent event as "never happened." Today's incident was the same trap.
A judgment like "this number is too big to make sense" sounds logically solid on the surface, but it only holds within the range of past data that AI happens to know. The claims worth trusting are the ones that hold regardless of world knowledge, like "these two numbers inside the document contradict each other." A claim of the form "this doesn't exist in the world" — however confidently an AI states it — I've decided not to take at face value without checking it myself.
A bug that had the wrong company name attached
Looking closely at one report, I found that the section handling financial analysis was pulling the correct ticker but attaching a completely different company's name to it. The numbers themselves belonged to the right company; only the label on top was wrong.
The cause was simple — the part that looks up financial data by ticker and the part that attaches a human-readable company name were running independently, and the naming logic silently failed to recognize the domestic ticker format. Because the numbers were correct, it looked fine at a glance, but the report was actually going out with the right data under the wrong label — which made it tricky to catch. I fixed the name injection and, as a safeguard, added a rule that defaults to neutral whenever a company name can't be resolved but the report still reaches an extreme verdict.
A dilemma that came with switching to a smarter model
I swapped the local model used for writing reports for a bigger, smarter one. On clean input, it was clearly better — the older model tended to retreat into "hold" whenever things were ambiguous, while the new one reaches firm conclusions backed by stated reasoning.
But that created a dilemma. When fed input that contained the mislabeled-name bug above, the old model's vague "hold" verdict caused relatively little harm, while the new model confidently used the wrong information as grounds for an extreme verdict, including a strong sell.
A smarter model can be more dangerous — as judgment improves, so does its ability to package a mistake in more convincing form when the input is wrong. So I only rolled the new model out live after fixing the naming bug first, and for now I've also turned on a monitor that automatically pulls any unusually extreme verdict out of the report until a human reviews it.
Also today
- I removed some of the multiple daily scheduled reports and moved that role (watching for anomalous price spikes) to real-time monitoring instead. Rather than sweeping at fixed times, it now catches anomalies the moment they happen.
- I added a freshness indicator at the bottom of reports, showing how current the data is. There had been a few past incidents where data quietly ran several days stale; now that's visible at a glance.
The biggest lesson from today: the more confident an AI's "that doesn't make sense" answer sounds when verifying something, the more it deserves a second look. That confidence isn't necessarily from actually checking — it can just be logic that only holds together inside the AI's own outdated view of the world.
Top comments (0)