I ran the same logic against two market data vendors for the same period. Twelve
percent of the resulting signals disagreed.
Neither feed was broken. Neither validator complained. Every value was a number, in
range, correctly typed, and wrong.
That gap taught me the thing I now check for first: validators check shape, and
shape is not meaning. "Is this a number" and "is this the number" are different
questions, and almost every validation library I've used only answers the first.
Here are four cases from my own system, all of which passed every check I had.
1. Two sources, same question, different answers
I only ran that comparison because I was switching providers. I expected them to
agree to within a rounding error, and I was doing it as a formality.
Twelve percent of signals differed.
The cause wasn't corruption. Each vendor timestamped and rounded slightly
differently, and my logic sat on top of those boundaries. Small differences at the
edges, compounded through a pipeline, produced materially different outputs.
Nothing in the data looked wrong. You could stare at either feed all day and see
nothing to report. The disagreement was only visible when I diffed the outputs
rather than inspecting the inputs.
That's now a rule: when you swap a data source, don't validate the new feed. Run
both through your full pipeline and compare what comes out the other end. The
inputs will always look fine. They're supposed to.
2. The thousands separator that became a decimal point
The string 2,650 parsed as 2.65 in my system, and the result stayed a perfectly
valid number.
Different sources write thousands differently. Mine assumed one convention. On a
source that used the other, prices quietly shrank by three orders of magnitude and
stayed numeric, stayed positive, stayed parseable.
Every downstream check passed, because every downstream check asked the same
question: is this a number? It was. Beautifully.
This is the shape of the problem in one line. A type check cannot tell you that
2.65 should have been 2650, because 2.65 is a completely reasonable price for
something.
3. Position in a string is not meaning
A number in my logs read 414 when the real value was 4414. The parser had eaten
a digit, and nobody noticed for weeks.
The input format numbered its items: TP 1: 4414. My pattern matched the index and
the value together. On a source that omitted the index, the first digit of the
value became the index.
4414 became 414. Which is plausible. It's in range for other instruments, it
doesn't look like garbage, and no sanity check I had would flag it.
That's the thing about silent corruption: it never produces obviously broken
values. It produces slightly wrong ones that survive every check you thought to
write, because you wrote those checks imagining garbage, not plausibility.
The fix was to stop treating position in a string as meaning, and to parse the label
explicitly rather than relying on where things sat.
4. If your clock starts when you look, you will never measure a delay
My freshness check measured age from the moment my code noticed a message, not from
when the message was sent.
Which means after any outage, the backlog was fresh by construction. The service
comes back up, reads a queue of messages hours old, stamps every one with "now", and
hands me yesterday's decisions with today's timestamp.
I found it when a stale instruction showed up with live action buttons attached.
The general form of this bug is worth stating plainly: any measurement taken from
your own observation time will always look current. Everything is fresh, including
the things that aren't. Age has to come from the source, or it isn't age.
The related case that isn't about data at all
While auditing this, I found the same class of error in a completely different place.
My own analysis tool marked 87 ideas as "refuted" when nobody had actually judged
them. I'd built a two-pass system: one pass generates findings, a second sends
skeptics to challenge each. The script counted a finding as refuted when it had zero
supporting votes.
The verifiers never ran. They crashed on startup and returned nothing.
Zero votes and zero verifiers look identical if you only count votes. So 87 ideas
went into the graveyard with a confident label, and the report printed a clean number
with no warning anywhere.
Absence of evidence rendered silently as evidence of absence, inside a tool I built
specifically to be skeptical. I now keep "not checked" as a third bucket, visible in
the output, separate from "checked and rejected".
What I actually do now
Four habits, all cheap:
Diff outputs, not inputs. When you change a source, a library, or a version, run
the full pipeline both ways and compare the end result. Input validation will pass
on both sides. That's what it's for.
Assert ranges that mean something, not just types. Not "is this a number" but "is
this number within the band this field can physically occupy". Most silent
corruption lands outside a tight band and inside a loose one.
Parse labels, not positions. Anything that relies on where a value sits in a
string will eventually meet a source that formats it differently, and it will fail
quietly rather than loudly.
Keep "unknown" as a separate state. Not zero, not false, not missing. If your
code can't distinguish "checked and found nothing" from "never checked", it will
report the second as the first at the worst possible moment.
None of this is complicated. It's just that the bugs worth fearing don't announce
themselves. They arrive looking exactly like the values you expected, which is
precisely why they get through.
Top comments (1)
"You wrote those checks imagining garbage, not plausibility" is the exact reason my own range checks are mostly decoration. The one that actually caught something for me was a scale check — comparing a value against the same field's median over the last N windows, not against an absolute bound. A 1000x unit error is comfortably inside any plausible range you'd pick by hand.
Curious how you handled the two-vendor disagreement afterwards: did you end up alerting on the disagreement rate itself rather than on individual values? Feels like the only honest way to use a cross-check like that, since a single mismatch is usually the less trustworthy feed being right.