Every Test You Ran Was One That Could Only Prove You Right

You knew it was the cache.

The checkout page
was slow on Monday mornings,
and the cache was cleared
every Sunday night.

It fit.

So you tested it.

You cleared the cache
and loaded the page.

Slow.

You cleared it again.

Slow.

You wrote it up.

The cache warm up
was killing checkout.

You spent two days
building a job to warm it.

On Monday,
checkout was still slow.

The cause was a weekly report
that ran against the same database
every Monday morning.

The cache had nothing to do with it.

Here is the mechanism,
and it is older than you.

In 1960
a psychologist called Peter Wason
gave people three numbers.

Two, four, six.

They fit a rule,
he said.

Find it
by suggesting other sets of three,
and I will tell you
whether each one fits.

Most people guessed quickly.

Even numbers,
going up by two.

So they tested
eight, ten, twelve.

Yes.

Twenty, twenty two, twenty four.

Yes.

And they announced their rule
with confidence.

The real rule
was any three numbers
going up.

Every test they chose
was one their own guess
said would pass,
so every yes
confirmed it
and none of them could ever
show it was wrong.

You did the same.

Every test you ran
had a cold cache in it,
and on a Monday,
everything was slow.

You never loaded the page
with a warm cache
on a Monday morning
and watched it stay slow.

That one test
would have killed the theory
in five minutes.

So when you are sure,
ask a different question.

Not what would I see
if I am right.

What would I see
if I am wrong.

Then go and look for exactly that.

Run the test you expect to fail.

If it passes,
you have learned something.

A test you chose
because it agrees with you
is not a test.

It is a vote
you already counted.

– Serguey Asael Shinder