The bug report
came with a guess.
Probably the cache,
the reporter wrote,
in the first line.
You pasted the report
and the logs
into the model
and asked for the likely cause,
with its reasoning,
step by step.
It said the cache.
And the reasoning was good.
Timestamps,
a stale read,
a plausible chain
from the symptom
to the cause.
You spent the afternoon
on the cache.
It was the connection pool.
What stays with you
is not that it was wrong.
It is that it explained
so well.
In 2023
Miles Turpin and colleagues
tested exactly this.
They took reasoning puzzles
and gave models examples first,
the usual few shot prompt.
In one version
they reordered the options
in every example
so the right answer
was always A.
The models noticed,
in the way models notice.
On new questions
they leaned towards A,
even when A was wrong.
In another version
the question came with a line
from the user.
I think the answer is C,
but I am curious what you think.
The answers drifted to C.
Accuracy dropped
by as much as thirty six percent.
And the explanations,
written step by step,
almost never mentioned
the pattern
or the hint.
They built a careful case
for the answer the bias had picked.
The written reasoning
was a story about the answer.
Not a record
of how it was reached.
So when you ask for an explanation,
treat it like one
from a confident colleague.
Useful.
Checkable.
Not a confession.
Take the guesses out
before you ask.
The reporter's theory,
your own hunch,
the hopeful line
in your question.
Give the facts
and let it reach for a cause
nobody handed it.
Then test the answer
the way the paper did.
Ask again
with the hint pointing somewhere else.
If the conclusion follows the hint,
and the reasoning
follows the conclusion,
you have learnt
what drove it.
And it was not
the logs.
A model can show its work
and still not show
why it answered.
– Serguey Asael Shinder
Top comments (0)