You cannot judge an answer
about something you do not know.
That is the whole trouble
with using a model
on unfamiliar code.
It explains the legacy module,
the explanation is fluent,
and you have no way
to tell fluent from correct.
So calibrate it on known ground.
Take a file you wrote yourself.
Something you know cold.
Something with a subtlety in it
that took you two days
to get right.
Ask the model what it does.
Ask where it breaks.
Ask what the edge cases are.
Then read the answer
as a test of the model,
not a test of the code.
You will learn three things,
and all of them are useful later.
You will learn what it misses.
Usually not the logic.
Usually the reason.
It will describe the retry loop
and not notice
that the retry exists
because one downstream service
lies about its status code.
You will learn how it sounds
when it is wrong.
This is the valuable part.
Most people assume
a wrong answer will feel different.
It does not.
The sentence that is subtly false
arrives at exactly the same volume
as the sentence that is true,
with the same tidy structure,
in the same calm voice.
Seeing that once,
on code you can check,
is worth more
than any warning you have read.
And you will learn
how much context it needed.
If it got your own file wrong
until you pasted the caller,
then it will get
the unfamiliar file wrong too,
and you will not notice,
because there is no caller
you know to paste.
Do this again
when the model changes.
Do it again
when you move to a codebase
with different conventions.
It costs ten minutes.
You are not testing
whether the tool is good.
You are measuring
how wrong it tends to be
in your particular corner
of the world,
so that you know
how hard to check.
– Serguey Asael Shinder
Top comments (0)