A new starter asked the internal assistant
how refunds work.
It gave a clear answer,
in three tidy steps,
describing a process
we stopped using in 2023.
She followed it.
Why would she not.
The page it was reading
had been deleted eighteen months earlier.
Deleted from the wiki.
Not from the index,
because the index was built in March
from an export taken in March,
and nobody has ever thought of
rebuilding a search index
as part of retiring a document.
This is the part people miss
when they put a model
on top of their own documents.
You have not given it your knowledge.
You have given it a photograph
of your knowledge,
taken on a particular day,
and the photograph does not age
the way you assume it does.
Retrieval ranks by resemblance.
Not by date.
Not by authority.
Not by whether anybody
still believes the thing.
The obsolete page usually wins,
because obsolete pages
were written when the process was new
and people explain things carefully
when they are new.
Today's version lives in half a thread
and somebody's head.
And the model cannot help you here.
To it, the retired policy
and the current one
are the same kind of object.
Text, confidently phrased,
sitting in a folder you pointed it at.
It has no way to know
which of them survived a meeting.
So treat the corpus as production.
The index is a build artefact.
It is rebuilt when the source changes,
and a delete means a delete,
verified, not assumed.
Every document carries an owner
and a date it was last confirmed,
and anything past its date
is either confirmed again
or removed from the index,
which is a decision somebody makes
in ten seconds
rather than a slow rot
nobody is watching.
Show the source and its date
in every answer,
because a human will notice 2019
where a model will not.
And keep a handful of questions
whose correct answer changed.
Ask them after every rebuild.
An assistant that is wrong
sounds exactly like one that is right.
The only difference is what you fed it,
and you are the only one
who can check that.
– Serguey Asael Shinder
Top comments (0)