The team fine tuned a model
on the product catalogue.
Thousands of lines
like this one.
The ZR-7 is our only fully waterproof walking boot.
Ask it about the ZR-7
and it knew.
Waterproof.
Walking.
Boot.
The sizes,
even.
Then a customer asked
which of your boots is waterproof.
It named a different one.
Confidently.
The fact was in the training data,
hundreds of times.
Just pointing the other way.
In 2023
researchers led by Lukas Berglund
gave this a name.
The reversal curse.
They fine tuned models
on invented sentences
of the form A is B,
a made up person
is the director of a made up film.
Asked who directed the film,
the models did no better than chance.
They found it in the big models too,
with real people.
GPT-4 could say who Tom Cruise's mother is
about four times in five.
Given her name
and asked who her son is,
about one time in three.
Training on a sentence
teaches the model to continue it
in the order it was written.
It does not, by itself,
store a fact you can enter
from either side.
People do that without noticing.
Models, mostly, do not.
The odd part
is that inside the prompt
it is fine.
Put the sentence in the context,
ask the reverse question,
and the model answers it,
because now it is reading,
not remembering.
So think about the direction
of every question your users will ask.
Customers rarely start with the code.
They start with the need,
the size,
the symptom.
If you fine tune,
write the facts both ways,
and in the ways people actually ask.
Better,
keep the facts in retrieval,
where the model reads them
at the moment it answers,
and use fine tuning
for tone and format.
And test in the direction
your users come from.
If every test question
starts with the product name,
you have only checked
the half of the fact
you already knew it had.
– Serguey Asael Shinder
Top comments (0)