DEV Community

Serguey Asael Shinder
Serguey Asael Shinder

Posted on

You Embedded Questions and Documents the Same Way and Your Model Expected Them Labelled

The search over the help centre
was built in a week.

An open embedding model.

Every article,
split and embedded.

Every customer question,
embedded the same way.

Nearest neighbours.

Done.

It worked.

Mostly.

Short questions did badly.

"refund card"
returned other short things.

Titles.

Headings.

Fragments that looked
like the question,
not the answer to it.

The team tried bigger chunks,
smaller chunks,
a reranker.

Nobody had read
the model card to the end.

The model was from the E5 family.

Its card says
every input should start with
"query: "
or
"passage: ".

And then, in the FAQ,
plainly:
yes, you need the prefix,
or you will see
a performance drop.

Here is the mechanism.

Some embedding models
are trained asymmetric.

During training,
questions carried one label
and documents carried another.

The model learned
two slightly different jobs.

For a query:
what is this person looking for.

For a passage:
what does this text contain.

Leave the labels off
and both sides
get embedded as the same kind of thing.

Then a short question
sits nearest
to other short text,
because that is what
it most resembles.

Similar wording wins.

The answer loses.

And nothing errors.

The scores still come back
between zero and one,
the top result still looks plausible,
and the dashboard stays green.

The fix was two strings:

query: refund card
passage: To refund a card payment...

Then re-embed every document,
because the stored vectors
were made without the label.

E5 is not alone.

Nomic's text model
wants "search_query: "
and "search_document: ".

BGE suggests
an instruction
in front of short queries.

Others want none at all.

So the prefix
is part of the model's interface,
like an argument
a function cannot work without.

Your code will not fail
when you forget it.

It will just return
something close
to the right answer,
every day,
forever.

Read the model card
to the last line.

Then write a test
with ten real questions
and the article each one should find.

– Serguey Asael Shinder

Top comments (0)