I had three categories of document coming in, and a semantic classifier that
scored every one against all three. It worked. Except one of the three never
showed up near the top — not rarely, never.
The obvious explanations were that there were fewer of them, or that they
were simply worse matches. Both were wrong.
They matched just as well. They were losing on geometry.
How a whole lane can lose without being worse
The setup is the ordinary one. Each category is described in a paragraph;
that paragraph becomes a vector. Each document becomes another vector.
Cosine against all three, keep the highest: that's its lane, and that number
is its fit.
The bug is in the last step. Cosines get compared across lanes as if they
were the same unit. They are not.
A lane paragraph written in common vocabulary, heavily covered by the
model's training data, lands in a dense region of the space. Everything near
it scores high — 0.55, 0.60. A lane paragraph written in industry-specific
vocabulary lands somewhere sparser, where the same degree of real fit
produces 0.30 or 0.35. That gap isn't measuring fit. It's measuring how much
text the model has seen that talks like this.
So when you sort by raw cosine, the sparse lane loses every time. Not
usually — every time. Its best possible document scores below a dense lane's
mediocre one.
The fix is five lines
Don't compare across lanes. Normalise within each one.
Route every document to its lane by max cosine, same as before. Then, per
lane, take the mean and standard deviation of those cosines, and turn each
raw cosine into a z-score: how many deviations above its own lane's average
this document sits.
by_lane = {ln: [] for ln in lane_names}
for r in recs:
by_lane[r["lane"]].append(r["cos"])
lane_stats = {}
for ln, v in by_lane.items():
arr = np.array(v) if v else np.array([0.0])
lane_stats[ln] = {"mu": float(arr.mean()),
"sd": float(arr.std() + 1e-9),
"n": len(v)}
for r in recs:
s = lane_stats[r["lane"]]
z = (r["cos"] - s["mu"]) / s["sd"]
A z of 2.0 means the same thing in all three lanes: strong, for whatever
this is. The lane's absolute scale drops out, and with it the advantage it
never earned.
The epsilon on the denominator isn't superstition. A lane with one document
in it has zero deviation, and without it the z comes back infinite. That
lane existed.
Making it count
The z-score isn't the ranking. It's one signal feeding a larger score, and
how it feeds in is its own decision. Symbol names changed for privacy,
structure as it runs:
const cfg = rules.semantic; // { w_sem: 0.4, z_cap: 2.0 }
const map = rules._semantic && rules._semantic.map;
if (cfg && cfg.w_sem && map && item.id) {
const rec = map[item.id];
if (rec && Number.isFinite(rec.z)) {
const zCap = cfg.z_cap != null ? cfg.z_cap : 2.0;
const bonus = cfg.w_sem * Math.max(0, Math.min(rec.z, zCap));
if (bonus > 0) { b += bonus; trace.push(`sem:${rec.lane}`); }
}
}
Three things there are worth the space.
The clamp starts at zero, so a negative z never subtracts: the semantic
signal can only lift a document, never sink one. A weak semantic match can
mean "bad fit", but it can also mean "this text is written strangely", and I
didn't want the second one burying anything.
The upper cap is duller and more important. A lane with few documents has a
tiny standard deviation, and a tiny deviation produces enormous z-scores.
Without the cap, the emptiest lane wins the entire list — which is the
original bug again, running backwards.
And the whole guard fails open. No cache, no config, or an item with no
embedding, and the branch is skipped and the score comes out identical to
what it was before any of this existed. I was adding a signal to something I
already used daily. The one thing that couldn't happen was a missing file
quietly changing an ordering I'd already come to trust.
The general shape
None of this is about semantic search. It's arithmetic that applies the
moment a score ranks across groups that don't share a scale.
Grades from different teachers. Latency percentiles from services with
different load profiles. Product reviews in categories where people rate
with different harshness. Anything classified by a model that has seen far
more of one kind than another. The number looks comparable because it has
the same name and the same range, and it isn't.
The tell is specific: a whole category that never reaches the top. Poor
performance mixes in. Total absence is structural. When an entire group is
missing from your best-of list, the first thing to suspect isn't the data —
it's the comparison.
The fix is always the same. Normalise within the group, then compare across
groups. Cap the output, because small groups produce extreme values. And
decide deliberately whether the signal is allowed to subtract, or only to
add.
Limits
The mechanism is correct by construction: normalising within a group before
comparing across groups is arithmetic, not a bet. What I don't have is a
measured before-and-after. I didn't freeze a pre-change ranking to compare
against, so what I can claim is that the sparse lane stopped being invisible
— not how much the ordering improved in quality.
Recording the baseline is what I'd do differently. The change took an
afternoon; the measurement that would have proved it needed starting before
I touched anything.
One corpus, one embedding model, three lanes I defined by hand. The lanes
are prose paragraphs, so rewriting one moves its vector and shifts its mean
— the z-score is robust to lane size, not to me editing the description. I
haven't measured how much.
The slowest part wasn't the fix. It was seeing that the sparse lane wasn't
failing.
I'd spent weeks reading that list and assuming a third of the corpus simply
didn't match anything well. The data had been saying otherwise the whole
time. I was sorting by a number that didn't mean the same thing in every
row.
It's no accident that it took so long. A whole category missing reads like a
verdict, not a bug. And as long as it reads like a verdict, you don't go
looking.
Top comments (0)