DEV Community

Nerav Doshi
Nerav Doshi

Posted on Edited on Originally published at pipelineandprompts.com

Re-Ranking the Top 20 With Local Models: Only the 12B Fixed the Miss, and It Demoted Two Right Answers

Entries 17 and 19 filtered the ranking the embedding search produced. The cap decides which chunks to hide, and it can't change why the managed-versus-self-hosted question lost in the first place. This entry changes the ranking itself: take the top 20 chunks for each question, ask a local model to rate each one from 0 to 10 for how well it answers the question, re-sort by that rating, then run the same cap sweep on the new order. Ties keep their original embedding order, and chunks 21 and beyond stay below.

I ran three scorers: llama3.2:1b, llama3.2:3b and gemma3:12b, each twice per question at temperature 0. That's 12 questions × 20 chunks × 2 runs, or 480 scoring calls per model. The labels are the same strict ones from Entry 18. A question can now list a second acceptable article, but I haven't decided which ones, so none were added. That matters for what follows.

~/mcp-env/bin/python3 eval_rerank.py
Enter fullscreen mode Exit fullscreen mode

Eight of the twelve questions had the right article first in every column, so only the four that moved are worth showing:

Question Embedding only 1B 3B 12B
6, switch LLM providers 1 1 2 2
7, AI agent reading the cluster 8 5 9 5
10, incident triage with a workflow tool 1 1 1 2
12, managed or self-hosted 7 6 7 1
Setting Cap Top 1 Top 3 Top 5 Chunks from right article in top 5
embedding only none 0.83 0.83 0.83 3.17
embedding only 1 0.83 0.92 1.00 1.00
1B none 0.83 0.83 0.92 3.33
1B 1 0.83 0.92 1.00 1.00
3B none 0.75 0.83 0.83 3.50
3B 1 0.75 0.92 0.92 0.92
12B none 0.75 0.92 1.00 3.58
12B 1 0.75 0.92 1.00 1.00

The 12B repaired the clear miss. For the managed-or-self-hosted question it rated a chunk of the managed-vs-self-hosted article an 8, ahead of the Kubernetes-at-scale explainer at 7, and moved the article from seventh to first. That's a one-point margin on a ten-point scale, so I wouldn't lean on it. With no cap it also put all twelve right articles in the top five while handing back 3.58 chunks from the right article on average. None of Entry 19's cap settings reached 1.00 in the top five and kept more than one chunk of depth, so on these twelve questions this is the first setup that did both.

It also lost ground at the top. Two questions the embedding search got right fell to second place, which is why top-1 drops from 0.83 to 0.75. The printed top three for question 6 shows why: the winner, a prompt-versioning chunk, scored 9, and two chunks of the labeled article scored 9 as well. Ordering among tied scores comes from my tie-break by embedding rank, so the scorer didn't prefer the other article. My tie-break put it first. Question 10 follows the same pattern at 7. A 0-to-10 integer scale produces ties at the top of a list like this constantly, so part of the damage is the scale and the tie-break, not the 12B's judgment. I haven't tested a finer scale or a smarter tie-break.

The 3B scored below the no-re-ranking baseline on top-1 and on mean reciprocal rank (0.81 against 0.86), even though its depth went up. It lost question 6 the same way, pushed the agent question from eighth to ninth, and left the managed question at seventh, with Kubernetes-at-scale chunks rated 8 at the top of its list. That's the same article the embedding search picked, so it repeated that mistake where the 12B didn't. The 1B barely changed anything: top-1 unchanged, top-5 up by one question because the agent question moved from eighth to fifth. Its top picks for the two misses were rated 2 out of 10, four of its 480 replies weren't numbers, and on one question it gave all twenty chunks the same score, so I wouldn't read that move as skill.

Entry 13 worried that a 1B model gives different answers to the same prompt. Here all three models produced identical top-five lists across both runs for all twelve questions. That doesn't clear the 1B in general, because Entry 13 used default settings and I set temperature to 0. It only shows that temperature 0 makes this setup repeatable.

Re-ranking added nothing once a cap was applied. The 12B at cap 1 matches the embedding-only cap-1 row on top-3 and top-5 and is worse on top-1. Its value is in the no-cap case, where it keeps the depth.

Some limits on all of this. Twelve questions means one question is eight points, so ten right versus nine right is a one-question difference. The labels are strict and mine, and "chunks from the right article" counts chunks without checking that each one is useful. And I haven't measured cost: every question now needs 20 calls to the scorer, and the 12B is an 8 GB model. The script doesn't time anything.

The first thing in this series that changed the ranking, not just the list, gave a mixed answer, and the answer depended heavily on the scorer's size. The 12B fixed the one clear miss and improved depth and top-five together, at the price of top-one accuracy through ties. The 3B scored below the baseline on top-1, and the 1B barely moved anything. The next questions are whether tie handling closes the two demotions, whether the 12B's advantage holds on questions outside these twelve, and whether 20 calls to an 8 GB model per query is a price worth paying.

Top comments (0)