Entry 20 ended with two right answers demoted by ties. gemma3:12b gave the labeled article and a rival the same score out of 10, and my tie-break fell back to embedding order, which put the rival first. It also left a cost question open, because the script didn't time anything. This entry tests two ways to break the ties and times every scoring call.
I decided the three variants before running anything:
- A: Entry 20's setup unchanged, 0 to 10 with ties kept in embedding order. If it didn't reproduce Entry 20's numbers, that would be the story instead.
- B: the same prompt, asking for 0 to 100.
- C: the same integer scores as A, but ties broken by the model's own uncertainty. Ollama can return the probability of each possible next token, so for every reply I took the probabilities the model gave the digits 0 to 9 at the score position and computed an expected score. A chunk scored 9 with most of the remaining probability on 8 ranks below one scored 9 with the rest on 10. C can only reorder chunks that already share a score.
Same twelve questions and strict labels as Entries 18 to 20, top 20 chunks per question, temperature 0, two runs each.
~/mcp-env/bin/python3 eval_rerank_v2.py
A reproduced Entry 20 exactly: the same four questions moved, to the same positions, and the summary row matched to two decimals. The collection was still 741 chunks. So the new rows are comparable to the old ones.
The same eight questions had the right article first in every column, so here are the four that moved:
| Question | Embedding only | A, 0-10 | B, 0-100 | C, 0-10 + tie-break |
|---|---|---|---|---|
| 6, switch LLM providers | 1 | 2 | 2 | 1 |
| 7, AI agent reading the cluster | 8 | 5 | 2 | 3 |
| 10, incident triage with a workflow tool | 1 | 2 | 1 | 1 |
| 12, managed or self-hosted | 7 | 1 | 1 | 1 |
| Setting | Top 1 | Top 3 | Top 5 | MRR | Chunks from right article in top 5 |
|---|---|---|---|---|---|
| embedding only | 0.83 | 0.83 | 0.83 | 0.86 | 3.17 |
| A, 0-10 | 0.75 | 0.92 | 1.00 | 0.85 | 3.58 |
| B, 0-100 | 0.83 | 1.00 | 1.00 | 0.92 | 3.75 |
| C, 0-10 + tie-break | 0.92 | 1.00 | 1.00 | 0.94 | 3.50 |
That table is with no cap. With a cap of one chunk per article, every setting's top-5 hit rate stays the same or rises to 1.00 and the chunk count drops to 1.00, as in Entries 19 and 20, so the cap still adds nothing once the re-ranker is in place.
The finer scale didn't remove the ties, because the model doesn't use it. Across 240 replies on the 0 to 100 scale, gemma3:12b used only ten values: 0, 10, 20, 30, 40, 60, 70, 75, 85 and 90. It never used 50, 80 or 100, or anything that wasn't a multiple of five. On the 0 to 10 scale it used nine values and never 5 or 10. A 101-point scale with ten values in use is no finer than the 11-point one. Question 6 shows it: three chunks scored 90, the rival first, exactly as they'd tied at 9 before. B still won back question 10 and moved question 7 to second, so the wider scale changed some judgments, but not by spreading them out. On average 1.92 chunks shared the top score under B, against 2.75 under A, and in four questions the tied chunks came from more than one article (five under A).
C used information the integer threw away. For question 6, the four chunks tied at 9 had expected scores of 8.96, 8.82, 8.73 and 7.10, and the highest belonged to the labeled article. That fixed question 6 and question 10 without changing a single score, and C ended with the right article first on 11 of 12 questions. The one miss is question 7, the agent question that has never come first in any setup in this series. The 12B scored an n8n workflow chunk 8 there, ahead of everything else, with no tie involved.
Then the cost. To time it cleanly I ran a separate single pass per model: the plain 0-10 prompt, no probabilities requested, twenty chunks per question.
| Scorer | Seconds per question (20 calls) | Range | All 12 questions |
|---|---|---|---|
llama3.2:1b |
4.4 | 4.2 to 4.6 | 52 s |
llama3.2:3b |
13.6 | 12.4 to 15.6 | 163 s |
gemma3:12b |
94.8 | 69.8 to 136.9 | 19 minutes |
The only scorer that helped in Entry 20, and the one that makes the tie-break work here, costs about a minute and a half per question on this Mac mini (Apple M4, 16 GB). The 1B is twenty times faster, and Entry 20 showed it barely changes anything. That's not a price for an interactive search box. It might be fine for a batch job or a nightly report.
Some limits on all of this. Twelve questions means one question is eight points: C's 0.92 against B's 0.83 is question 6 and nothing else. C's tie-break comes from the same model's judgment, so it can't fix a confident wrong score like question 7's, only ties. The timings come from one pass, and the variant runs were slightly faster per call because the second run of each question repeated identical prompts, which Ollama partly caches. Other light work ran on the machine at the same time, including a few container commands near the end, which may explain some of the 12B's spread. The embedding lookup itself wasn't timed; it's small next to twenty model calls. And the labels are still strict and mine.
So the ties were mostly a tie-break problem, not a scale problem. Asking for a finer scale got round numbers back. Reading the model's own uncertainty, already available in the same call, closed both of Entry 20's demotions and left one miss. The open questions are whether this holds on questions written after seeing these results, and what the agent question's label should be. That's the next entry.
Top comments (1)
The ten-values-out-of-101 count is the most useful line here. It's a cheap check anyone can run on their own judge before trusting a wider scale.
One caution on reading the summary table: with 12 questions, each question is 0.083 of top-1, so C beating B (0.92 vs 0.83) is exactly question 6, and B beating A on MRR comes down to questions 10 (2 to 1) and 7 (5 to 2). Both moves look real given how C works, but a second set of 12 could flip the order of B and C. Since the expected-score tie-break can only reorder tied chunks, counting how many questions had a tie at the top at all would show how much room it ever has to help.