DEV Community

Nerav Doshi
Nerav Doshi

Posted on Edited on Originally published at pipelineandprompts.com

Swept the Per-Article Cap From 1 to 3: The Gains Run Out Before the Costs Do

Entry 18 compared no cap with a cap of one chunk per article, and ended by saying the middle was untested. Same twelve questions this time, at caps of none, 1, 2 and 3. The labels are unchanged from Entry 18, so this run doesn't settle whether a question can have more than one acceptable article. That still needs someone to read the articles.

The first run failed before it started:

ModuleNotFoundError: No module named 'chromadb'
Enter fullscreen mode Exit fullscreen mode

The python3 on the PATH no longer had the package, and I didn't dig into why. Pointing the script at the venv from Entry 08, which already had it, got past that:

~/mcp-env/bin/python3 eval_cap_sweep.py
Enter fullscreen mode Exit fullscreen mode

The first thing worth checking was whether the setup still matched Entry 18. It did. The no-cap row and the cap-1 row reproduce Entry 18's numbers exactly, and the collection is still 741 chunks, so the new rows are comparable to the old ones.

#     none      1      2      3   |    none      1      2      3
1        1      1      1      1   |       5      1      2      3
2        1      1      1      1   |       4      1      2      3
3        1      1      1      1   |       4      1      2      3
4        1      1      1      1   |       5      1      2      3
5        1      1      1      1   |       2      1      2      2
6        1      1      1      1   |       3      1      2      3
7        8      5      6      7   |       0      1      0      0
8        1      1      1      1   |       4      1      2      3
9        1      1      1      1   |       5      1      2      3
10       1      1      1      1   |       3      1      2      3
11       1      1      1      1   |       3      1      2      3
12       7      3      5      6   |       0      1      1      0
Enter fullscreen mode Exit fullscreen mode

The left half is where the labeled article lands in the list at each cap. The right half is how many of the top five chunks come from it. Ten questions put the right article first at every setting. Everything that moves comes from question 7, the AI agent one, and question 12, the managed versus self-hosted one.

Cap Right article in top 1 Top 3 Top 5 Chunks from it in top 5 (avg) Different articles in top 5 (avg)
none 0.83 0.83 0.83 3.17 2.00
1 0.83 0.92 1.00 1.00 5.00
2 0.83 0.83 0.92 1.75 3.42
3 0.83 0.83 0.83 2.42 2.75

Cap 1 pulls both misses into the top five, and the managed question into the top three. Cap 2 gets the managed question to fifth place and leaves the agent question sixth, so it rescues one of the two. At cap 3 neither makes the top five, and the hit rates match having no cap at all. Meanwhile the depth cost keeps coming. Cap 3 hands back 2.42 chunks from the right article instead of 3.17, which is about three quarters of the depth, in exchange for no change in hit rates on these questions. By that point the cap is only trading depth for variety.

Variety is the other thing the table shows. With no cap, the top five chunks come from only two different articles on average. A cap of one forces five. The settings in between land where you'd expect.

No setting fixed the misses. Neither question reached first place at any cap. The cap only decides which articles share the list, and it does nothing about why the managed question lost in the first place, which Entry 18 traced to a winning chunk that matched the question's wording rather than its intent.

Entry 18 also left a worry open: that the series' own entries about MCP, n8n and RAG were crowding slots two through five. They weren't. Across the twelve questions that have real answers, no top five contained a chunk from one of them without a cap. One appeared under cap 1, in one question, and I haven't checked which. The entries that won the controls in Entry 18 didn't show up for questions with real answers.

This is a small set. Twelve questions means one question moves a score by eight points, two of them drive every difference above, and one of those two, the agent question, may not be a miss at all. Cap 2's single gain is one question moving from seventh place to fifth, which isn't enough to call it better than no cap.

So the cap works as a dial between depth and variety, and in this data it pays off only at the tight end and only for questions that were already missing. If a person is reading five sources, cap 1 was the only setting that did much. If a model is reading five chunks to write an answer, cap 1 hands it a single chunk from the right article. The thing none of these settings changes is the ranking itself, and the next test is probably there: re-scoring the top twenty before filtering. A 1B model as a re-ranker is a risky bet after Entry 13, so I'd treat it as an experiment, not an expectation.

Top comments (0)