The Highest-Scoring Duplicate Pair in My Corpus Is Not the Same Story, So No Threshold Reaches Zero

Every feed reader says "deduplicated". This one says how wrong it was, because the dedup was measured against labelled input rather than eyeballed - and the calibration line prints under the digest.

Repo: https://github.com/dev48v/ai-daily-digest - PUBLIC, MIT, 44 tests, standard library only, no API key, no network.
Running in your browser: https://dev48.infy.uk/agentlab/vol2-02-ai-daily-digest.html

The pipeline is four lines, and the whole problem is in one of them:

items   = fetch(sources)
stories = cluster(items, threshold)     # <- here
ranked  = rank(stories, weight)
digest  = ranked[:8]

Grouping by title finds none of it

The corpus is 32 labelled items over 8 sources, 16 real stories, 22 genuine duplicate pairs. What most readers do - seen.add(url), set(titles) - ships as a control, measured rather than dismissed:

def test_exact_titles_finds_none_of_the_real_duplicates():
    result = evaluate(exact_titles(ITEMS), STORIES)
    assert result.true_merges == 0        # zero. of twenty-two.

So it needs shingles, a similarity and a threshold, swept at all 101 thresholds x 4 similarities x 2 linkages, with both mistakes on the table: two stories collapsed into one, and the same story printed twice.

threshold	clusters	false merges	missed	errors
0.00	1	474	0	474
0.04	18	6	9	15
0.05	19	5	9	14
0.10	21	5	13	18
0.24	30	2	22	24

No threshold reaches zero

The best row still gets 14 pairs wrong, and that is a property of the input rather than of the tuning. The highest-scoring pair of all 496 is not the same story:

mw-nvda / mw-amd      0.867   Nvidia quarterly results top forecasts on AI chip demand
                              AMD quarterly results top forecasts on AI chip demand
best real duplicate   0.231

Earnings desks write from templates, so two different companies get one sentence with the ticker swapped. Meanwhile one funding round arrives as "OpenAI closes $40B round at a $300B valuation" and "SoftBank leads record 40 billion dollar investment in OpenAI", which after normalisation share exactly one token and it is the company name. Any threshold low enough to catch the real duplicates merges the template pair; any threshold high enough to avoid it merges nothing.

Two more fell out of measuring instead of assuming: bigram shingles look like a refinement and collapse recall from 0.59 to 0.18, and SimHash - the standard recommendation, implemented in full over 64 bits - never beats merging nothing, because ten tokens of headline do not move 64 bits far enough to separate anything.

The controls, in both directions

exact_titles() is the one that fails and oracle(items, labels) is handed the answer key, so "the clusterer is honest" is a claim with something to compare against. And the honest pipeline cannot cheat structurally, because the label is not a field on the item:

assert "story" not in Item.__dataclass_fields__
assert "labels" not in inspect.signature(cluster).parameters
assert "labels" in inspect.signature(oracle).parameters

Same role peeking() plays in Vol 2 #01. Ranking is held together the same way: named contributions summing to the total within 1e-12, each item's reason generated from the argmax rather than written beside it. The moment the parts stop adding up, the explanation is decoration.

And the section I meant to write

I intended to claim that folding recency and importance into one weight makes one axis cosmetic. The sweep says otherwise: 82% of the range gives a top five that is neither the importance ordering nor the recency ordering, so the blend is real.

What collapses instead is the reachable set. 201 dial positions produce 19 distinct top-fives and 3 possible winners, and 14 of the 19 stories cannot come first at any setting - a story both older and less important than another can never outrank it, so only the Pareto frontier is in the running. The dial is not fake; it has far fewer positions than it looks like, and tuning it to surface one of those 14 is time spent on something that cannot happen.

Agent Lab Vol 2, Project 02. Project 01 was a screenshot-to-code loop that knows when its own feedback signal is lying to it; this one is a pipeline that knows how often it is wrong and says so: https://dev48.infy.uk/agentlab.php