My highest-value data category was starved because of how I ranked ingest targets

I run a pipeline that mines public complaints and scores them for commercial intent. Last week I pulled approval rates by category and found something that had been quietly costing me the best data I had.

Here is the table that started it:

category                approved/total   rate   mean intent
HR / Recruiting              28/  38      74%      80.1
Data / Analytics            177/ 242      73%      73.9
Dev Tools / SaaS Infra      560/1001      56%      71.9
Finance / Accounting         86/ 206      42%      75.6

HR and recruiting has the best approval rate and the highest mean commercial intent of any category with real volume. It also has the smallest candidate pool by an order of magnitude: 38 against dev tools' 1,001.

My first assumption was that the classifier or the editorial gate was rejecting HR items. It was not. 74% of them get approved, the highest rate in the corpus. They were simply never being fetched.

The actual cause

The ingest walks a list of targets per source, and a per-source time budget cuts the tail. So target ordering decides what the corpus is made of.

The ordering was: protect the proven high-priority targets, then rotate everything else by staleness. That is a reasonable design — it stops unproven targets displacing measured ones at random, which an earlier jitter-based approach did (the newly-reached targets measured 65% in scope against 80% for the established ones).

But there were 9 protected targets against roughly 13 reachable slots per run. So about 4 rotating slots, against 105 never-fetched targets on that source. r/humanresources and r/recruiting were sitting in a queue roughly 26 weeks deep.

11 of the 13 HR targets on active sources had never been fetched once.

Why the fix is priority 8 and not 9

The obvious move is promoting them into the protected band. That is wrong: the protected band was already 9 of ~13 slots, so adding three more would consume the rotation entirely and recreate the same starvation for everything else.

The rotating pool sorts by staleness ascending, then priority descending. Every never-fetched target ties at staleness 0 — so priority is the tiebreak among them. Setting the HR targets to 8, one below the protected threshold, moves them to the front of the unfetched queue without displacing the protected head or the rotating tail.

UPDATE source_targets t
SET priority = 8
FROM sources s
WHERE t.source_id = s.id
  AND s.status = 'active'
  AND t.is_active = true
  AND t.last_fetched_at IS NULL
  AND t.priority < 8
  AND ( ... explicit target list ... )

Guarded on last_fetched_at IS NULL and priority < 8 so it is idempotent, and scoped to an explicit list of target values so it cannot touch anything else. 11 rows.

The part that generalises

If your pipeline has a budget that cuts a tail, the ordering of that queue silently decides what your data is about. Mine had produced a corpus that was 44% developer tools, and I had been reading that as a finding about where the opportunities are. It was not a finding. It was an artifact of which targets the budget happened to reach.

Two things worth checking in your own pipeline:

1. What fraction of your configured sources have never run? Mine was 79%. Some of that was deliberate (a source disabled because its content was a decade old), but most of it was queue depth.
2. Is your highest-yield segment also your smallest? That combination usually means starvation, not a real ceiling.

The corpus is at ddmarketer.com if you want to see what came out of it, and there is a free MCP server so your agent can query it directly — claude mcp add --transport http ddmarketer https://www.ddmarketer.com/api/mcp. But the pipeline lesson is the transferable part.