Three decisions in apps/ai-tools/src/etl/compare.ts that I made primarily to control cost, and that turned out to improve the pages:
1. Group by pipeline_tag before pairing anything
The straightforward approach to generating model comparison pages is to pair every model with every other model. At 1,664 models in the AI Tools directory, that's ~1.38 million pairs. Obviously wrong for budget reasons. Less obviously wrong: most of those pairs would produce useless pages. Comparing a text-generation model to an object-detection model in a "which should I choose" framing assumes the user is choosing between use cases, not between implementations. That's not what anyone searching for a model comparison is doing.
Grouping by pipeline_tag before pairing constrains pairings to models competing for the same job. text-generation models are compared to other text-generation models. image-classification models are compared within that group. The pages that result are actually answerable: "here are two models that both do X, here's how they differ." The pairwise comparison pattern I described earlier works because the comparison is scoped to a real substitution decision.
The implementation is a Map<pipeline_tag, models[]> built from the full dataset, followed by nested pairing within each group:
const byPipe = new Map<string, typeof models>();
for (const m of models) {
if (!m.pipeline_tag) continue;
const arr = byPipe.get(m.pipeline_tag) ?? [];
arr.push(m);
byPipe.set(m.pipeline_tag, arr);
}
Models without a pipeline_tag are excluded. In the HuggingFace API response, pipeline_tag is null for a significant fraction of entries — typically older or unusual models without a canonical task category. Comparing null-pipeline models to each other or to categorized models generates pages with no useful scoping, so excluding them is the right call.
2. Top-4 per pipeline by downloads
Even within a pipeline, all-vs-all pairing at scale is wasteful. The text-generation pipeline alone has hundreds of models. A comparison between two low-download models nobody uses produces a page nobody searches for.
The cut: sort each pipeline's models by downloads descending, take slice(0, Math.min(4, length)), then pair within that group. Top-4 means at most 6 pairs per pipeline (4 choose 2). This is deterministic — given the same dataset, it always generates the same pair list.
Downloads as a proxy for relevance has limits. A newly released model from a major lab might not have high download counts yet but would produce valuable comparison pages. I'm missing those for now. The alternative — using likes, trending signals, or recency — adds complexity and data-fetching surface. Downloads are available on every model record from the existing ETL, so there's no extra call needed.
The comparison content itself is generated with Claude Haiku via the same JSON extraction pattern used in the other ETLs: structured output, regex fallback, model_used tracking. Fallback comparison content is generic ("both are X models, see individual pages for specifics") but still publishable. The goal is to never have a missing page for a pair that exists in the database.
3. COMPARE_LIMIT env var as the budget dial
Even with pipeline grouping and top-4 slicing, the total pair count across all pipelines can be high. A CI run that generates 200 comparisons at 1,024 tokens each costs real money. The COMPARE_LIMIT env var caps the total:
const MAX = Number(process.env.COMPARE_LIMIT ?? 50);
const chosen = pairs.slice(0, MAX);
This means the ETL is idempotent across runs. New pairs that already exist in the database are skipped first (SELECT 1 FROM model_compare WHERE pair_slug = ?). The cap applies to how many new pairs are processed in a single run, not to the total in the database. Running the same ETL 4 times with COMPARE_LIMIT=50 will fill in 200 pairs as long as there are that many new ones pending.
The default of 50 matches the nightly CI budget for this project. A manually triggered run with COMPARE_LIMIT=200 can fill gaps faster when I've added a new pipeline category or refreshed the model list.
What this enforces without intending to: the ordering within pairs.slice(0, MAX) determines which comparisons get generated first. Since pairs are assembled pipeline by pipeline in insertion order of byPipe, the first 50 pairs will tend to cluster in the pipelines that appear earliest in the dataset. That means some pipelines are overrepresented in early runs. I haven't addressed this — a shuffle or round-robin across pipelines would distribute more evenly — but it also means the most-downloaded models within the first few pipelines get their comparisons first, which is probably the right priority order anyway.
The pattern holds more broadly: budget constraints that force explicit prioritization often improve the output selection, because the unconstrained version (all-vs-all, unlimited generation) produces both more noise and more API cost simultaneously.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)