Most AI directories fail the trust test in the first five seconds: the #1 tool looks purchased. We're the ToolPotion Media Team. We run ToolPotion — a free directory with 153 category boards and editorial scoring we call ToolRank. Full disclosure: this is our product. What follows is the ranking and taxonomy machinery we'd want another builder to steal.
If you're building a directory, a comparison site, or just trying to trust someone else's leaderboard, this is the part that matters.
Trust is a product requirement, not a blog tone
Users assume pay-to-play until proven otherwise. Our rule: placement cannot rewrite ToolRank. Optional featured spots exist for makers who want visibility; they're disclosed and sit beside — not inside — the score. Category pages say that out loud. Opacity looks like corruption even when you're honest.
As of the November 2026 index we publish boards such as AI video generators (578 measured) and AI agent builders (687 measured). Numbers move. The auditability of the method shouldn't.
Taxonomy before scraping
Source: Types of AI Tools Explained
Early directories treat every listing as the same object. An app you sign up for, an open model you call via API, a Discord, and a newsletter are different purchases with different failure modes.
We use 19 types across four families: things you use (apps, agents, platforms…), things you build with (models, frameworks, repos…), things you learn from, and places you go. Settle the family question before you write the scraper. Evaluating a framework when the user needed an app wastes weeks — for them and for your support inbox.
Practical schema minimum per listing: type + category + at least one task + one audience. Product-name SEO still matters; job-based navigation is what turns a CSV into a decision tool. We also index by job role for that reason.
Ranking signals people can argue with
Source: AI Model Leaderboards Explained
Leaderboards mislead when the method is invisible. We wrote up how arena-style ratings work — and where they break — because ToolRank isn't "vibes." Whether you use open-web traction signals, human preference arenas, or vendor benchmarks, publish the recipe. Readers should be able to disagree with your weights without guessing them.
Comparisons are load-bearing infrastructure
A wall of cards isn't enough. Side-by-side comparisons and explainers reduce bounce and support tickets. They also create URLs communities will actually link.
Rules we enforce on ourselves:
- Source from vendor docs and dated plan pages
- If a number is fuzzy the day we check, omit it
- Never invent neat table cells for SEO
Fabricated benchmarks destroy the one asset a directory has: trust.
Source: AI Agent Platform Buyer's Guide 2026
Freshness ops beat a bigger CSV
AI tools rename, reprice, and die. Plan for broken-link detection (parked domains → unpublish), category retags when products pivot, duplicate detection across domains and GitHub mirrors, and human review for edge cases.
Automation drafts. Humans decide ambiguous cases — including the "AI agent" that's really a thin chat wrapper. An honest app page beats a misleading agent badge.
Submission UX is part of ranking quality
Garbage in, ToolRank out. Makers need a free submission path that's honest about review: sign in, name/URL/type/category/description, we review, dedicated page ships. Updates via contact. Promotion is optional, priced separately, and does not rewrite scores.
What makers hate: silent black holes, surprise fees after "free," misfiled categories. What they'll promote for you: a clean page, correct shelf, and translations. We publish in 11 languages — expensive, useful outside English-only query streams.
Browse the live category index if you want to see the shelves as users see them.
Metrics that aren't vanity
Forget total listings as a north star. Prefer:
- % of searches that end in an outbound click to a vendor
- Time-to-first-honest-shortlist for a stated job
- Correction rate (how often we fix type/category after publish)
- Maker return rate to update their own page
If those are healthy, the CSV size is a side effect.
If you're building your own board
- Write taxonomy before ingestion
- Publish ranking method where users can find it
- Separate paid placement from score
- Require task + audience on every listing
- Budget freshness ops from day one
- Ship comparisons that cite sources, not vibes
That's the playbook. Not "scrape Product Hunt and add ads."
— ToolPotion Media Team
Top comments (0)