I built a tier list that re-rates 245+ AI tools every week — the automation behind it

이서형 — Sun, 09 Aug 2026 09:23:28 +0000

AI tool reviews rot faster than anyone can rewrite them. A tool that was S-tier in March ships a broken pricing change in June, a "top 10" listicle from last year recommends products that no longer exist, and every directory slowly turns into a graveyard of dead links.

I run AI Tier List, a bilingual (EN/KO) directory that ranks 245+ AI tools from S to D. My answer to review rot: don't re-review by hand. Make a pipeline re-rate everything weekly, and let humans only approve or reject.

The architecture

Everything runs on one weekly GitHub Actions cron (Next.js 16 + Prisma + Neon Postgres + Vercel):

weekly cron (Sun 00:00 UTC)
  ├─ collect Google Trends per tool     → trend scores
  ├─ collect OpenRouter usage rankings  → weekly LLM leaderboard
  ├─ deactivate dead tools              → site checks + trend slump
  ├─ discover new tools                 → search + AI triage
  ├─ re-evaluate tiers (LLM)            → PendingUpdate rows
  └─ generate weekly blog draft         → MDX

The key design decision: the LLM never writes directly to the live site. Re-evaluations land in an approval queue (PendingUpdate table). I review diffs in an admin panel and approve batches. The pipeline proposes; a human disposes. That one boundary is what keeps automated content from becoming automated garbage.

Two collectors do the heavy lifting:

Trend collector — Google Trends per tool, weekly. A tool in a sustained slump gets flagged; if its website also starts failing health checks, it gets deactivated automatically. Dead products remove themselves from the directory.
OpenRouter collector — real token-usage data powers a weekly LLM leaderboard. No opinions, just "which models did people actually route traffic to this week," with usage share, pricing, and context length.

What the tier actually means

Each tool stores bilingual tierReason, strengths, and weakness fields, and the tier maps S→5 … D→1 into review schema markup. When the weekly re-evaluation moves a tool, the reason is regenerated with it — so the rating and its justification never drift apart. Rating history is kept, so a tool page can show "was B in June, A since July."

Honest numbers (the unglamorous part)

Build-in-public means real numbers, so: ~1,500 URLs in the sitemap, about 130 pages currently earning Google impressions, roughly 90 search clicks in the last 28 days, and the domain started with zero backlinks. Impressions grew ~7x in the past month after fixing a sitemap discovery bug, but authority is clearly the bottleneck — programmatic comparison pages sit at position 60–70 while the data behind them is solid.

The most surprising lesson so far: infrastructure bugs cost more than content quality. One broken sitemap fetch kept half the site out of Google's index for weeks. No amount of content work would have mattered until that one HTTP request succeeded.

The question I keep hitting

Distribution for indie directories in 2026 is rough — most free submission queues have converted to $50–$350 paid placements. For those of you shipping side projects: what free distribution channels still actually work for you?

Happy to answer anything about the pipeline, the approval-queue pattern, or the SEO trenches.

DEV Community: 이서형

I built a tier list that re-rates 245+ AI tools every week — the automation behind it

The architecture

What the tier actually means

Honest numbers (the unglamorous part)

The question I keep hitting