A few weeks ago I started AIVerse, a directory of AI tools. The premise was simple: every "best AI tools" list I could find was either a paid placement farm or so stale that half the pricing was wrong. So I built the boring version instead — one that treats pricing and category data as something to be maintained, not written once.
This post is about the data side, because that is where the surprises were.
The shape of the data
Current state of the catalogue, pulled today rather than typed from memory:
| Pricing model | Tools | Share |
|---|---|---|
| Free | 2,198 | 50.8% |
| Freemium | 1,425 | 33.0% |
| Paid | 700 | 16.2% |
| Total | 4,323 | across 92 categories |
Category concentration is steeper than I expected. Four categories hold a quarter of everything:
- Video Generators — 294
- Image Generators — 269
- Productivity — 253
- Developer Tools — 247
The remaining 88 categories average about 37 tools each. 708 listings carry a 2026 publish date, which tells you how fast the supply side is still growing.
One number I do not want to hide: only 614 listings have a verification level recorded. "Free" is therefore a claim taken from the vendor page, not something I confirmed by signing up. That gap is exactly the difference between a directory and a database, and it is the thing I am slowly paying down.
The bug that actually mattered
While exporting every listing to check link health, I diffed the websiteUrl field against vendor domains and found 30 rows pointing not at the vendor but at an affiliate redirect:
capcut → capcutaffiliateprogram.pxf.io/9LPy7W
pxf.io is Impact's link domain; partnerlinks.io is another one. The tracking URLs carried a referring sub-ID (?s=...), which means every "Visit website" click on those 30 pages was crediting someone else's affiliate account. Worse, the descriptions on those same rows said things like "You can use CapCut online at capcutaffiliateprogram.pxf.io" — a templated sentence that turned out to be repeated across 2,084 pages of the site.
Three fixes, in order of how much they mattered:
- Reject tracking domains at every entry point. There is one shared list, imported by the crawler, the CSV importer and the public submission form. Previously each path had its own validation and they had drifted apart.
-
Resolve the 30 existing rows by following the redirect chain to the final vendor domain. Four needed manual judgment: one landed on a
/frlocale, one self-redirected in a loop, one dead-ended on the tracker, and one pointed at a completely different product page. - Strip the templated sentence at render time, so those pages stopped showing the tracker before the underlying data was cleaned.
How the pipeline works now
- Crawl the upstream catalogue.
- Reject tracking domains and anything that is not a vendor-owned URL.
- Strip boilerplate — four known template markers. Whatever is left under 25 words gets queued for a human instead of being rendered.
- Hand-write the top 150 pages by traffic, four facts each: free tier limit, price tiers, two pros, two cons, and who it is for.
- Store that hand-written copy in a git-tracked JSON overlay that takes precedence over the crawled description at render time.
Step 5 is the one I would recommend to anyone building the same thing. My first attempt wrote copy straight into the production SQLite database through an admin form. That sounds fine until you notice the deploy hook restores the live database from a backup on every push — so the copy existed only on the server that was about to be overwritten. Content in git gets a diff, a review and a rollback. Content in a production table gets none of those unless you build them yourself.
What is still broken
- 9 of 150 pages written. The queue orders by traffic, so this is the head of the distribution, but 94% of the hand-written coverage is missing.
- No affiliate links yet. Zero of 4,323 listings have a commercial link. That is deliberate: I would rather send traffic to the vendor for free than send it to a tracker that does not pay.
- The traffic numbers are inherited, not measured. They come from the source catalogue. The only numbers I trust are outbound clicks, and the total is 4. Four clicks. That is the real baseline — everything else is a guess.
Why publish this
Because the honest version is more useful than a launch post. If you are building a directory, the tracking-domain contamination and the "content in the production DB" trap will both find you, and they are easier to spot when someone else has already paid for them.
If you want to look at the result: AIVerse, free AI tools
Start with the free filter — 2,198 listings, filterable by category, with the pricing field printed on every card so you can see what the data quality actually looks like. If a listing is wrong, there is a submission form and I fix it.
Top comments (0)