Ten full catalogues in a day: what the arr ecosystem's own backends will and will not let a directory carry

Sonarr, Radarr, Lidarr, Readarr and their cousins each sit on a metadata backend. We wanted the whole of every one of those in nichedb, so this morning we checked each backend live for one question: can a public directory that sells a premium tier carry the data, whole, with attribution? Then we built the ones that said yes.

What said no

TMDB and the IMDb datasets are non-commercial without a written contract. MyAnimeList's terms forbid aggregating its content anywhere else, which rules out Jikan too. Trakt, OMDb and Deezer are personal use only. RAWG bans redistribution outright, MobyGames charges for commercial use and treats any AI use as commercial, Fanart.tv wants written consent, Comic Vine revokes keys used commercially, and BoardGameGeek now needs a commercial licence and a bearer token. Sonarr's Skyhook and Radarr's api.radarr.video are proxies of TVDB and TMDB with no licence of their own, so they add nothing the originals do not. Spotify and Google Books forbid storing anything at all.

What said yes, and is now in

Every show TVmaze knows, close to 90,000, CC BY-SA, walked at 250 a page and then kept current from its updates feed. Every game IGDB knows, about 350,000; IGDB says in its own words that it prefers the data stored and served. Every film and every video game on Wikidata, CC0, walked in numeric id windows over SPARQL. Every team and every player TheSportsDB knows; the players land as OpenProfile.md people with their Twitter, Instagram, Facebook and YouTube, so an athlete who later serves their own profile merges rather than duplicates. Every anime on Kitsu. Project Gutenberg's whole catalogue and every LibriVox audiobook, both public domain. Every app on Steam through the keyless store query, since the old app list endpoint is dead.

Each source is a capped, resumable walk: a run asks for a fixed number of pages, keeps its place in a cursor, ends its pass when it runs past the end, and starts over on a schedule. Each one states its licence in its own description, including the one that has none stated.

How it was built

One agent per source, each on its own branch, each saving real responses as test fixtures and probing the live API from its own code path. Then a second agent per source whose job was to prove the first one wrong: rerun the tests, hit the real API with the adapter's exact URLs, run two consecutive pulls and check the second starts where the first stopped, and read the licence sentence against the actual terms. Seven of eight were corrected by review. Wikidata's truthy date property drops precision, so year-only dates arrive as the first of January; an inner subquery without DISTINCT let a doubly-typed item shorten a page and skip rows; a conditional request compared against the wrong timestamp; three licence sentences claimed more than the terms grant. None of that would have shown up in a green test suite.

The pattern is written down at logicsrc.com/asdlc. The catalogues are at nichedb.dev.