Building a news aggregator that pulls from hundreds of independent outlets raises an obvious temptation: some sites block AI crawlers by name, so why not just... not identify as one? We built Rhyter Times's source pipeline on the opposite rule.
The rule
Our crawler always identifies itself honestly. If a publisher's robots.txt says AI crawlers aren't welcome, we don't crawl that site — full stop. Not with a different user agent, not by finding a technicality that lets us through anyway. If a site blocks AI crawlers by name but leaves the door technically open for everyone else, we treat that as the publisher's real answer, not a loophole to route around.
This costs us real coverage. A meaningful share of major outlets explicitly block AI crawlers, and we simply don't include them. That's the tradeoff for doing this honestly, and it's the only version of "our own source directory" we're willing to build — evading blocks would defeat the entire point the moment anyone checked.
What we ended up with anyway
Building it this way still got us to 400+ real, permissioned sources across 10 world regions and 5 topics: national broadcasters, university research newsrooms, regional business press, and a long tail of country-specific outlets that never show up in typical aggregator source lists, because nobody bothered building relationships with them at scale before.
The result is live at rhytertimes.com, and the same source directory now powers an API and MCP server for anyone building on top of it — aristocles.com.au/products/news, including transparency into exactly which outlets feed a given response, not just "trust us."
Curious how other people building on public data draw this line — happy to discuss in the comments.
Top comments (0)