Search directories become unreliable when the source data changes faster than the pages built from it. I ran into that problem while building ShortDramaBuzz, an independent directory for short-form vertical dramas.
The goal was not to scrape everything and hope for the best. It was to publish only fields that could still be traced to an official listing.
Treat each catalog as an adapter
Every platform exposes a different page structure, but each collector maps into the same small contract: an official ID, title, source URL, destination URL, cover, timestamp, and only the facts that the source actually publishes.
That makes source-specific breakage visible. A changed page structure fails one adapter instead of quietly corrupting every page.
Fail closed on suspicious refreshes
A daily refresh should not automatically become a daily deployment. I added checks for duplicate IDs and slugs, missing required fields, stale snapshots, implausible record drops, broken internal links, and orphan pages.
If a catalog suddenly falls by more than the allowed threshold, the build keeps the last known-good snapshot. A stale but verified page is better than a fresh page built from a broken crawl.
Separate source facts from editorial context
Official titles, episode totals, labels, counters, and destinations stay attributable to the source. Genre introductions, comparisons, and viewing guidance are written separately. Missing fields are omitted rather than guessed.
The same rule applies to duplicate listings. When a platform remounts the same title under a new numeric ID, the directory keeps one canonical guide and redirects the duplicate instead of manufacturing two unique pages.
Make the methodology public
Trust is easier to evaluate when readers can see what is collected, how often it is checked, and what stops publication. The full ShortDramaBuzz data methodology documents the current sources, validation rules, and limitations.
The useful lesson for any fast-changing directory is simple: freshness matters, but provenance and failure behavior matter more.
Top comments (0)