Most “news database” projects start as search tabs
Someone opens a news search UI, types a topic, scrolls a page of headlines, and calls it research. That habit is fine for a quick read. It fails when the real job is a countable news corpus: export N rows for a model, size a vertical before you bid a vendor, or prove that last month’s Tech lane was tens of millions of articles - not “the first page looked busy.”
Search engines optimize for relevance and a short result set. A news database (structured article rows behind an API) optimizes for filters, dates, publishers, and a total you can trust. Perigon’s News Database and how to query it walks the product census and Articles calls. This page owns a different question: which job are you running, and what breaks if you pick the wrong one?
1) What is the difference between news search and a countable corpus?
2) When is an alerts feed enough instead?
3) Which failure modes waste time and API quota?
4) What decision checklist should a desk run before the first pull?
News search gives pages; a corpus gives rows you can count
News search (Google News style, site search, “top results for q”) answers: what should I read right now? You get ranked hits. You rarely get a durable total for “how many English Business articles mentioned shipping in August,” and you cannot treat the first page as a sample of the full store.
A countable news corpus answers: how many rows match these fences, and can I page them under a budget? Each row is an article record - title, publisher domain, pub date, plus enrichment when the vendor offers it (companies, categories, language). You ask for a total first, then decide whether to export, sample, or tighten filters.
| Job signal | News search habit | Countable corpus habit |
|---|---|---|
| Success metric | Good headlines on page one | Sane numResults (or equivalent) before export |
| Time box | “Last week” in the UI | Explicit from / to on publication time |
| Scope | Keywords in a box | Keywords plus source, category, company, language |
| Risk | Missing the long tail | Pulling the long tail without a size check |
If your deliverable is a briefing for a human, search or an alerts product may win. If your deliverable is a spreadsheet, a training slice, a vendor bake-off, or a volume chart, you need the corpus path.
Alerts and feeds are a third lane, not a tiny database
Teams often collapse three tools into one word - “news DB.” Keep them separate.
- Search - ad hoc discovery; short result pages.
- Alerts / monitors / feeds - ongoing slice of coverage pushed to email, Slack, or a dashboard. Great for “tell me when this beat moves.” Weak for one-shot corpus sizing.
- Queryable news database - the full (or plan-depth) article store with filters and totals. Great for research pulls and product backends. Overkill if you only needed a daily digest.
Use a feed when the desk already knows the beat and wants continuity. Use a countable corpus when you need a number, a dated export, or a reproducible filter set. Use plain search when a person is still figuring out which story matters today.
Failure modes that burn quota and credibility
Page-first pulls. Teams set page size to 50, walk pages, and stop when they feel “done.” On a multi-million-row match set that is neither a census nor a fair sample. Count first. Raise page size only after the total looks sane for the budget.
Keyword-only fences on hot topics. Broad q strings on Tech or Politics can look like “the whole database.” Add publisher allowlists, category, language, or company pins before you claim a vertical size.
Treating reprints as unique events. Corpus row counts include coverage volume, not unique world events. If the brief needs narratives, cluster or story APIs belong in the plan - do not invent uniqueness from raw article totals.
Mixing plan depth with index size. Archive lookback is often plan-gated. A short lookback is not proof the vendor’s store is small; it is proof your key cannot see farther. Say that clearly in bake-off notes.
Copying someone else’s billion-row slide. Index totals move daily. Re-run a size check for the window you publish. Perigon’s news database guide shows a live open-index census pattern; do not paste an old deck number into a new RFP.
Decision checklist before the first corpus pull
Run this in order. Stop early if a cheaper lane wins.
1) Write the deliverable in one sentence: briefing, alert, volume chart, export, or model sample.
2) If the sentence is “keep me updated,” buy or configure a feed/monitor - skip the full corpus pull.
3) If the sentence needs a total or an export, freeze filters (topic, dates, language, sources, companies) before any large page loop.
4) Request a count-only call (one sample row + total) and record the number with the window and filters.
5) Only then raise size / pages - or tighten fences if the total blows the budget.
6) Label what you measured: article rows, not unique events, unless you also used a clustering path.
That sequence is the ops spine. Endpoint param teaching and the current open-index scale live on the Perigon news database post linked above - keep this page for the job choice, not the full Articles cookbook.
Key takeaways
- News search optimizes for readable hits; a countable news corpus optimizes for filterable rows and totals.
- Alerts and feeds solve continuity; they are not a substitute for corpus sizing or exports.
- Count before you paginate. Broad keywords without fences waste quota and invent fake “database” stats.
- Say whether you measured article rows or unique narratives - then re-check live totals before you publish a hard number.
Top comments (0)