Keyword search is the wrong tool for monitoring anything

There is a specific bug that shows up in almost every monitoring system I have seen, and it never announces itself. The system runs fine. It sends its reports on time. Nobody gets an error. And the whole time, it is quietly missing most of what it was built to find.

The bug is keyword search.

How it starts

You are asked to watch some external source for things that matter to your company. Job postings, public tenders, competitor announcements, regulatory filings, whatever. The source has an API with a q= parameter. So you do the obvious thing:

results = api.search(q="ERP", limit=50)

You run it. Fifty results come back. They look reasonable. You ship it.

Here is what actually happened. Your query matched sixteen thousand records. The API returned the first fifty, ordered by whatever the API felt like ordering by. The other 15,950 were silently discarded - not by you, by the pagination default you never looked at.

And critically: there is no error. No exception, no warning, no truncation flag in the response. The system reports success. You will not find out from your logs. You will find out six months later when someone asks why you missed the biggest opportunity of the quarter, and the honest answer is that you never saw it.

Why raising the limit does not fix it

The first instinct is to page through everything. Sometimes you can. Usually you discover one of these:

• the API caps total pagination depth (first 1,000 results, then nothing)
• rate limits make a full sweep take hours
• the result set changes between pages, so you get duplicates and gaps
• 16,000 records is mostly noise anyway, and now you are paying to process it

You can fight this. I have. You end up with retry logic, cursor management, deduplication across pages, and a job that takes forty minutes and still cannot promise completeness.

The real problem is upstream: you are asking the wrong question. "Which records contain this string" is not the question. The question is "which records belong to the category I care about" - and on most serious data sources, somebody has already answered it.

Classification already exists, and it is free

Structured sources almost always carry an official classification, assigned at publication time by the publisher, because some regulation or process required it.

• EU public procurement notices carry CPV codes - a controlled vocabulary the buyer must select from when publishing
• Company filings carry industry classification codes
• Scientific papers carry subject categories
• Product feeds carry taxonomy IDs
• Job boards carry role categories

These fields exist specifically so that machines can filter. They are usually indexed, usually exact-match, and - this is the important part - the result set is bounded. Filtering "category = IT services, published in the last 7 days" returns maybe 900 records. You can take all of them. There is no page two you are quietly dropping.

The switch looks like this:

# before: unbounded, silently truncated, unknowable loss
results = api.search(q="ERP", limit=50)

# after: bounded, complete, and you can prove it
results = api.search(
    query="classification-cpv IN (72000000, 48000000) AND publication-date >= today(-7)",
    limit=100,
)

You do not have to take my word for any of this - the TED API is public and needs no key, so you can measure it yourself in about five minutes. Run a full-text query for a common term like ERP, then compare the totalNoticeCount the API reports against how many records it will actually hand you. That gap is the part you were never seeing. Then run the category version and watch the total drop to something you can fetch completely.

Precision improves, which is nice. But the real prize is the thing that does not show up as a percentage: the stream becomes complete. Before, you cannot say what you are missing. After, you can.

The obvious objection

"Publishers misclassify things all the time."

They absolutely do. This is a real weakness and pretending otherwise would be dishonest. Someone publishing a major software procurement will occasionally file it under office supplies, and category filtering will sail straight past it.

The fix is not to go back to keyword search. It is to run a second, independent net:

primary   = fetch_by_category(CATEGORY_CODES)      # complete, bounded
secondary = fetch_by_product_names(["salesforce", "uipath", "databricks"])
results   = deduplicate(primary + secondary)

The second pass is narrow on purpose. It searches for proper nouns - product names, vendor names, specific systems. Proper nouns are rare enough that the result set stays small and you are not back to 16,000 hits. It exists purely to catch what the first net structurally cannot.

Two nets with different failure modes beat one net with unknown holes.

The general principle

The thing keyword search really costs you is not precision. It is knowing what you don't know.

A system that returns 50 of 16,000 matches and a system that returns 900 of 900 matches can look identical from the outside. Same shape of output, same runtime, same absence of errors. The difference only shows up in what never reached you, which is exactly the thing you cannot measure from the inside.

So when you build the next monitor, before you reach for q=, spend ten minutes reading the API docs for a classification field. Ask:

1. Is there an official taxonomy on this data? Usually yes. It is usually documented badly.
2. Is my filtered result set bounded? If you cannot fetch all of it, you have not solved the problem, you have moved it.
3. What does my system do when a source returns nothing? If the answer is "reports zero and exits successfully", you have a second silent failure waiting.

That last one deserves its own article, and it is the one that has bitten me hardest.

---

I build and maintain automation and monitoring systems - mostly Python and RPA, mostly against public data sources that were never designed to be read by machines.