Filter spam out of web search results for an LLM in three layers: set domain allow or block lists in the search request, score or rerank what comes back in your own code, and let the model drop sources that fail a quality check before it answers. Exa, Tavily, Parallel, Perplexity, Linkup and Firecrawl take include and exclude domain lists, Brave takes Goggles that boost, downrank or discard sites, and a Google SERP API takes Google's own site: and -site: operators.
-
Allow lists are precise but fragile. They remove spam and good pages you did not list. Parallel's docs warn that source policies "can significantly reduce result quality", and Tavily added a
prefermode for this reason. - Block lists have hard caps. Perplexity's Search API takes 20 domains, Linkup 100, Tavily 150 excluded, Parallel 200 entries in total. A block list of thousands of spam domains belongs in your own code.
- Some poisoning is upstream. Since September 2026, Google answers some automated searches with a full page of the wrong sites. A domain filter cannot see that.
A pipeline that works for most agents:
- Choose an index and category that fit the task: news, research papers or the open web.
- Add a short block list to the search request, and an allow list only for a closed set of sources.
- Apply your own domain tiers to the results: trusted, neutral, blocked.
- Rerank the rest against the query and keep the top few.
- Ask the model to grade each source before it uses it, and to cite only the ones it keeps.
What does each search API let you filter in the request?
Each row comes from the vendor's docs, read on November 16, 2026.
| API | Allow | Block | Limit | Other quality controls |
|---|---|---|---|---|
| Exa | includeDomains |
excludeDomains |
Paths and *.substack.com wildcards |
category: news, publication (scholarly), company, people, financial report
|
| Tavily |
include_domains, mode restrict or prefer
|
exclude_domains |
300 include, 150 exclude |
topic: "news", safe_search, a relevance score per result |
| Parallel | include_domains |
exclude_domains |
200 entries in total; no paths in turbo
|
after_date |
| Perplexity Search API | search_domain_filter |
Same field, with a - prefix |
20 domains, allow or block, not both | search_language_filter |
| Linkup | includeDomains |
excludeDomains |
100 domains | A quality scoring model on by default |
| Firecrawl | includeDomains |
excludeDomains |
One or the other per request |
categories: research, pdf
|
| Brave | Goggles $boost
|
Goggles $downrank, $discard
|
3 Goggles per request | Works on Web, News and LLM Context |
| Google SERP APIs |
site: in the query |
-site: in the query |
Query length | Google's own ranking |
A few details matter in practice. Firecrawl's docs say its domain fields "add site: and -site: operators to the query internally". Tavily's prefer mode searches the rest of the web too, so a trusted-source list does not return an empty result. Linkup says it blocks malware and phishing categories by default and applies "a quality scoring model" to rank sources (Linkup FAQ).
For the common request "only reputable news and research papers", the category filters do most of the work:
// Exa: scholarly sources only
{ "query": "retrieval augmented generation evaluation", "category": "publication" }
// Tavily: news, minus two sites you do not trust
{ "query": "EU AI Act enforcement", "topic": "news",
"exclude_domains": ["example-content-farm.com", "example-aggregator.net"] }
Brave's Goggles are the most flexible. One request can boost a site, push another down and remove a third:
$boost=3,site=reuters.com
$downrank=5,site=example-content-farm.com
$discard,site=example-spam.com
On a Google SERP API the filter is the query itself. In one test on November 16, 2026, the query retrieval augmented generation evaluation site:arxiv.org on cloro returned 10 of 10 organic results from arxiv.org.
How do you score source quality after the search?
The request filters are short lists. Your own code has no cap, so keep the long lists there.
Domain tiers. Keep a file of trusted, neutral and blocked domains, and apply it to every result before the model sees it:
from urllib.parse import urlparse
TRUSTED = {"reuters.com", "apnews.com", "nature.com", "arxiv.org"}
BLOCKED = {"example-content-farm.com", "example-spam.com"}
def host(url):
h = urlparse(url).hostname or ""
return h[4:] if h.startswith("www.") else h
def keep(results, max_n=5):
scored = []
for r in results:
h = host(r["url"])
if h in BLOCKED or any(h.endswith("." + b) for b in BLOCKED):
continue
bonus = 1.0 if h in TRUSTED else 0.0
scored.append((r.get("score", 0) + bonus, r))
return [r for _, r in sorted(scored, key=lambda x: -x[0])[:max_n]]
Rerank. A reranker model scores each result against the query, and the low scores go. Jina's Reranker API (https://api.jina.ai/v1/rerank) and Tavily's per-result score are two options; Tavily's own extract guide filters at a score above 0.5 as an example.
Let the model grade. Before the model answers, ask it to rate each source and to drop the weak ones. Give it criteria it can check in the text: who published it, whether it is a primary source, whether it has a date, whether the same text appears on several domains. This costs tokens on every call, so run it only on the five sources you kept.
Treat page text as data. A spam page can carry instructions written for an LLM. Keep retrieved text in a separate part of the prompt and tell the model that it holds no instructions.
What does "poisoned search data" mean?
The term covers two different problems, and they need different fixes.
Pages built to be retrieved. Content farms, affiliate pages and AI-written copies rank for a query or match an embedding. Domain lists, Goggles, reranking and model-side grading handle this kind.
A degraded results page. Here the search engine itself returns the wrong page to automated traffic. Nozzle, a rank tracker, named it a "poisoned SERP" in October 2026, and DataForSEO published the same finding. The page looks normal: ten results, ads, an AI Overview. Most of the sites that usually rank are gone, and no error tells you.
We measured it on our own Google data. At the worst point, 30 September to 12 October 2026, 71.5% of cloro's Google pages lost most of their usual sites, with single days at 83%. SerpApi's pages, scored against the same baseline, reached 73.2% from 24 to 29 September, with single days at 89%. After cloro added checks against reference queries with a known first page, it returned the full page on 95.8% of reads in the 12 hours to 06:00 UTC on 31 October. The method and the other providers' numbers are in poisoned Google SERPs, measured.
A domain filter does not catch this, because the wrong page passes every check. Two tests do: run a few reference queries whose correct top results you know, and alert when most of the usual domains vanish at once. And ask your provider how it measures result quality as well as its success rate.
How this was checked
This article is published by cloro, a Google SERP API. It belongs to the "Google SERP APIs" row of the filter table and is not ranked against the other APIs. The parameters and limits of the other APIs come from each vendor's docs, read on November 16, 2026, and are linked in the table. The site: test was one live request on that date. The poisoned-SERP figures are cloro's own measurements on reference queries, and the linked post describes the method and its limits. The code samples are minimal sketches: test each filter on your own queries before you rely on it.
Top comments (0)