Building an AI News Aggregator Without Drowning in Duplicate Stories

Most AI news aggregator demos stop at the pleasant part: run a search, hand the results to an LLM, and render a neat summary.

The frustrating part begins on the second refresh.

An article from last month appears next to today's announcement. Three different URLs turn out to be the same press release. Five publishers repeat one wire story, making it look as if five independent sources confirmed the news. Then the summary introduces a detail that was not in any of the retrieved pages.

At that point, the project is no longer a search box with a summarizer attached. It is a small editorial system, and most of the work sits between retrieval and generation.

This post walks through that middle layer. The example is a feed that tracks AI chip export controls, but the same approach works for product launches, regulatory updates, scientific news, or any other narrow topic that changes often.

The search query is not the product

Starting with a broad query such as AI chips creates an impossible filtering job. The phrase covers hardware releases, benchmark results, earnings calls, research papers, stock commentary, supply-chain rumors, and government policy.

For a useful feed, the topic needs boundaries. In this case, I would limit it to new export rules, enforcement actions, official statements, company responses, and meaningful supply-chain effects. I would also use a seven-day window and exclude product reviews, undated explainers, and opinion pieces that do not add new reporting.

That scope can be expressed through a handful of short searches:

AI chip export controls
advanced semiconductor export restrictions
AI accelerator export license
chip export controls company response

This works better than a single oversized query. A regulator may write about an “export licensing requirement” while a publisher calls the same change an “AI chip restriction.” Several small queries catch that variation and are easier to debug when irrelevant results get through.

The retrieval source matters too. RSS is excellent for publishers that are already on a watch list. A dedicated News API is convenient when normalized article metadata is the priority. Web search is useful when the important source may be a regulator, a company newsroom, or a specialist publication that was not known in advance.

There is no need to choose only one. A practical feed can use RSS for known sources and web search for discovery.

Search results are pages, not stories

The first pass should be cheap. Before downloading full pages or calling a model, inspect the title, URL, snippet, domain, and publication date.

A title that does not contain the expected company, regulator, country, or policy term is usually easy to reject. Category pages and tag indexes can go as well. If a result has no publication date, treat the date as unknown rather than assuming it is recent. Search freshness filters help, but pages are sometimes updated or republished in ways that make old reporting look new.

It is worth keeping the original title, URL, publisher, and date even when the full article is fetched later. Those fields become part of the audit trail behind the final summary.

Here is a small Python baseline. It runs several searches, asks for results from the last week, removes repeated URLs, and sorts what remains. The example uses the Cloudsway Smart Search API because it can return both search metadata and extracted page text; the rest of the pipeline is provider-independent.

import os
from urllib.parse import urlsplit, urlunsplit

import requests


API_URL = "https://aisearchapi.cloudsway.net/api/search/smart"
HEADERS = {"Authorization": os.environ["CLOUDSWAY_API_KEY"]}


def canonical_url(url: str) -> str:
    parts = urlsplit(url)
    return urlunsplit(
        (parts.scheme, parts.netloc.lower(), parts.path.rstrip("/"), "", "")
    )


def search_news(query: str) -> list[dict]:
    response = requests.get(
        API_URL,
        headers=HEADERS,
        params={
            "q": query,
            "count": 20,
            "freshness": "Week",
            "enableContent": "true",
            "mainText": "true",
            "contentType": "TEXT",
        },
        timeout=20,
    )
    response.raise_for_status()
    return response.json().get("webPages", {}).get("value", [])


queries = [
    "AI chip export controls",
    "advanced semiconductor export restrictions",
    "AI accelerator export license",
]

seen = set()
articles = []

for query in queries:
    for item in search_news(query):
        url = canonical_url(item["url"])
        if url in seen:
            continue

        seen.add(url)
        articles.append(
            {
                "title": item.get("name"),
                "url": url,
                "published": item.get("datePublished", ""),
                "text": item.get("mainText", ""),
                "score": item.get("score", 0),
            }
        )

articles.sort(
    key=lambda article: (article["published"], article["score"]),
    reverse=True,
)

print(articles[:10])

Set CLOUDSWAY_API_KEY in the environment before running the script. The quick start covers authentication, and the search reference lists the available parameters.

This code is intentionally incomplete. It retrieves candidates and removes exact URL duplicates. The harder duplicate problem comes next.

The duplicate problem has more than one layer

Some duplicates are mechanical. Tracking parameters, fragments, and trailing slashes create different URLs for the same page. Normalizing the URL handles many of these.

Syndication is less obvious. The same article may appear on several domains with a slightly modified headline. Comparing normalized titles, opening paragraphs, named entities, and content hashes can catch most copies.

But two articles about the same event are not necessarily duplicates. A government notice and a chipmaker's response belong to one story, yet both may be valuable. Deleting either one loses context; displaying them as separate events makes the feed repetitive.

This is where event clustering becomes more useful than another deduplication rule.

For each candidate, build a compact representation from the headline, entities, publication time, and central claim. Group candidates that share the same entities and describe the same development within a reasonable time window. Embeddings are helpful when headlines use different language, but they should not decide on their own. Two articles can sound similar while referring to different rules, countries, or dates.

Inside a cluster, choose a lead source based on relevance, recency, source quality, and completeness. Keep the other independent sources attached. An official document may be best for establishing what changed, while reporting from a specialist publication may better explain the commercial impact.

A useful sanity check is to ask where each article's information originated. Ten sites repeating one wire report are still one line of reporting.

Let the model in late

The LLM should see an event cluster, not a raw page of search results.

Give each retained source an ID such as S1, S2, and S3, then provide only the title, date, relevant passage, and URL metadata returned by retrieval. Ask the model for a neutral headline, a short summary, why the event matters, and the source IDs behind each factual statement.

The important restriction is that the model cannot create a URL. It can only refer to an ID from the supplied evidence. After generation, the application maps the IDs back to stored URLs and rejects anything outside the set.

That still does not make every summary correct. A real page may be cited for a claim it does not support. The final check should compare the claim with the cited passage, not merely confirm that the URL exists.

The output also needs a way to express uncertainty. If credible sources disagree, preserve the disagreement. If there is not enough evidence, say so. For a sensitive story, requiring a primary source or two genuinely independent reports is usually safer than letting the model turn a weak signal into a confident update.

What I would add before scheduling it

The next production feature would not be a nicer UI. It would be logging.

Store the query that found each result, the filters it passed, its cluster assignment, and the source IDs used in the summary. When a bad item appears in the feed, this makes it possible to tell whether retrieval, date handling, clustering, ranking, or generation failed.

Caching matters for the same reason. A monitoring job should not repeatedly fetch and summarize pages that have not changed. Retries need limits and backoff, and a single domain should not be allowed to dominate the feed just because it publishes aggressively.

I would also keep a manual review path for low-confidence clusters and high-impact claims. Automation is useful here because it reduces a noisy stream to a manageable set of evidence. It should not remove the ability to inspect how a conclusion was reached.

Finally, an aggregator should link to original reporting instead of reproducing full articles. Facts can be summarized, but the article's language, images, and other protected expression come with separate copyright and licensing considerations.

Closing thought

The summarizer is the most visible part of an AI news product, but it is not the part that makes the feed trustworthy.

That comes from narrower queries, careful date handling, separating copied articles from independent coverage, grouping pages into events, and keeping generated statements tied to material the system actually retrieved.

If you have built a monitoring feed, I am curious how you handle event-level duplication. Do embeddings work well enough for your topic, or have you ended up combining them with entity and date rules?

---

Note: AI tools were used to help edit this post.