DEV Community

Elowen
Elowen

Posted on

How Overlapping Google News Pages Can Trigger Duplicate Alerts

A monitoring pipeline can store a clean final table and still process the same story twice. This happens when each row reaches the alert check before paginated results are deduplicated.

I tested this failure mode offline with two real Google News responses for the query coffee. The start=0 request returned 13 results. The start=10 request returned 10. Concatenating both arrays produced 23 rows.

The start parameter is an offset. It does not guarantee that the previous response contains exactly 10 results. In this sample, the start=0 request returned 13 results, so the two response ranges overlapped.

One URL appeared twice

The 23 rows contained 22 unique URLs. This Axios story appeared in both responses:

https://www.axios.com/local/charlotte/2026/08/17/wnba-player-downtown-concord-coffee-shop
Enter fullscreen mode Exit fullscreen mode

It had an in-response rank of 13 in the start=0 request and 3 in the start=10 request. A program that checks every row would send that story into the alert decision twice.

The offline simulation ran 23 alert checks before deduplication. The duplicated Axios URL appeared in two checks. Deduplicating first reduced the input to 22 checks.

No live monitoring system or notification service was connected during this test. These numbers describe an offline simulation of the processing order.

Normalize the URL before comparing rows

The same page can arrive with a trailing slash, a fragment, or UTM tracking parameters. Cleaning those parts creates a more stable deduplication key.

function normalizeUrl(raw) {
  const url = new URL(raw);
  url.hash = '';

  ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content']
    .forEach((key) => url.searchParams.delete(key));

  if (url.pathname !== '/') {
    url.pathname = url.pathname.replace(/\/+$/, '');
  }

  return url.toString();
}
Enter fullscreen mode Exit fullscreen mode

This rule keeps other query parameters. Some sites use them to identify different pages, so removing every parameter can merge separate results.

Deduplicate before the alert check

The reliable order is: concatenate the responses, normalize each URL, deduplicate the rows, then run the alert logic.

const seen = new Set();
const uniqueRows = [];

for (const row of rows) {
  const key = normalizeUrl(row.url);
  if (seen.has(key)) continue;

  seen.add(key);
  uniqueRows.push({ ...row, normalized_url: key });
}

for (const row of uniqueRows) {
  console.log('Run alert check once for:', row.normalized_url);
  // Call your own alert decision function here.
}
Enter fullscreen mode Exit fullscreen mode

In this offline simulation, the input dropped from 23 rows to 22 unique URLs. The Axios story entered the alert check once after deduplication.

Keep the start value and the original in-response rank with each row. They show which requests produced an overlap. Use the normalized URL as the deduplication key for this workflow.

When testing a paginated monitor, inspect the rows received by the alert function as well as the final stored table. Display-layer deduplication cannot undo an earlier duplicate check.

Top comments (0)