Find keyword cannibalisation with the Search Console API in 20 lines

#python #api #seo #webdev

If you run a content site with more than a few hundred pages, some of your pages are competing against each other in Google. You probably suspect it. Here is how to prove it with the Search Console API and about twenty lines of Python.

I ran this on a site with ~5,000 URLs and 23,000 ranking queries. It found 175 queries where three or more of my own pages were splitting the same search. One topic had thirty pages fighting over six queries, none of them ranking above position 11.

The trick: query the two-dimension breakdown

Most people pull Search Console data one dimension at a time — queries, or pages. The useful view is both at once:

from google.oauth2 import service_account
from googleapiclient.discovery import build

creds = service_account.Credentials.from_service_account_file(
    "service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)
svc = build("searchconsole", "v1", credentials=creds)

rows, start = [], 0
while True:
    batch = svc.searchanalytics().query(
        siteUrl="https://example.com/",
        body={
            "startDate": "2026-05-01",
            "endDate": "2026-07-30",
            "dimensions": ["query", "page"],   # <- both
            "rowLimit": 25000,
            "startRow": start,
        },
    ).execute().get("rows", [])
    rows += batch
    start += 25000
    if len(batch) < 25000:
        break

Note the pagination. rowLimit maxes out at 25,000 per call and the two-dimension breakdown blows past that fast — my 90-day pull returned 46,813 pairs. Without the loop you silently get a truncated picture and draw the wrong conclusions.

Grouping

from collections import defaultdict

by_query = defaultdict(list)
for r in rows:
    q, page = r["keys"]
    by_query[q].append((page, r["impressions"], r["position"]))

problems = []
for q, hits in by_query.items():
    impressions = sum(i for _, i, _ in hits)
    if impressions < 150 or len(hits) < 3:
        continue
    best = min(p for _, _, p in hits)
    problems.append((impressions, len(hits), best, q))

problems.sort(reverse=True)

The filter that matters most

My first version flagged everything with 3+ pages and produced a list I could not act on. The top entry had 29 pages on one query — and its best position was 2.0. That is not a problem. That is a brand SERP with sitelinks, working exactly as intended.

Cannibalisation only hurts when nobody wins:

problems = [p for p in problems if p[2] > 8]   # p[2] = best position

That single line cut my list from 175 entries to 5 real ones. Everything else was either a brand query or a topic where one page was already ranking fine and the others were harmless long-tail noise.

Sitelinks are the trap here. On a brand query, Google reports an impression for every sitelink URL in the block, but the click only lands on the main result. So you see six URLs at position 1.4 with 0% CTR and conclude you have a catastrophic CTR problem. You do not. You have one result and five sitelinks. Check whether the homepage in the same query is picking up the clicks before you "fix" anything.

Fixing it without deleting pages

The textbook answer is consolidate-and-redirect. That is often the right call, but it is also destructive and hard to reverse across dozens of URLs.

The cheaper first move is to stop the pages competing:

Pick the page that already ranks best on the head term. It has the links and the history; do not fight it.
Give it the exact-match title for that query.
Retitle every satellite to a distinct sub-intent — a different question, not a variation of the same one.
Add a link from each satellite to the primary, with a stable HTML marker so the operation stays idempotent:

MARKER = "<!--canonical-hint-liver-->"
if MARKER not in body_html:
    body_html = MARKER + link_block + body_html

The marker matters more than it looks. It lets you re-run the script safely, which you will, because you will get the intent assignment wrong on the first pass.

What I would do differently

I would have run this detector before writing 200 more pages on topics I already covered. The impressions were never the constraint — the site had plenty. The constraint was that the pages were dividing the signal instead of concentrating it.

If you are about to commission another batch of articles, run the query-times-page breakdown first. It takes ten minutes and it will probably tell you to write fewer, better pages.

The site behind this is INTI, a Belgian organic ginger elixir with a very over-enthusiastic content history. The consolidated topic cluster from the example is ginger and the liver, if you want to see the pattern in production.

What is the worst cannibalisation you have found on your own site? I am curious whether thirty pages on one topic is a record or merely average.

DEV Community