How to compare 100 Google Trends keywords on one 0-100 scale (beyond the 5-term limit)

#python #datascience #seo #webscraping

Google Trends is the go-to free source for "which of these is more popular?". But it has two limits that break any serious keyword research:

Max 5 terms per comparison.
Each request is rescaled to its own maximum = 100.

So if you export 20 batches of 5 keywords and paste them together, a "50" in batch 1 and a "50" in batch 7 are different things. In a simulation with known ground truth (Google-style rescaling + integer rounding), that naive concatenation was off by more than 20 index points.

The method: anchor chaining

Query the first 5 keywords and pick a mid-popularity anchor.
Query every other keyword as anchor + 4 others. The anchor gives you the ratio between batches.
Problem: if a keyword is 1,000x smaller than the anchor, Google rounds it to 0 or "<1". So keywords with poor precision are re-measured in overlapping batches with keywords of similar size, building a chain of anchors.
Solve one scale factor per batch with weighted least squares over all shared keywords (weights come from rounding precision, so a series summing to 3 counts far less than one summing to 3,000).
Rescale so the global maximum is 100.
Self-check: take 5 keywords that were measured in different batches, query them directly together and compare with the chained values.

Measured accuracy (real runs)

Run	Keywords	Self-check mean abs. error (0-100)
Worldwide, 5 years	100	0.187 points (max 1.01)
US-CA, 2024, YouTube	25	0.149 points
US, 12 months, cloud + residential proxy	25	0.592 points

Under 1 point on a 0-100 scale, versus 20+ points for naive concatenation. Google samples its data, so a small error is expected even between two identical queries.

Try it

I packaged this as an Apify Actor: Google Trends Scraper – Compare 100 Keywords. Country/region/metro, preset or custom dates, Web/YouTube/News/Images/Shopping, CSV/JSON output, and the self-check report in every run.

A minimal Python example (12 keywords, prints the self-check error and saves a chart) is on GitHub: https://github.com/carlos-calatayud/google-trends-compare-100-keywords

from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("satin_armeria/google-trends-advanced-comparator").call(run_input={
    "keywords": ["chatgpt", "claude", "gemini", "perplexity", "copilot", "midjourney"],
    "geo": "", "timeframe": "today 12-m",
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)

What it costs

$4 per 1,000 keywords compared ($0.004 each) plus a $0.00005 start fee. Keywords that fail or return no data are not charged. 100 keywords ≈ $0.40; the 12-keyword example ≈ $0.05. Apify's free plan includes $5/month of credit.

Limitations (honest)

Not an official Google API; data is sampled and can vary slightly between runs.
Extremely small keywords next to huge ones may still return "no data".
A 100-keyword run takes a few minutes because requests are throttled politely to avoid rate limits.

Feedback on the method is welcome — especially edge cases where the chain breaks.

DEV Community