DEV Community: Raspberry Rondel

Full-Page Screenshots of Chinese, Japanese and Emoji Pages with Headless Chromium (No Tofu Boxes)

Raspberry Rondel — Sun, 18 Oct 2026 16:28:07 +0000

If you take website screenshots with headless Chromium for visual QA, monitoring or archiving, two problems show up the moment the pages aren't plain English:

Tofu boxes. Chinese, Japanese, Korean and emoji come out as empty rectangles (□□□), because the server or container has no font for them.
Half-empty full-page shots. The page is the right height, but images further down are grey placeholders and "load more" content never appears, because nothing scrolled the page.

There's also a subtler third one: the text renders, but Traditional Chinese is drawn with the Japanese font, so some characters and punctuation have the wrong regional shape.

This post reproduces all three on a small local test page and fixes them with Playwright for Python. Everything below was run, and the output is copied from that run.

Test setup: Linux, Python 3.13, Playwright for Python 1.64.0 with its bundled Chromium headless shell (156.0.8078.4), and the Debian/Ubuntu packages fonts-noto-cjk and fonts-noto-color-emoji.

The test page

site/index.html has a title with an emoji, one paragraph each in Traditional Chinese, Simplified Chinese, Japanese and Korean, an emoji line (including a skin-tone modifier, a flag and a ZWJ family), and then:

two images with native loading="lazy",
two images that use the common JavaScript pattern (data-src swapped in by an IntersectionObserver),
an "infinite scroll" sentinel that appends a new section (with an image) three times as you reach the bottom.

The <html> tag says lang="zh-Hant", and the Simplified Chinese paragraph says lang="zh-Hans". Some real sites use script subtags like these.

<html lang="zh-Hant">
...
<p id="zh-Hant">繁體中文：臺灣的「鐵路便當」很好吃。數字 0123456789，標點「」、。！</p>
<p id="zh-Hans" lang="zh-Hans">简体中文：这是一个用于测试网页截图的页面。</p>
<p id="ja" lang="ja">日本語：ウェブページのスクリーンショットをテストします。</p>
<p id="ko" lang="ko">한국어: 웹 페이지 스크린샷 테스트입니다.</p>
<p id="emoji">Emoji: 😀 🎉 👍🏽 🇹🇼 👨‍👩‍👧 ❤️</p>
...
<img class="js-lazy" data-src="img/3.svg" alt="js lazy image 3">

The full page, the images and a script that runs every case are in the companion folder (site/, run_demo.sh).

The script

shoot.py does four things in this order:

Scrolls down in 600 px steps with a 400 ms pause, until the page height stops growing for two rounds at the bottom (or a time/height limit is hit), then scrolls back to the top so sticky headers sit where a visitor would see them.
Waits for images that are still loading (with a timeout) and for document.fonts.ready.
Reports fonts. It asks Chromium, through the DevTools Protocol method CSS.getPlatformFontsForNode, which installed font actually drew each element you list, and how many glyphs came from each.
Captures with full_page=True and animations="disabled".

"""Full-page screenshot with headless Chromium (Playwright for Python).

- scrolls the page first so lazy-loaded images and infinite-scroll content load
- waits for images and web fonts before capturing
- reports which real font Chromium used for each element you name (via CDP),
  so you can catch tofu boxes before you look at the PNG

Usage:
  python shoot.py URL out.png [--no-scroll] [--locale zh-TW] [--fallback-lang zh_TW.UTF-8]
                  [--check "#id1,#id2"]
"""
import argparse
import os
import time

from playwright.sync_api import sync_playwright

SCROLL_JS = """
async ({ step, pauseMs, maxSecs, maxHeight }) => {
  const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
  const height = () => Math.max(document.documentElement.scrollHeight,
                                document.body ? document.body.scrollHeight : 0);
  const start = Date.now();
  let y = 0, lastHeight = -1, stableRounds = 0;
  while (Date.now() - start < maxSecs * 1000) {
    y = Math.min(y + step, height());
    window.scrollTo(0, y);
    await sleep(pauseMs);
    const h = height();
    if (y + window.innerHeight >= h) {          // reached the bottom
      stableRounds = (h === lastHeight) ? stableRounds + 1 : 0;
      if (stableRounds >= 2 || h >= maxHeight) break;   // nothing new loaded
    }
    lastHeight = h;
  }
  window.scrollTo(0, 0);                         // sticky headers back in place
  return { height: height(), seconds: (Date.now() - start) / 1000 };
}
"""

WAIT_ASSETS_JS = """
async (timeoutMs) => {
  const imgs = [...document.images].filter((i) => !i.complete);
  const each = imgs.map((i) => new Promise((r) => { i.onload = i.onerror = r; }));
  await Promise.race([Promise.all(each), new Promise((r) => setTimeout(r, timeoutMs))]);
  if (document.fonts) await document.fonts.ready;
}
"""

IMG_STATS_JS = """
() => {
  const imgs = [...document.images];
  const loaded = imgs.filter((i) => i.complete && i.naturalWidth > 0).length;
  return { images: imgs.length, loaded };
}
"""


def platform_fonts(page, selectors):
    """Ask Chromium which installed fonts it actually used to draw each element."""
    cdp = page.context.new_cdp_session(page)
    cdp.send("DOM.enable")
    cdp.send("CSS.enable")
    root = cdp.send("DOM.getDocument")["root"]["nodeId"]
    report = {}
    for sel in selectors:
        node = cdp.send("DOM.querySelector", {"nodeId": root, "selector": sel})["nodeId"]
        if not node:
            report[sel] = "not found"
            continue
        res = cdp.send("CSS.getPlatformFontsForNode", {"nodeId": node})
        fonts = [f"{f['familyName']} ({f['glyphCount']})" for f in res["fonts"]]
        report[sel] = ", ".join(fonts) or "(no text)"
    return report


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("url")
    ap.add_argument("out")
    ap.add_argument("--no-scroll", action="store_true")
    ap.add_argument("--locale", default="zh-TW")
    ap.add_argument("--width", type=int, default=1280)
    ap.add_argument("--height", type=int, default=800)
    ap.add_argument("--max-height", type=int, default=20000)
    ap.add_argument("--fallback-lang", default="",
                    help="LANG for the browser process, e.g. zh_TW.UTF-8; picks the CJK "
                         "fallback face for text that has no lang= of its own")
    ap.add_argument("--check", default="", help="comma-separated CSS selectors to font-check")
    args = ap.parse_args()

    with sync_playwright() as p:
        env = {**os.environ, "LANG": args.fallback_lang} if args.fallback_lang else None
        browser = p.chromium.launch(env=env)
        context = browser.new_context(
            viewport={"width": args.width, "height": args.height},
            locale=args.locale,          # navigator.language / Accept-Language (not the fallback font)
            device_scale_factor=1,
        )
        page = context.new_page()
        t0 = time.time()
        resp = page.goto(args.url, wait_until="load", timeout=60_000)
        print("status:", resp.status if resp else None)

        if not args.no_scroll:
            info = page.evaluate(SCROLL_JS, {"step": 600, "pauseMs": 400,
                                             "maxSecs": 15, "maxHeight": args.max_height})
            print(f"scrolled: height={info['height']}px in {info['seconds']:.1f}s")
        page.evaluate(WAIT_ASSETS_JS, 10_000)
        print("images:", page.evaluate(IMG_STATS_JS))

        if args.check:
            for sel, fonts in platform_fonts(page, args.check.split(",")).items():
                print(f"font {sel}: {fonts}")

        page.screenshot(path=args.out, full_page=True, animations="disabled", timeout=120_000)
        print(f"saved {args.out} in {time.time() - t0:.1f}s")
        browser.close()


if __name__ == "__main__":
    main()

Install and run:

python3 -m venv .venv && . .venv/bin/activate
pip install playwright==1.64.0
playwright install chromium
python shoot.py https://example.com/ out.png --check "h1,p"

Problem 1: tofu boxes in a slim container

To simulate a slim image without having to build one, the demo gives Chromium a private fonts.conf that only lists one Latin font (DejaVu Sans), via the FONTCONFIG_FILE environment variable:

FONTCONFIG_FILE=$PWD/slim-fonts/fonts.conf \
  python shoot.py "$URL" a-slim-noscroll.png --no-scroll \
  --check "#title,#zh-Hant,#zh-Hans,#ja,#ko,#emoji"

== A: slim fonts, no scrolling ==
status: 200
images: {'images': 4, 'loaded': 2}
font #title: DejaVu Sans (15)
font #zh-Hant: DejaVu Sans (39)
font #zh-Hans: DejaVu Sans (21)
font #ja: DejaVu Sans (28)
font #ko: DejaVu Sans (23)
font #emoji: DejaVu Sans (25)
saved a-slim-noscroll.png in 0.2s

In the screenshot every CJK character is a box, and so is most of the emoji line. The two emoji DejaVu Sans happens to cover (😀 and ❤) come out as black-and-white outlines instead of colour emoji. Note what the font report says: the glyphs are still "counted" against DejaVu Sans, which has no CJK glyphs at all. So the check isn't "is there a font?", it's "is CJK text being drawn by a font that can't contain CJK?" If your Chinese paragraph reports only a Latin family, you have tofu, and you can fail the job before anyone looks at the PNG.

Fix: install CJK and emoji fonts in the image and rebuild the font cache.

RUN apt-get update && apt-get install -y --no-install-recommends \
      fonts-noto-cjk fonts-noto-color-emoji \
    && fc-cache -f && rm -rf /var/lib/apt/lists/*

With the normal system fonts:

== B: CJK + emoji fonts, no scrolling ==
status: 200
images: {'images': 4, 'loaded': 2}
font #title: Arimo (1), Noto Sans CJK JP (13), Noto Color Emoji (1)
font #zh-Hant: Arimo (11), Noto Sans CJK JP (28)
font #zh-Hans: Noto Sans CJK JP (21)
font #ja: Noto Sans CJK JP (28)
font #ko: Arimo (6), NanumGothic (17)
font #emoji: Arimo (12), Noto Color Emoji (6)
saved b-fonts-noscroll.png in 0.3s

No more boxes, and the emoji (including 👍🏽, 🇹🇼 and the ZWJ family) come from Noto Color Emoji. But look at the Chinese lines: both the Traditional and the Simplified paragraph were drawn with Noto Sans CJK JP. That's problem 3, which we'll come back to. First, the missing images.

Problem 2: lazy-loaded content is missing

Run B didn't scroll. The screenshot is 3,207 px tall: the two native lazy images near the top loaded, the two data-src images are grey boxes showing their alt text, and none of the three "infinite scroll" sections exist. full_page=True captures the whole document, but it doesn't scroll the way a visitor does, so IntersectionObserver callbacks for content further down never fire.

With the scroll step:

== C: CJK + emoji fonts, scroll first ==
status: 200
scrolled: height=5358px in 3.6s
images: {'images': 7, 'loaded': 7}
...
saved c-fonts-scroll.png in 4.0s

All 7 images loaded, and the page grew from 3,207 to 5,358 px as the three batches were appended. The loop doesn't stop at the first time it reaches the bottom: it waits until the height has stayed the same for two rounds, because infinite-scroll pages add content a moment after you arrive (the test page waits 300 ms).

Things that matter on real pages:

Always cap it. Some feeds never end. The script stops after 15 seconds or 20,000 px, whichever comes first. Pick limits that fit your pages.
Scroll back to the top before capturing, or sticky headers and "back to top" buttons end up in odd places.
networkidle is not a substitute. Lazy content only starts loading after the scroll, so an idle network before scrolling tells you nothing about it.
Disable animations (animations="disabled"), so carousels and fade-ins are captured in a stable state.

Problem 3: Chinese drawn with the Japanese font

Chinese, Japanese and Korean share a lot of code points (Han unification), but each region has its own preferred glyph shapes, and punctuation is placed differently: in Traditional Chinese fonts ， and 。 usually sit in the centre of the square, while Japanese fonts put them in the lower left. Noto Sans CJK ships TC, SC, HK, JP and KR versions of the same characters, and Chromium has to pick one.

In run B, Chromium picked JP for both lang="zh-Hant" and lang="zh-Hans". We then tried a few variations on the same setup (runs B and D to G in run_demo.sh):

What we changed	Font Chromium picked for the Chinese text
`lang="zh-Hant"` / `lang="zh-Hans"` (original page)	Noto Sans CJK JP for both
`lang="zh-TW"` / `lang="zh-CN"`	Noto Sans CJK TC / Noto Sans CJK SC
No `lang` on `<html>` (Traditional paragraph), browser started with the default `LANG=C.UTF-8`	Noto Sans CJK JP
No `lang` on `<html>` (Traditional paragraph), browser started with `LANG=zh_TW.UTF-8`	Noto Sans CJK TC
No `lang` on `<html>` (Traditional paragraph), Playwright `locale="zh-TW"` / `"ja-JP"` / `"en-US"`	Noto Sans CJK JP in all three
Explicit `font-family` per `:lang()`	the font you named

This was one Chromium build on one Linux font setup, so treat the table as "check yours" rather than a rule, but it shows two useful things. In this setup, script-only tags (zh-Hant, zh-Hans) didn't pick the Chinese faces, while region tags (zh-TW, zh-CN) did. And Playwright's locale option changes navigator.language and Accept-Language, but not the fallback font.

If you own the page, use region tags or name the faces explicitly. site/fixed.html does both:

:lang(zh-TW) { font-family: "Noto Sans CJK TC", "Noto Sans TC", "PingFang TC", "Microsoft JhengHei", sans-serif; }
:lang(zh-CN) { font-family: "Noto Sans CJK SC", "Noto Sans SC", "PingFang SC", "Microsoft YaHei", sans-serif; }
:lang(ja)    { font-family: "Noto Sans CJK JP", "Noto Sans JP", "Hiragino Sans", "Yu Gothic", sans-serif; }
:lang(ko)    { font-family: "Noto Sans CJK KR", "Noto Sans KR", "Apple SD Gothic Neo", "Malgun Gothic", sans-serif; }

== D: fixed page (lang=zh-TW / zh-CN + :lang() CSS), scroll first ==
status: 200
scrolled: height=5358px in 3.6s
images: {'images': 7, 'loaded': 7}
font #title: Noto Sans CJK TC (14), Noto Color Emoji (1)
font #zh-Hant: Noto Sans CJK TC (39)
font #zh-Hans: Noto Sans CJK SC (21)
font #ja: Noto Sans CJK JP (28)
font #ko: Noto Sans CJK KR (23)
font #emoji: Noto Sans CJK TC (12), Noto Color Emoji (6)
saved d-fixed-page.png in 3.9s

Each language now gets its own face. (The stack also lists the macOS and Windows names, so real visitors get the right fonts too.)

If you don't own the page and it has no lang attribute, the browser process's LANG decides which face fills in. shoot.py --fallback-lang zh_TW.UTF-8 passes that to chromium.launch(env=...):

== E1: page without lang=, browser started with default LANG ==
font #zh-Hant: Arimo (11), Noto Sans CJK JP (28)
== E2: page without lang=, browser started with LANG=zh_TW.UTF-8 ==
font #zh-Hant: Arimo (11), Noto Sans CJK TC (28)

This only helps text that has no lang of its own. A page that declares lang="zh-Hant" still got JP in our test, and we deliberately don't inject CSS into pages we don't own: for monitoring and archiving, the screenshot should show the page as its authors shipped it.

Checklist

[ ] Install fonts-noto-cjk and fonts-noto-color-emoji (or equivalents) and run fc-cache -f
[ ] Font-check a few CJK elements with CSS.getPlatformFontsForNode; fail the job if CJK text is drawn only by a Latin font
[ ] For your own pages: use zh-TW / zh-CN / zh-HK region tags or :lang() font stacks
[ ] For pages without lang: start the browser with the LANG of the region you expect
[ ] Scroll in steps until the height stops growing, with a time and height cap, then scroll back to the top
[ ] Wait for pending images and document.fonts.ready; capture with animations="disabled"
[ ] Respect robots.txt and the site's terms, and screenshot for QA, monitoring or archiving, not to republish other people's content

This article was written with AI assistance. The code was run and the output above is copied from that run.

How to Build a B2B Contact List from Company Websites Responsibly

Raspberry Rondel — Sat, 17 Oct 2026 08:38:06 +0000

You have a list of target companies from a trade-show catalogue or a CRM export, and you need each one's general business contact: the sales inbox, the switchboard, the official LINE account. The manual way is to open every homepage, find "Contact" or "關於我們", copy what's there into a spreadsheet, and repeat. It's slow, and you end up with phone numbers in five formats and no record of where anything came from.

This post shows how to automate that responsibly: follow robots.txt, read only a few pages per site, collect only public business contact info, and keep the source of every value. A runnable Python example is included.

1. Decide what you are collecting (and what you are not)

The target is company contact points: role-based addresses (info@, sales@), main lines, fax and toll-free numbers, and the company's own Facebook page and LINE official account. A named employee's email or a sole trader's mobile is personal data under GDPR and Taiwan's PDPA, even when it's on a public page.

Minimise. Keep only the columns you'll use.
Keep provenance. Store the URL each value came from, so you can answer "where did you get my address?" and re-check stale data.

2. Crawl politely: robots.txt and Crawl-delay

Before you fetch anything, read /robots.txt and follow it (RFC 9309):

Skip every path that is Disallowed for your user agent.
Honour Crawl-delay if it's set. urllib.robotparser exposes it as crawl_delay(). Without it, still wait at least a second per host.
If robots.txt returns a 5xx error, don't crawl that site. RFC 9309 says to treat a server error as "complete disallow". A 404 means there are no rules.
Send an honest User-Agent that says who you are. Don't log in, don't rotate proxies to look like someone else, and stop if you hit a CAPTCHA or bot challenge. A site that blocks you has answered your question.

3. Find the contact and about pages

Contact details usually sit in the homepage header or footer, a contact page and an about page, so you don't need a full crawl. Fetch the homepage, pick same-site links whose URL or anchor text matches contact, about, 聯絡, 關於 or お問い合わせ, and visit a few. About four pages per domain covers most small company sites. If nothing matches, try /contact and /about directly.

4. Extract emails, phones, LINE and social links

Emails. Take mailto: links first, then scan visible text, and filter obvious junk: asset names (banner@2x.png) and placeholders (name@domain.com). If a site deliberately obfuscates its address, respect that and don't decode it.

Phones. Regexes alone will match dates, order numbers and prices. Two rules help a lot:

Require context. Only parse a phone number from a tel: link or a line that has a label nearby: Tel, Phone, 電話, 客服, 傳真, Fax.
Let libphonenumber do the parsing. The phonenumbers package (a Python port of Google's libphonenumber) understands Taiwanese formats such as (02)2345-6789, 02-2345-6789, +886 2 2345 6789 and 0800-000-123. It normalises all of them to E.164 (+886223456789), which is what makes de-duplication work.

In Taiwan, capture extensions (分機 123, 轉 123) separately, and strip the 8-digit tax ID (統一編號) that often sits next to the number in the footer.

LINE. Businesses in Taiwan and Japan often publish a LINE official account instead of an email. Look for line.me/R/ti/p/@account, lin.ee short links and page.line.me.

Social profiles. Keep the company's own profiles and drop share buttons (/intent/tweet, /sharer).

5. De-duplicate with provenance

The same number often appears on three pages in three formats. Key phones by E.164 (plus extension) and emails by their lower-cased form, and keep a list of source URLs per key: one record per domain, with an audit trail.

6. A runnable example

The script below puts sections 2–5 together using only the standard library plus phonenumbers. To test it safely, I ran it against a fictional company site served from my own machine. All addresses use example.com, and the numbers are made up. Don't point a test script at real third-party sites.

The test site has a robots.txt that blocks /private/ and sets a crawl delay:

User-agent: *
Disallow: /private/
Crawl-delay: 1

The homepage links to an about page, a contact page and a page under /private/. Its footer puts a phone number and a tax ID on the same line:

<!doctype html><html lang="zh-Hant"><head><meta charset="utf-8"><title>Example Trading Co.</title></head>
<body>
<header><a href="/">Home</a> <a href="/products.html">Products</a> <a href="/about-us.html">關於我們</a> <a href="/contact.html">聯絡我們</a></header>
<p>Fictional demo company.</p>
<img src="/img/logo@2x.png" alt="logo">
<a href="https://twitter.com/intent/tweet?url=https://example.com">Share on X</a>
<footer>
  <a href="https://www.facebook.com/exampletrading">Facebook</a>
  <a href="https://lin.ee/AbCdEfG">加入 LINE 好友</a>
  <a href="/private/contact-staff.html">Staff only</a>
  電話 (02)2345-6789｜統一編號 12345678
</footer>
</body></html>

The contact page mixes contacts with things that must not be extracted (an order number, a date, a placeholder email, an image name):

<!doctype html><html lang="zh-Hant"><head><meta charset="utf-8"><title>聯絡我們</title></head>
<body>
<h1>聯絡我們</h1>
<p>業務洽詢：<a href="mailto:sales@example.com">sales@example.com</a></p>
<p>客服信箱：info@example.com</p>
<p>電話：<a href="tel:+886223456789">(02)2345-6789</a> 分機 123</p>
<p>傳真：02-2345-6790</p>
<p>免付費客服：0800-000-123</p>
<p>Order no. 2026101700123, updated 2026-10-17</p>
<p>Example form text: name@domain.com (image: banner@2x.png)</p>
<p>LINE 官方帳號：<a href="https://line.me/R/ti/p/@exampletrading">@exampletrading</a></p>
<footer><a href="https://www.facebook.com/exampletrading">Facebook</a></footer>
</body></html>

The extractor, extract_contacts.py (pip install phonenumbers):

"""Polite public-business-contact extractor (tutorial demo).

Reads a site's robots.txt, honours Disallow and Crawl-delay, visits the homepage
plus up to 3 contact/about pages, and returns one de-duplicated record per domain.
Only run it on sites you are allowed to crawl. Demo data is fictional.
"""
import json, re, sys, time, urllib.error, urllib.request, urllib.robotparser
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit
import phonenumbers

UA = "ContactListTutorialBot/0.1 (+https://example.com/bot)"
MAX_PAGES = 4
PAGE_HINTS = re.compile(r"contact|about|聯絡|聯繫|關於|お問い合わせ", re.I)
EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,24}")
JUNK_EMAIL = re.compile(r"\.(png|jpe?g|gif|svg|webp)$|^(you|name|user|email)@(example|domain)\.", re.I)
PHONE_CONTEXT = re.compile(r"tel|phone|電話|客服|傳真|fax|免付費", re.I)
FAX = re.compile(r"傳真|fax", re.I)
EXT = re.compile(r"^\s*(?:分機|轉|ext\.?|#)\s*(\d{1,6})", re.I)
LINE = re.compile(r"https?://(?:lin\.ee|line\.me|page\.line\.me)/[^\s\"'<>]+", re.I)
TAX_ID = re.compile(r"統一編號\s*[:：]?\s*\d{8}")
SHARE = re.compile(r"/(?:intent|share|sharer)", re.I)
SOCIAL = re.compile(r"https?://(?:www\.)?(facebook|instagram|linkedin|x|twitter|youtube)\.com/[^\s\"'<>]+", re.I)


class Page(HTMLParser):
    """Collects visible text (one line per block) and link targets."""
    def __init__(self):
        super().__init__(); self.links, self.lines, self._buf = [], [], []
    def handle_starttag(self, tag, attrs):
        href = dict(attrs).get("href")
        if href: self.links.append(href)
        if tag in ("p", "br", "li", "div", "footer", "header", "h1", "h2", "tr"): self._flush()
    def handle_data(self, data): self._buf.append(data)
    def _flush(self):
        line = " ".join("".join(self._buf).split())
        if line: self.lines.append(line)
        self._buf = []


def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=10) as r:
        return r.status, r.read().decode("utf-8", "replace")


def load_robots(base):
    rp = urllib.robotparser.RobotFileParser()
    try:
        _, body = fetch(base + "/robots.txt")
        rp.parse(body.splitlines())
    except urllib.error.HTTPError as e:
        if e.code >= 500:
            return None          # server error: treat as "do not crawl"
        rp.parse([])             # 404 etc.: no rules, everything allowed
    return rp


def extract(start_url, region="TW"):
    parts = urlsplit(start_url); base = f"{parts.scheme}://{parts.netloc}"
    record = {"domain": parts.hostname, "status": "ok", "pages": [], "skippedByRobots": [],
              "emails": {}, "phones": {}, "line": {}, "socials": {}, "junkFiltered": 0}
    rp = load_robots(base)
    if rp is None:
        record["status"] = "robots-unavailable"; return record
    delay = rp.crawl_delay(UA) or 1.0           # at least 1 s between requests
    queue, seen = [base + "/"], set()
    while queue and len(record["pages"]) < MAX_PAGES:
        url = queue.pop(0)
        if url in seen: continue
        seen.add(url)
        if not rp.can_fetch(UA, url):
            record["skippedByRobots"].append(url); continue
        if record["pages"]: time.sleep(delay)
        try:
            status, html = fetch(url)
        except urllib.error.URLError as e:
            record["pages"].append({"url": url, "error": str(e)}); continue
        record["pages"].append({"url": url, "http": status})
        page = Page(); page.feed(html); page._flush()
        collect(page, url, region, record)
        if url == base + "/":                    # discover contact/about pages from the homepage
            for href in page.links:
                full = urljoin(url, href).split("#")[0]
                if urlsplit(full).netloc == parts.netloc and PAGE_HINTS.search(href):
                    queue.append(full)
    if not (record["emails"] or record["phones"] or record["line"] or record["socials"]):
        record["status"] = "no-contacts-found"
    for key in ("emails", "phones", "line", "socials"):    # sets -> sorted lists for JSON
        record[key] = {k: sorted(v) if isinstance(v, set) else v for k, v in record[key].items()}
    return record


def add(bucket, key, source):
    bucket.setdefault(key, set()).add(source)


def collect(page, url, region, rec):
    for href in page.links:
        if href.startswith("mailto:"):
            add(rec["emails"], href[7:].split("?")[0].lower(), url)
        elif LINE.match(href):
            add(rec["line"], href, url)
        elif SOCIAL.match(href) and not SHARE.search(href):
            add(rec["socials"], href.rstrip("/"), url)
    for line in page.lines:
        for e in EMAIL.findall(line):
            if not JUNK_EMAIL.search(e):
                add(rec["emails"], e.lower(), url)
            else:
                rec["junkFiltered"] += 1
        line = TAX_ID.sub(" ", line)                # 統一編號 (8-digit tax ID) is not a phone
        if not PHONE_CONTEXT.search(line):
            continue                                # phones need context (Tel, 電話, 傳真 ...)
        for m in phonenumbers.PhoneNumberMatcher(line, region):
            e164 = phonenumbers.format_number(m.number, phonenumbers.PhoneNumberFormat.E164)
            ext = EXT.match(line[m.end:])
            key = e164 + (f";ext={ext.group(1)}" if ext else "")
            entry = rec["phones"].setdefault(key, {"kind": "fax" if FAX.search(line) else "phone", "sources": []})
            if url not in entry["sources"]: entry["sources"].append(url)


if __name__ == "__main__":
    print(json.dumps(extract(sys.argv[1]), ensure_ascii=False, indent=2))

Serve the site and run it:

python3 -m http.server 8000 --bind 127.0.0.1 --directory site &
python3 extract_contacts.py http://127.0.0.1:8000/

Real output (the demo script run_demo.sh picks a free port and appends the server log):

{
  "domain": "127.0.0.1",
  "status": "ok",
  "pages": [
    {
      "url": "http://127.0.0.1:48023/",
      "http": 200
    },
    {
      "url": "http://127.0.0.1:48023/about-us.html",
      "http": 200
    },
    {
      "url": "http://127.0.0.1:48023/contact.html",
      "http": 200
    }
  ],
  "skippedByRobots": [
    "http://127.0.0.1:48023/private/contact-staff.html"
  ],
  "emails": {
    "info@example.com": [
      "http://127.0.0.1:48023/about-us.html",
      "http://127.0.0.1:48023/contact.html"
    ],
    "sales@example.com": [
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "phones": {
    "+886223456789": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/",
        "http://127.0.0.1:48023/about-us.html"
      ]
    },
    "+886223456789;ext=123": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    },
    "+886223456790": {
      "kind": "fax",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    },
    "+886800000123": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    }
  },
  "line": {
    "https://lin.ee/AbCdEfG": [
      "http://127.0.0.1:48023/"
    ],
    "https://line.me/R/ti/p/@exampletrading": [
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "socials": {
    "https://www.facebook.com/exampletrading": [
      "http://127.0.0.1:48023/",
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "junkFiltered": 2
}
--- server access log (shows robots.txt first and the Crawl-delay spacing) ---
[17/Oct/2026 16:31:02] "GET /robots.txt HTTP/1.1" 200 -
[17/Oct/2026 16:31:02] "GET / HTTP/1.1" 200 -
[17/Oct/2026 16:31:03] "GET /about-us.html HTTP/1.1" 200 -
[17/Oct/2026 16:31:04] "GET /contact.html HTTP/1.1" 200 -

What to notice:

robots.txt was fetched first, and /private/contact-staff.html was skipped even though it matched the "contact" hint.
Requests are about one second apart, as Crawl-delay: 1 asks.
(02)2345-6789 and +886 2 2345 6789 became one E.164 key with two sources. The 分機 123 version kept its own key, and the fax line was labelled fax.
The tax ID, order number, date, placeholder email and image file name produced no contacts (junkFiltered: 2 counts the two email-like strings).
The share-on-X link was dropped, and both LINE link styles (lin.ee and line.me/R/ti/p/@…) were kept.

It's a teaching script. It reads phones from labelled text (not tel: links), doesn't try /contact when nothing matches, and has no JavaScript rendering, retries or parallelism.

7. GDPR and PDPA: your responsibilities

This section is general information, not legal advice.

You decide which sites to read and how to use the results, so compliance is your responsibility, not the tool's.
GDPR (EU/EEA and UK). You need a lawful basis, usually legitimate interest (Art. 6(1)(f)) backed by a documented balancing test. If any personal data ends up in the list, you have to tell those people within a reasonable time (Art. 14) and honour objections (Art. 21). National ePrivacy rules often require prior consent before you send marketing email to individuals.
Taiwan PDPA (個人資料保護法). Data the person made public themselves, or that was lawfully made public, may be collected, but only for a specific purpose (Art. 19). The notice obligation can be waived for such data (Art. 9). If you use it for marketing, you must offer a way to refuse at first contact and stop as soon as the person objects (Art. 20).
In practice: prefer role-based addresses, keep source URLs, honour "no solicitation" notices, offer an opt-out in every first message, delete what you don't need, and never build profiles of individuals.

Checklist

[ ] Company contact points only
[ ] Fetch robots.txt first; skip disallowed paths; 5xx means don't crawl
[ ] Crawl-delay (≥1 s per host), honest User-Agent, no logins or proxies, stop at challenges
[ ] Homepage plus 3 contact/about pages per domain
[ ] Phones only with context; normalise to E.164; keep extensions; strip 統一編號
[ ] Filter junk emails; don't decode obfuscated ones
[ ] Keep LINE official account links; drop share buttons
[ ] De-duplicate by normalised key and keep source URLs
[ ] Document lawful basis and purpose; offer an opt-out

This article was written with AI assistance. The code was run and the output above is copied from that run.

PDF to Markdown for RAG: Fix Reading Order, Tables and Headings Before You Chunk

Raspberry Rondel — Sat, 17 Oct 2026 06:34:09 +0000

If your RAG pipeline answers questions about PDFs badly, look at the chunks before you tune the embeddings or the prompt. A lot of retrieval problems start at extraction: two-column pages read straight across, tables turned into a stream of numbers, and headings that become ordinary lines of text. Then you spend an afternoon cleaning text by hand before you can even chunk it.

This post covers what goes wrong, why converting to Markdown (not plain text) fixes most of it, how the common open-source libraries compare, and a small Python script you can run locally to see the difference.

Three things plain-text extraction breaks

1. Reading order in two-column layouts. Most extractors work line by line, top to bottom. On a two-column paper, line 1 of the left column is followed by line 1 of the right column. Every chunk then mixes two unrelated passages, and the embedding ends up halfway between them.

2. Tables. A table flattened into text keeps the values and loses the structure. "Q1 Q2 Q3 / 12 15 9" no longer tells the model which number belongs to which column. A Markdown table (| Q1 | Q2 | Q3 |) keeps the header attached to each row, and LLMs read that format well.

3. Headings. Headings are the best chunk boundaries you have, and they're also useful metadata ("this chunk is from 3.2 Limitations"). In plain text they look like any other line. In Markdown they're #/## lines, so a splitter such as LangChain's MarkdownHeaderTextSplitter or a 10-line function can cut on them.

CJK documents add one more problem: justified Chinese text is often stored with a space between every character, and line breaks land mid-sentence. Both get in the way of keyword search and chunk-size math.

The options, briefly

All of these are real, maintained projects. Check licenses yourself before you ship; the ones below are from PyPI/GitHub metadata as of October 2026.

Library	Approach	Good at	Watch out for	License
pypdf	Pure-Python text extraction	Zero dependencies, fast	No layout, tables or headings	BSD-3
pdfminer.six	Layout analysis (chars → lines → boxes)	Fine-grained positions, fonts, sizes	You build structure yourself	MIT
pdfplumber	Built on pdfminer.six	Ruled-table detection, easy word/char access	Borderless tables need tuning	MIT
PyMuPDF / pymupdf4llm	MuPDF engine, Markdown output	Fast; Markdown output ready to use	AGPL-3.0 or commercial license	AGPL / commercial
Docling (IBM)	ML layout + table models	Complex layouts, many formats	Heavier install, model downloads	MIT
Marker	ML pipeline, optional LLM	Hard layouts, equations	GPU helps; check model-weight terms	Apache-2.0 (code)
MinerU	ML pipeline, OCR	Scanned docs, Chinese	Heavy; custom license	MinerU license
unstructured	Element partitioning	Many file types, element metadata	Hi-res mode pulls in a lot	Apache-2.0
MarkItDown (Microsoft)	Converts many formats to Markdown	Office files, quick conversions	PDF handling is lighter than PDF-specific tools (uses pdfminer/pdfplumber)	MIT

A rough rule of thumb: if your PDFs have a text layer (born-digital reports, papers, government documents), a layout-aware rules approach on pdfminer.six/pdfplumber is light, deterministic and cheap to run. If they're scanned or very visual, you need OCR or an ML pipeline (Docling, Marker, MinerU), and you pay for it in compute.

A runnable example: two columns + a table → Markdown → chunks

We'll generate a small sample PDF so you don't need any files, then compare extract_text() with a ~90-line converter.

python -m venv .venv && source .venv/bin/activate
pip install pdfplumber==0.11.10 fpdf2==2.8.9

make_sample.py builds a page with a title, two columns, and a ruled table:

# make_sample.py - build a small two-column PDF with headings and a ruled table
from fpdf import FPDF

LEFT = ("Retrieval quality depends on chunk boundaries. When one chunk mixes "
        "two sections, its embedding blurs both topics. Splitting at headings "
        "keeps each chunk about one thing.")
RIGHT = ("Tables are worse. Flattened into plain text, a row of values "
         "loses its column headers, so the model cannot tell which number "
         "belongs to which metric.")

pdf = FPDF(format="A4")
pdf.add_page()
pdf.set_font("Helvetica", "B", 20)
pdf.cell(0, 12, "Chunking PDFs for RAG", new_x="LMARGIN", new_y="NEXT")
y = pdf.get_y() + 4
for x, title, body in [(10, "Why boundaries matter", LEFT), (110, "Why tables break", RIGHT)]:
    pdf.set_xy(x, y)
    pdf.set_font("Helvetica", "B", 14)
    pdf.multi_cell(90, 8, title)
    pdf.set_x(x)
    pdf.set_font("Helvetica", "", 10)
    pdf.multi_cell(90, 5, body)
pdf.set_y(150)
pdf.set_font("Helvetica", "B", 14)
pdf.cell(0, 8, "Formats compared", new_x="LMARGIN", new_y="NEXT")
pdf.set_font("Helvetica", "", 10)
with pdf.table(col_widths=(60, 40, 40)) as table:  # fpdf2 draws ruled borders
    for row in [("Output", "Headings", "Tables"), ("Plain text", "lost", "flattened"), ("Markdown", "# / ##", "pipe table")]:
        r = table.row()
        for cell in row:
            r.cell(cell)
pdf.output("sample.pdf")
print("wrote sample.pdf")

pdf_to_md.py does four things: groups words into line fragments (splitting at the column gutter), orders them with a recursive XY-cut (split at a vertical gutter if one exists, otherwise at the widest horizontal gap), marks headings by font size relative to the page's median size, and swaps ruled tables for Markdown tables using pdfplumber's find_tables().

# pdf_to_md.py - minimal text-layer PDF -> Markdown -> heading-aware chunks
import statistics, sys
import pdfplumber

def lines_of(words, gap=12):
    """Group words into line fragments; a wide horizontal gap (a column gutter) starts a new fragment."""
    rows = {}
    for w in words:
        rows.setdefault(round(w["top"]), []).append(w)
    frags = []
    for ws in rows.values():
        ws.sort(key=lambda w: w["x0"])
        cur = [ws[0]]
        for w in ws[1:]:
            if w["x0"] - cur[-1]["x1"] > gap:
                frags.append(cur); cur = []
            cur.append(w)
        frags.append(cur)
    return [{"x0": f[0]["x0"], "x1": f[-1]["x1"], "top": min(w["top"] for w in f),
             "bottom": max(w["bottom"] for w in f), "size": f[0]["size"],
             "text": " ".join(w["text"] for w in f)} for f in frags]

def gaps(intervals):
    intervals = sorted(intervals); out = []; end = intervals[0][1]
    for a, b in intervals[1:]:
        if a > end:
            out.append((end, a))
        end = max(end, b)
    return out

def xy_cut(items, min_gutter=7):
    """Recursive XY-cut: split at a column gutter if there is one, else at the widest horizontal gap."""
    if len(items) <= 1:
        return items
    width = max(i["x1"] for i in items) - min(i["x0"] for i in items)
    cols = [g for g in gaps([(i["x0"], i["x1"]) for i in items]) if g[1] - g[0] >= min_gutter]
    for a, b in sorted(cols, key=lambda g: g[0] - g[1]):          # widest gutter first
        left = [i for i in items if i["x1"] <= a]; right = [i for i in items if i["x0"] >= b]
        if min(max(i["x1"] for i in left) - min(i["x0"] for i in left),
               max(i["x1"] for i in right) - min(i["x0"] for i in right)) > 0.2 * width:
            return xy_cut(left) + xy_cut(right)
    rows = gaps([(i["top"], i["bottom"]) for i in items])
    if not rows:
        return sorted(items, key=lambda i: (i["top"], i["x0"]))
    a, b = max(rows, key=lambda g: g[1] - g[0])
    return xy_cut([i for i in items if i["bottom"] <= a]) + xy_cut([i for i in items if i["top"] >= b])

def page_to_markdown(page):
    tables = [t for t in page.find_tables() if len(t.extract()) >= 2]   # ruled tables only
    def in_table(w):
        return any(t.bbox[0] <= w["x0"] and w["x1"] <= t.bbox[2] and
                   t.bbox[1] <= w["top"] and w["bottom"] <= t.bbox[3] for t in tables)
    words = [w for w in page.extract_words(extra_attrs=["size"]) if not in_table(w)]
    body = statistics.median(w["size"] for w in words)
    items = lines_of(words)
    for t in tables:
        rows = [[(c or "").replace("\n", " ").strip() for c in r] for r in t.extract()]
        md = ["| " + " | ".join(rows[0]) + " |", "|" + "---|" * len(rows[0])]
        md += ["| " + " | ".join(r) + " |" for r in rows[1:]]
        x0, top, x1, bottom = t.bbox
        items.append({"x0": x0, "x1": x1, "top": top, "bottom": bottom, "size": body,
                      "text": "\n".join(md), "table": True})
    out = []
    for it in xy_cut(items):
        if it.get("table"):
            out.append(it["text"])
        elif it["size"] >= body * 1.8:
            out.append("# " + it["text"])
        elif it["size"] >= body * 1.3:
            out.append("## " + it["text"])
        elif out and not out[-1].startswith(("#", "|")):
            out[-1] += " " + it["text"]                  # re-join wrapped lines into one paragraph
        else:
            out.append(it["text"])
    return "\n\n".join(out)

def chunk_by_heading(md):
    chunks, current = [], []
    for block in md.split("\n\n"):
        if block.startswith("#") and current:
            chunks.append("\n\n".join(current)); current = []
        current.append(block)
    return chunks + ["\n\n".join(current)]

if __name__ == "__main__":
    with pdfplumber.open(sys.argv[1] if len(sys.argv) > 1 else "sample.pdf") as pdf:
        page = pdf.pages[0]
        print("=== naive extract_text() ===")
        print(page.extract_text()[:240], "...\n")
        md = page_to_markdown(page)
        print("=== markdown ===")
        print(md, "\n")
        for i, c in enumerate(chunk_by_heading(md)):
            print(f"--- chunk {i}: {c.splitlines()[0]!r} ({len(c)} chars)")

Run it:

python make_sample.py && python pdf_to_md.py

Here's what extract_text() gives you. Look at lines 3-5: each one runs straight across both columns.

Chunking PDFs for RAG
Why boundaries matter Why tables break
Retrieval quality depends on chunk boundaries. When Tables are worse. Flattened into plain text, a row of
one chunk mixes two sections, its embedding blurs both values loses its column headers, so the model cannot
topics. Splitting at headings keeps each chunk about tell which number belongs to which metric.
one thing.
Formats compared
Output Headings Tables
Plain text lost flattened
Markdown # / ## pipe table

And here's the converter output (tested with Python 3.13, pdfplumber 0.11.10, fpdf2 2.8.9):

# Chunking PDFs for RAG

## Why boundaries matter

Retrieval quality depends on chunk boundaries. When one chunk mixes two sections, its embedding blurs both topics. Splitting at headings keeps each chunk about one thing.

## Why tables break

Tables are worse. Flattened into plain text, a row of values loses its column headers, so the model cannot tell which number belongs to which metric.

## Formats compared

| Output | Headings | Tables |
|---|---|---|
| Plain text | lost | flattened |
| Markdown | # / ## | pipe table |

--- chunk 0: '# Chunking PDFs for RAG' (23 chars)
--- chunk 1: '## Why boundaries matter' (196 chars)
--- chunk 2: '## Why tables break' (170 chars)
--- chunk 3: '## Formats compared' (134 chars)

Each column comes out as one paragraph under its own heading, the table keeps its header row, and the chunker splits on headings. In a real pipeline you'd merge tiny chunks (like the title-only chunk 0) into the next one and add the heading path as metadata.

Where this toy version breaks

It works on this page, but real documents will break it quickly:

Paragraphs: it joins every non-heading line until the next heading. Real code needs to detect paragraph breaks (indents, vertical spacing) and hyphenation.
Borderless "three-line" academic tables aren't detected by find_tables() with default settings.
Rotated margin text (arXiv side stamps), headers/footers and page numbers end up in the body.
Symbol fonts map glyphs into Unicode's Private Use Area, and some fonts give (cid:NN) garbage. You want to detect those pages, not embed them.
Scanned pages have no text layer at all, and you should flag them rather than index empty strings.

Wrapping up

For born-digital PDFs, most of the RAG quality you lose at extraction comes back once you keep reading order, headings and tables. Start with a small rules-based converter like the one above, look at your chunks, and only move to a heavier ML pipeline when your documents actually need it.

This article was written with AI assistance. The code was run and the output above is copied from that run.

How to Compare a Long Keyword List in Google Trends on One 0-100 Scale

Raspberry Rondel — Sat, 17 Oct 2026 06:24:51 +0000

If you've ever tried to chart 12 keywords from Google Trends, you've hit the wall: one comparison only holds a handful of terms (the current Explore page allows up to 8; the older compare box allowed 5). The obvious workaround (run several smaller requests and paste the results together) gives you numbers that look right but mean nothing when you put them side by side.

This post explains why, and walks through the anchor-term rescaling method that fixes it, with a small pandas implementation you can run offline.

Why separate requests aren't comparable

Google Trends doesn't return search volume. It returns a relative index: within a single request, the highest point across all terms in that request becomes 100, and everything else is scaled against it and rounded to an integer.

So "60" in request A and "60" in request B are on different scales. If request A contains a giant term, everything else in A gets squashed toward zero. If request B contains only small terms, one of them gets stretched to 100. Concatenating the two is like merging two charts that use different y-axes without telling anyone.

The fix: a shared anchor term

The trick is to put the same term (the anchor) in every request. Because the anchor's real interest is the same no matter which request it's in, any difference in its numbers between requests comes from the different scaling. That gives you a conversion factor.

The procedure:

Pick an anchor term.
Split your other keywords into groups small enough that each group plus the anchor fits in one request (groups of 4 plus the anchor, 5 terms per request, work under both the old and the new limit).
Fetch each group with identical geo, timeframe, and category.
Pick a reference group (e.g. the first one where the anchor isn't all zeros).
For each group g, compute a factor:

   factor_g = sum(anchor in reference group) / sum(anchor in group g)

Multiply every series in group g by factor_g. Report the anchor once, from the reference group.
Renormalize globally: divide everything by the overall max and multiply by 100, so the biggest point across all keywords is 100 again.

Why sums instead of per-point ratios?

You could compute a ratio for each date and average them, but each data point is an integer from 0 to 100. A point where the anchor reads 2 could really be anywhere from 1.5 to 2.5. That's ±25% on one point. Summing the whole series first and dividing once spreads that rounding error across the full timeline. Points where the anchor reads 0 would also break a per-point ratio (division by zero), but they're harmless in a sum.

A runnable implementation in pandas

The code below has no network calls. The DataFrames are made-up example data so you can see the math work. In real use you'd fill them with whatever your fetcher returns (one DataFrame per request, indexed by date).

import pandas as pd

LOW_ANCHOR_MEAN = 5  # below this, integer rounding error gets amplified


def rescale_groups(groups: list[pd.DataFrame], anchor: str):
    """Put several Google Trends requests (each with its own 0-100 scale)
    onto one shared 0-100 scale using an anchor term present in every group.

    Returns (combined DataFrame, factors dict, warnings list).
    """
    warnings, factors, scaled = [], {}, []

    # Reference = first group where the anchor has any signal at all.
    ref_idx = next((i for i, g in enumerate(groups) if g[anchor].sum() > 0), None)
    if ref_idx is None:
        raise ValueError("anchor is zero in every group; cannot rescale")
    ref = groups[ref_idx]
    ref_sum = ref[anchor].sum()

    for i, g in enumerate(groups):
        # Groups must share the exact same date axis to be comparable.
        if not g.index.equals(ref.index):
            warnings.append(f"group {i}: date axis differs from reference; skipped")
            continue
        s = g[anchor].sum()
        if s == 0:
            warnings.append(f"group {i}: anchor is all zeros; skipped")
            continue
        mean = s / len(g)
        if mean < LOW_ANCHOR_MEAN:
            warnings.append(
                f"group {i}: anchor mean {mean:.2f} < {LOW_ANCHOR_MEAN}; "
                "rounding error is amplified, consider a more popular anchor"
            )
        factors[i] = float(ref_sum / s)
        cols = g.columns if i == ref_idx else g.columns.drop(anchor)
        scaled.append(g[cols] * factors[i])

    combined = pd.concat(scaled, axis=1)
    peak = combined.to_numpy().max()
    combined = (combined * (100 / peak if peak > 0 else 0)).round(2)
    return combined, factors, warnings


# ---- EXAMPLE DATA (made up for illustration, not real Google Trends values) ----
dates = pd.date_range("2026-01-04", periods=6, freq="W")

# Request 1: anchor + two big terms. "term_a" sets this request's 100.
group0 = pd.DataFrame(
    {"anchor": [20, 22, 21, 19, 23, 20],
     "term_a": [80, 90, 100, 85, 95, 88],
     "term_b": [40, 42, 45, 41, 39, 44]},
    index=dates,
)

# Request 2: anchor + two small terms. With no giant term here,
# the anchor itself is stretched toward 100.
group1 = pd.DataFrame(
    {"anchor": [87, 96, 91, 83, 100, 87],
     "term_c": [30, 35, 33, 31, 36, 34],
     "term_d": [10, 12, 11, 9, 13, 10]},
    index=dates,
)

combined, factors, warnings = rescale_groups([group0, group1], anchor="anchor")

print("factors:", {k: round(v, 4) for k, v in factors.items()})
print("warnings:", warnings or "none")
print(combined)

Running it (Python 3, pandas) prints:

factors: {0: 1.0, 1: 0.2298}
warnings: none
            anchor  term_a  term_b  term_c  term_d
2026-01-04    20.0    80.0    40.0    6.89    2.30
2026-01-11    22.0    90.0    42.0    8.04    2.76
2026-01-18    21.0   100.0    45.0    7.58    2.53
2026-01-25    19.0    85.0    41.0    7.12    2.07
2026-02-01    23.0    95.0    39.0    8.27    2.99
2026-02-08    20.0    88.0    44.0    7.81    2.30

In raw request 2, term_c peaked at 36 and looked like it might be close to term_b. Once it's on the shared scale you can see it's really around 7-8, a small fraction of term_b. That's exactly the kind of wrong conclusion that naive concatenation produces.

Note the output has decimals. That's expected: rescaled values are estimates, not Google's native integers.

Pitfalls (the part that actually bites)

1. An anchor with too little volume

This is the big one. If your anchor averages, say, 3 in a group, each point's ±0.5 rounding is a huge relative error, and the factor inherits it. Every keyword in that group is then multiplied by a noisy factor. If the anchor rounds to 0 across a whole group, you can't rescale that group at all.

In the implementation I base this on, a group whose anchor mean is below 5 triggers a warning, and a group whose anchor is all zeros is marked as failed instead of guessed. I'd recommend both behaviors: flag it, don't invent numbers.

2. Choosing the anchor

A good anchor is:

Steady: no huge spikes, so it isn't the 100 in some groups and tiny in others for reasons unrelated to scaling.
Fairly popular in your geo and timeframe, so it stays well above zero in every group.
Not dwarfed by your biggest keyword. If one keyword is 50× the anchor, the anchor gets crushed toward 0 in that keyword's group.

A practical check: after a run, look at the anchor's mean per group and at the factors. Factors far from 1 combined with a low anchor mean are where the error lives.

3. Rounding error is real, even when everything is done right

Rescaling can't recover precision that Google rounded away. Low-volume keywords in a group with a large factor carry roughly ±0.5 × factor of uncertainty per point. Treat small differences between small keywords with suspicion.

4. Sampling noise

Google Trends is computed from a sample of searches, and recent data gets revised (the latest point is often flagged as partial). Two requests made at different times can return slightly different values for the same term. Fetch all your groups in one session, close together in time, and consider dropping partial points before comparing.

5. Timeframe, geo, and category must match exactly

The anchor only links groups if every request uses the same geo, timeframe, and category. Watch out for the date axis too: if one group comes back with a different set of dates, don't try to line it up. Exclude it. The implementation above (and the one I based it on) requires an identical date axis and skips any group that doesn't match.

6. Rate limits and HTTP 429

If you're using pytrends, you will meet TooManyRequestsError / HTTP 429 sooner or later, especially in loops. What helps:

Send requests one at a time with a delay between them (seconds, not milliseconds).
Retry only on 429 / 5xx / timeouts, with exponential backoff plus jitter, and honor Retry-After if present.
Don't retry a 400: that's a bad request and retrying won't fix it.
Cap the number of retries, and record a failed keyword as failed rather than silently dropping or zero-filling it.

7. Browser scrapers and UI changes

Google has redesigned the Trends pages more than once. Any scraper that drives a browser and reads values out of the rendered page depends on that page's layout and selectors, so a redesign is exactly the kind of change that breaks it. The approach I've had better luck with is to skip the DOM entirely and use the JSON endpoints that the Trends UI itself calls (the same data the "download CSV" button gives you). These are still unofficial and can change without notice, so build in validation either way.

A quick validation you should run

Before you trust a pipeline, compare it to ground truth: pick 5 terms, fetch them in one direct request (Google's own scale), then fetch them again split into anchored groups and rescale. The difference between the two tells you how much error your anchor choice introduces. With a weak anchor you'll see small but non-zero errors on the low-volume terms. That's the rounding issue from pitfall 1, and it's the best way I know to pick an anchor with data instead of guessing.

Wrap-up

Each Google Trends request is scaled to its own 0-100 peak, so separate requests can't be compared directly.
Put a shared anchor in every request, scale each group by sum(anchor_ref) / sum(anchor_group), then renormalize so the global max is 100.
Most of the error comes from the anchor: choose a steady, popular one, warn when its mean is low, and never rescale a group where it's all zeros.
Keep geo, timeframe, category, and date axis identical across groups, fetch close together in time, and back off politely on 429s.

This article was written with AI assistance. The code was reviewed before publishing.