Making a JavaScript-drawn page readable to crawlers: pre-rendering at build time and auto-noindex for thin pages

I have a Korean information site that is plain static HTML, rebuilt by a Python script every hour and pushed to GitHub Pages. Some screens (an issues ranking, calculators, a recommendation form) are drawn by JavaScript from a JSON file. For a human that's fine. For a crawler that doesn't run my scripts, the page was almost empty.

I measured the text a crawler can read on one of those JS-drawn pages: 687 characters before, 13,605 after pre-rendering. This post is how, plus the two clean-ups that came with it. All code is from the real build script, trimmed.

1. Pre-render the same data the JS would draw

The page used to contain only this:

<section id="issues"><noscript>JavaScript is required.</noscript></section>

The build already loads news/data.json for the front end, so the Python side now draws the same list into the HTML. The browser script replaces it on load, so users see no difference:

def news_static():
    """For crawlers and no-JS clients. The page's JS replaces this on load."""
    d = _rd('news/data.json')
    if not d:
        return '<noscript>JavaScript is required.</noscript>'
    I = {a['id']: a for a in d.get('items', [])}
    out = f'<p>{E(d["updated"])} · {len(I)} articles grouped by event</p><ol class="nw-static">'
    for x in d.get('issues', [])[:40]:
        arts = ''.join(
            f'<li>{E(I[a]["o"])} · <a href="{E(I[a]["u"])}" rel="nofollow noopener">{E(I[a]["t"])}</a></li>'
            for a in x['a'][:6] if a in I)
        out += f'<li id="is{x["n"]}"><h2>{E(x["t"])}</h2><p>{x["on"]} outlets</p><ul>{arts}</ul></li>'
    return out + '</ol>'

Three small decisions that mattered:

• Real headings and real links, not a <pre> dump. <h2> per item, <a> per source, with rel="nofollow noopener" on links that leave the site.
• A cap ([:40] items, [:6] links each), so the static copy stays a readable list instead of a data dump.
• Same data source as the JS. There's one JSON; the build renders it and the browser renders it. If they ever disagreed I'd rather see it in a diff than in two code paths.

2. Sitemap lastmod that only changes when the content does

If every page says "modified today" on every hourly build, the date carries no information. The build hashes each page's title + body after removing the parts that change on every build (D-day counters, clock times, dates), and keeps [hash, date] per URL in a JSON file in the repo:

volatile = r'D-\d+|D-DAY|\d{1,2}:\d{2}|\d{4}-\d{2}-\d{2}|\d{1,2}/\d{1,2}'
PAGE_HASH[path] = hashlib.md5(re.sub(volatile, '', title + body).encode()).hexdigest()

for u in urls:
    h = PAGE_HASH.get(u)
    if u not in lm or (h and lm[u][0] != h):
        lm[u] = [h, TODAY.isoformat()]

Known weakness: that last alternative also strips fractions like 3/4, so an edit that only changes one fraction into another won't move the date. Fine for deadlines and prices; not fine for everything.

3. Automatically noindex the thin generated pages

Generating pages from data means some pages end up with almost nothing of their own. After the build, one pass counts each detail page's own text. A line counts as "own" only if its shape (digits replaced by #) shows up on fewer than 5 pages, which filters out the shared header, footer and boilerplate:

freq = collections.Counter()
for u, (f, doc, lines) in got.items():
    freq.update({re.sub(r'[\d,.]+', '#', l) for l in lines})

for u, (f, doc, lines) in got.items():
    own = sum(len(l.replace(' ', '')) for l in lines
              if freq[re.sub(r'[\d,.]+', '#', l)] < 5)
    if own < 200:
        NOINDEX.add(u)       # dropped from sitemap too
        # inject <meta name="robots" content="noindex,follow"> into the file

noindex,follow (not nofollow), so link equity still flows through the page. Pages in NOINDEX are filtered out of the sitemap at write time. The threshold of 200 characters is a judgment call, not a magic number.

What is still not done

Several sections (calculators, the recommendation form, the fridge tool) still only have a <noscript> line in the HTML, because their output depends on user input and there is nothing meaningful to pre-render. For those, the surrounding page needs real explanatory text instead, which is a content job, not a code job.

I have no ranking numbers to show yet; the site is new. The 687 → 13,605 figure is the amount of text visible without running JS, not a traffic claim.

The site is in Korean. Build notes (also Korean): https://sungprofile.cloud/ailog/?src=devto