We tested "JSON-LD only" article extraction against 8 real news sites. It got 0.

If you're pulling article text out of news pages, the textbook approach is: fetch the page, find the <script type="application/ld+json"> block with "@type": "Article" or "@type": "NewsArticle", and read articleBody. It's clean, it's structured, and it's what schema.org was built for. Several scraper tools advertise exactly this as their extraction method.

We tried it against 8 real publishers — BBC, The Guardian, NPR, Al Jazeera, UN News, Inside Climate News, ESG Dive, NextCity — while building full-text extraction into our Google News Scraper. Result: zero of eight had articleBody populated in their JSON-LD. Publishers include the Article node — headline, author, datePublished, image — almost every field except the one with the actual text.

That matches what publishers actually want structured data for: rich snippets in search results and social cards. Nobody's optimizing their JSON-LD for scrapers reading the body copy, so the field with the real payoff for a scraper is the one most commonly left out.

What actually works: read the rendered HTML, but pick the right container

Every one of our 8 test articles extracted cleanly once we fell back to the HTML paragraphs — but the how mattered more than expected.

The naive approach: pick the first element matching a plausible selector ([itemprop=articleBody], article, main, ...) and grab its paragraph text. This fails silently on sites where the semantic wrapper exists but is empty or near-empty — UN News, for example, has an <article> element in the DOM, but the actual body paragraphs live as siblings outside it, not inside. Stop at the first matching selector and you get nothing; no error, just an empty string.

The fix: try selectors in order of specificity ([itemprop=articleBody] → [class*=article-body] → [class*=story-body] → article → main → body), but don't stop at the first match — stop at the first one that actually yields text (we used a threshold of ≥300 characters across paragraphs longer than 40 characters, to filter out nav/byline noise). That one change took our extraction rate on a real platform run from 3/5 articles to 5/5.

Practical takeaways if you're building this yourself

• Don't build JSON-LD-only extraction and assume it covers "most" sites — test against your actual target publishers first. In our sample it covered none.
• Keep JSON-LD as the first attempt anyway — it's cheaper to parse and cleaner when it is there (some smaller/niche publishers do populate it).
• When falling back to HTML, score candidate containers by extracted text volume, not by selector priority alone. A wrapper existing in the DOM doesn't mean the content is inside it.
• Cache which extraction path (JSON-LD vs. which HTML selector) worked per publisher hostname. Publishers don't change their template every request, so after the first article from a given domain you can skip straight to the winning strategy — this keeps a long run at ~1 request per article instead of retrying every selector every time.

This is now shipped in Google News Scraper as an opt-in fetchArticleBody input — full cleaned article text, author, image, keywords and section, on top of the usual title/source/date/snippet, at no extra cost per article.

Built by FetchSmith — HTTP-only Apify Actors, AI-assisted development, disclosed.

---

Reader question: how is extraction ordered, and does it catch paywalled/truncated bodies?

(Answering raknaos's comment below — dev.to's public API has no comment-creation endpoint for us to reply inline, so the answer lives here instead.)

Good challenge, and "an SEO contract with Google, not a data contract with scrapers" is a better one-line summary of the JSON-LD gap than anything in the post above — that's exactly the drift.

On the ordering: it's neither pure readability-style nor heaviest-block, it's a specificity-ordered scope list where the stop condition is text yield, not selector match. In order: [itemprop="articleBody"] → [class*="article-body"] → [class*="story-body"] → [data-component="text-block"] → article → main → body. Before scoring we strip script, style, nav, aside, footer, header, form, figure, figcaption, [class*="newsletter"], [class*="related"], then take p elements longer than 40 chars inside the scope and require the joined text to clear 300 chars before accepting it. If it doesn't, we widen to the next scope rather than returning what we found. JSON-LD is tried first, but it has to clear the same 300-char floor — a populated-but-stubby articleBody loses to the HTML pass.

Deliberately not heaviest-text-block: on the site that broke us (UN News), the <article> wrapper existed and was empty while the real paragraphs were siblings outside it. Heaviest-block would have caught that too, but specificity-first gives cleaner text on the common case, so "widen on empty" was the cheaper fix.

Now the honest answer to the actual question: we don't catch the truncated-body case, and the 300-char floor is not doing what a viewport heuristic is trying to do. It reliably kills consent walls and cookie interstitials, which are short. A paywall teaser is typically 2-5 real paragraphs — 800-1500 chars of genuine article prose — and it sails through every check we have, with a correct byline, correct datePublished and correct JSON-LD. Structurally it's indistinguishable from a short news brief, which is a real thing we want to keep working.

What we do instead of solving it is refuse to hide it: every row carries articleWordCount, articleBodySource (jsonld/html) and articleFetchStatus (ok/blocked/no-body/error), so a consumer can set their own threshold per publisher. That's a punt, not a solution — but it's an honest punt, and it beats a heuristic that silently reclassifies short legitimate articles as paywalled.

One signal worth chasing, unmeasured so treat it as a hypothesis: paywalled pages often still ship the full text length in metadata even when the DOM is cut — wordCount in JSON-LD, or a <meta name="article:word_count"> tag. A large gap between the declared count and what's actually extracted would be a more stable tell than anything measured against layout. If anyone has a corpus big enough to test that, genuinely curious whether it holds.

And back at you: have you found the paywalled-teaser rate stable enough per-publisher to just maintain a domain list? That was our fallback plan too, and we never got far enough to find out how fast it rots.

---

Reader note: how we catch the truncated-body cases (thanks @raknaos)

@raknaos asked the question this post ducked: the fallback ordering handles empty containers, but what about paywalled pages where every layer returns the teaser with high confidence? They said their own length-vs-viewport heuristic was "still too fragile to trust."

Honest answer at the time: we weren't catching them. A metered teaser arrives as real <p> paragraphs, clears the 300-character floor, and gets reported as articleFetchStatus: "ok" with a full-looking body. Nothing on the row says otherwise. We've now shipped a fix, and the interesting part is the two things that didn't work.

Attempt 1: trust isAccessibleForFree. Google's paywall markup has publishers declare gated content in JSON-LD. Clean signal, and it's well-populated — unlike articleBody, because Google actually penalizes cloaking if you lie about it. But it's a statement of intent, not of what your request received. theatlantic.com carries isAccessibleForFree: false on every article, and served us a complete 3,479-word body. Metered paywalls gate the Nth read, not the first. Flag alone: false positive.

Attempt 2: check whether the gated region actually arrived. The same markup makes the publisher name the paywalled region by CSS selector (hasPart[].cssSelector), so you can stop guessing and go look at that exact region in the HTML you were served. This felt airtight. It flagged two complete SCMP articles (849 and 1,123 words) as truncated — because scmp.com points its selector at .piano-metering__paywall-container, the client-side paywall overlay, which is correctly empty on a free read. An empty gated region means "the overlay didn't fire" at least as often as it means "the body was stripped."

What shipped: articleBodyComplete as three-state, where false requires two independent observations agreeing — the gated region came back empty and what we're holding is teaser-sized (≤220 words). One observation short of that and the answer is null, with articleBodyIncompleteReason: "paywall-declared-unverifiable". There's also an independent path: when a publisher declares its own wordCount, we surface it as articleDeclaredWordCount next to ours, and under 60% is false on our own numbers.

{ "articleWordCount": 849, "articleDeclaredWordCount": null,
  "articleBodyComplete": null, "articleBodyIncompleteReason": "paywall-declared-unverifiable" }

The rule we took from this, which is really the same lesson as the post: never let the field assert a cause you didn't observe. A confident false on those SCMP articles would have been worse than the silence we started with — silence is at least honestly ambiguous, whereas a wrong false is a number someone filters on. null is a first-class answer, and to your viewport heuristic: we'd keep it, just never let it vote alone.

So on the truncated-body cases specifically: no, we don't have a general solution, and length-based heuristics alone didn't get us one either. What we have is a narrow, checkable case — publisher names the gated region, region is empty, body is teaser-sized — and an explicit "unknown" everywhere else. On a live 25-article run that's 2 confirmed-complete, 3 unknown, 20 with no signal either way.

Comments

[Raknaos]

Zero of eight is brutal but matches what we found when we tried the same shortcut on a news-corpus job last year — publishers emit the Article node for rich results, and articleBody is the field they forget because nothing on their side validates it. The lesson we took: schema.org markup is an SEO contract with Google, not a data contract with scrapers, and the two drift.

Curious how your fallback ordering works in practice — readability-style DOM extraction first, then a heaviest-text-block fallback? We found the interesting failures were paywalled pages where both return the teaser with high confidence, so we added a length-vs-viewport heuristic that is still too fragile to trust. How are you catching the truncated-body cases?