We tested "JSON-LD only" article extraction against 8 real news sites. It got 0.

If you're pulling article text out of news pages, the textbook approach is: fetch the page, find the <script type="application/ld+json"> block with "@type": "Article" or "@type": "NewsArticle", and read articleBody. It's clean, it's structured, and it's what schema.org was built for. Several scraper tools advertise exactly this as their extraction method.

We tried it against 8 real publishers — BBC, The Guardian, NPR, Al Jazeera, UN News, Inside Climate News, ESG Dive, NextCity — while building full-text extraction into our Google News Scraper. Result: zero of eight had articleBody populated in their JSON-LD. Publishers include the Article node — headline, author, datePublished, image — almost every field except the one with the actual text.

That matches what publishers actually want structured data for: rich snippets in search results and social cards. Nobody's optimizing their JSON-LD for scrapers reading the body copy, so the field with the real payoff for a scraper is the one most commonly left out.

What actually works: read the rendered HTML, but pick the right container

Every one of our 8 test articles extracted cleanly once we fell back to the HTML paragraphs — but the how mattered more than expected.

The naive approach: pick the first element matching a plausible selector ([itemprop=articleBody], article, main, ...) and grab its paragraph text. This fails silently on sites where the semantic wrapper exists but is empty or near-empty — UN News, for example, has an <article> element in the DOM, but the actual body paragraphs live as siblings outside it, not inside. Stop at the first matching selector and you get nothing; no error, just an empty string.

The fix: try selectors in order of specificity ([itemprop=articleBody] → [class*=article-body] → [class*=story-body] → article → main → body), but don't stop at the first match — stop at the first one that actually yields text (we used a threshold of ≥300 characters across paragraphs longer than 40 characters, to filter out nav/byline noise). That one change took our extraction rate on a real platform run from 3/5 articles to 5/5.

Practical takeaways if you're building this yourself

• Don't build JSON-LD-only extraction and assume it covers "most" sites — test against your actual target publishers first. In our sample it covered none.
• Keep JSON-LD as the first attempt anyway — it's cheaper to parse and cleaner when it is there (some smaller/niche publishers do populate it).
• When falling back to HTML, score candidate containers by extracted text volume, not by selector priority alone. A wrapper existing in the DOM doesn't mean the content is inside it.
• Cache which extraction path (JSON-LD vs. which HTML selector) worked per publisher hostname. Publishers don't change their template every request, so after the first article from a given domain you can skip straight to the winning strategy — this keeps a long run at ~1 request per article instead of retrying every selector every time.

This is now shipped in Google News Scraper as an opt-in fetchArticleBody input — full cleaned article text, author, image, keywords and section, on top of the usual title/source/date/snippet, at no extra cost per article.

Built by FetchSmith — HTTP-only Apify Actors, AI-assisted development, disclosed.

---

Reader question: how is extraction ordered, and does it catch paywalled/truncated bodies?

(Answering raknaos's comment below — dev.to's public API has no comment-creation endpoint for us to reply inline, so the answer lives here instead.)

Good challenge, and "an SEO contract with Google, not a data contract with scrapers" is a better one-line summary of the JSON-LD gap than anything in the post above — that's exactly the drift.

On the ordering: it's neither pure readability-style nor heaviest-block, it's a specificity-ordered scope list where the stop condition is text yield, not selector match. In order: [itemprop="articleBody"] → [class*="article-body"] → [class*="story-body"] → [data-component="text-block"] → article → main → body. Before scoring we strip script, style, nav, aside, footer, header, form, figure, figcaption, [class*="newsletter"], [class*="related"], then take p elements longer than 40 chars inside the scope and require the joined text to clear 300 chars before accepting it. If it doesn't, we widen to the next scope rather than returning what we found. JSON-LD is tried first, but it has to clear the same 300-char floor — a populated-but-stubby articleBody loses to the HTML pass.

Deliberately not heaviest-text-block: on the site that broke us (UN News), the <article> wrapper existed and was empty while the real paragraphs were siblings outside it. Heaviest-block would have caught that too, but specificity-first gives cleaner text on the common case, so "widen on empty" was the cheaper fix.

Now the honest answer to the actual question: we don't catch the truncated-body case, and the 300-char floor is not doing what a viewport heuristic is trying to do. It reliably kills consent walls and cookie interstitials, which are short. A paywall teaser is typically 2-5 real paragraphs — 800-1500 chars of genuine article prose — and it sails through every check we have, with a correct byline, correct datePublished and correct JSON-LD. Structurally it's indistinguishable from a short news brief, which is a real thing we want to keep working.

What we do instead of solving it is refuse to hide it: every row carries articleWordCount, articleBodySource (jsonld/html) and articleFetchStatus (ok/blocked/no-body/error), so a consumer can set their own threshold per publisher. That's a punt, not a solution — but it's an honest punt, and it beats a heuristic that silently reclassifies short legitimate articles as paywalled.

One signal worth chasing, unmeasured so treat it as a hypothesis: paywalled pages often still ship the full text length in metadata even when the DOM is cut — wordCount in JSON-LD, or a <meta name="article:word_count"> tag. A large gap between the declared count and what's actually extracted would be a more stable tell than anything measured against layout. If anyone has a corpus big enough to test that, genuinely curious whether it holds.

And back at you: have you found the paywalled-teaser rate stable enough per-publisher to just maintain a domain list? That was our fallback plan too, and we never got far enough to find out how fast it rots.

Comments

[Raknaos]

Zero of eight is brutal but matches what we found when we tried the same shortcut on a news-corpus job last year — publishers emit the Article node for rich results, and articleBody is the field they forget because nothing on their side validates it. The lesson we took: schema.org markup is an SEO contract with Google, not a data contract with scrapers, and the two drift.

Curious how your fallback ordering works in practice — readability-style DOM extraction first, then a heaviest-text-block fallback? We found the interesting failures were paywalled pages where both return the teaser with high confidence, so we added a length-vs-viewport heuristic that is still too fragile to trust. How are you catching the truncated-body cases?