Beyond Link Previews: How to Extract Readable Article Content from URLs

Arthur Weale — Thu, 12 Nov 2026 04:10:57 +0000

When I started building GrabApe, I initially focused on extracting the information that makes link previews possible: page titles, descriptions, images and URLs.

These are the details most applications need when displaying a link. But as I worked on the project, I discovered another problem.

Sometimes, knowing what a webpage is about isn't enough. An application might need the actual article text so it can display, process, summarize or provide context from the content behind a URL.

That's where article extraction comes in.

A link preview tells you what a webpage is about. Article extraction helps you access what the webpage actually says.

In this article, I'll explain the difference between metadata extraction and article extraction, some of the challenges involved, and how I approached the problem while building GrabApe.

1. Understanding Metadata Extraction

When you share a URL in an application, you often see a preview containing information such as:

Page title
Description
Preview image
Website name
Canonical URL
Author information

This information is generally available through HTML metadata.

Many websites use Open Graph and Twitter Card metadata to describe their pages.

For example, a webpage might contain:

<meta property="og:title" content="Understanding API Development">
<meta property="og:description" content="A guide to building and working with APIs.">
<meta property="og:image" content="https://example.com/image.png">
<meta property="og:site_name" content="Example">

An application can retrieve these values and use them to generate a rich link preview.

This is useful for messaging applications, content platforms, bookmarking tools and many other products.

Metadata extraction is relatively straightforward because websites often provide these fields specifically for applications and social platforms to consume.

However, there is a limitation.

Metadata describes the webpage. It doesn't necessarily contain the actual article.

2. When Metadata Isn't Enough

Imagine you're building an application that allows users to save articles they find online.

You can extract the title, description and image, then display a nice preview.

But what happens when the user wants to read the article inside your application?

Or imagine you're building an AI-powered application that needs the content of a webpage to provide context for a summary or answer.

The title and description aren't enough.

You need the actual text.

There are several situations where article extraction becomes useful:

Reading applications: Retrieve readable article content instead of displaying only a link.
AI applications: Provide webpage content as context for summarization or other processing.
Research tools: Extract information from articles for further analysis.
Knowledge management: Save the content behind links alongside their metadata.
Search applications: Make the actual content of saved webpages available for indexing and searching.

In these situations, metadata extraction and article extraction serve different purposes.

Metadata helps identify and present a webpage. Article extraction attempts to retrieve its main readable content.

3. Why Article Extraction Is More Difficult Than It Looks

This is where things become interesting.

A webpage isn't necessarily structured around the article itself.

It can contain:

Navigation menus
Sidebars
Advertisements
Related articles
Cookie notices
Comment sections
Social sharing controls
Author information
Other interactive elements

If you simply extract all the text from the HTML document, you can end up with a lot of unnecessary content.

The challenge is identifying which parts of the page actually belong to the main article.

There is also the problem of inconsistent HTML structures.

One website might use semantic HTML elements such as <article> and <main>. Another might use custom <div> elements with classes and attributes. Some websites provide structured article information through JSON-LD, while others provide very little useful structure.

JavaScript-rendered pages introduce another challenge because some content might not be present in the initial HTML response.

Several techniques can help:

Semantic HTML: Identify elements intended to represent the main article.
Structured metadata: Look for article information in JSON-LD and other structured formats.
Content extraction libraries: Use tools such as Readability to identify the main readable content.
Browser rendering: Render pages when important content is generated through JavaScript.

None of these approaches works perfectly on every website.

A page can have a perfectly valid <article> element and still contain unrelated content inside it.

For example, an article element might include an entire discussion section with hundreds or thousands of words from user comments.

Extracting the whole element would technically retrieve the article's text, but it would also retrieve content that doesn't belong to the article itself.

That distinction became particularly important while developing GrabApe.

4. A Layered Approach to Article Extraction

While building GrabApe, I approached article extraction as a layered process rather than relying on one technique.

The extractor looks for opportunities to identify the main article content.

The current approach includes:

1. JSON-LD structured article content

Some websites provide article content through structured metadata, including an articleBody property.

When suitable structured article content is available, it can provide a useful starting point.

2. Semantic HTML

The extractor examines semantic HTML structures and attributes that indicate where article content might be located.

These include elements such as <article> and properties such as itemprop="articleBody".

3. Readability-based extraction

When the earlier approaches don't produce a sufficiently confident result, the extractor can use Readability as a fallback.

The idea is to provide multiple opportunities to identify readable content rather than depending entirely on one HTML structure.

However, identifying the article container isn't always the end of the problem.

Handling discussion sections

During testing, I encountered a webpage where the selected article container also included thousands of words from its comment section.

The main article was being extracted, but so were the comments.

This exposed an important distinction: finding an article container doesn't guarantee that everything inside it belongs to the article.

I introduced an additional conservative pruning step to identify clearly separate discussion regions.

Rather than relying on a website-specific selector, the approach combines several structural signals, including:

Comment-like HTML attributes.
Repeated user or profile structures.
Reply and interaction elements.
The relationship between a suspected discussion region and the main article content.

The extractor looks for combinations of evidence instead of treating a single class name or keyword as sufficient.

It also avoids removing ambiguous regions simply because they resemble a discussion.

The objective is to improve extraction quality without introducing rules that only work for individual websites.

This approach is still heuristic. Different websites have different HTML structures, and there will be cases where discussion content is difficult to distinguish from the main article.

I would rather preserve ambiguous content than aggressively remove legitimate article text.

5. Extracting Article Content with GrabApe

This is one of the capabilities I built into GrabApe.

GrabApe provides URL extraction through an API, allowing applications to retrieve structured metadata and optionally request readable article content.

The metadata endpoint supports ordinary link-preview use cases.

When article content is required, the application can use the POST extraction endpoint and include the article option.

This means developers don't have to treat every URL as a request for full article extraction.

They can request the information their application actually needs.

The API also distinguishes between different article extraction outcomes.

The content status can be:

success: Article content was successfully extracted.
unavailable: Article content could not be obtained.
failed: An extraction failure occurred.

These distinctions matter because an application shouldn't assume every website will provide extractable article content.

For example, if a page doesn't expose its content in a way the extractor can use, the application should be able to handle that outcome instead of assuming that an empty article is a successful extraction.

6. Making a Practical API Request

Here's an example of requesting article content through GrabApe.

The endpoint is:

POST https://grabape.p.rapidapi.com/v1/extract

The JSON request body:

{
  "url": "https://example.com/article",
  "include": ["article"]
}

The request also requires the appropriate RapidAPI authentication headers:

X-RapidAPI-Key: YOUR_RAPIDAPI_KEY
X-RapidAPI-Host: grabape.p.rapidapi.com
Content-Type: application/json

Let's break down the request.

url

The webpage you want to extract information from.

include

Specifies additional information you want the API to retrieve.

In this example, article requests readable article content alongside the metadata.

If you only need ordinary metadata, you can use the GET /v1/extract endpoint instead.

The API documentation is available here:

https://grabape.onrender.com/docs

You can also find the API on RapidAPI:

https://grabape.p.rapidapi.com

7. Handling Different Extraction Outcomes

One thing I wanted to avoid was making article extraction appear more reliable than it actually is.

Websites differ significantly in how they expose content. Some provide clean article structures, while others make extraction considerably more difficult.

That's why the API communicates the extraction outcome instead of treating every request as a guaranteed success.

For example:

Successful extraction

The API identifies and returns readable article content.

An application can use the returned text for reading, summarization or further processing.

Unavailable content

The API couldn't obtain usable article content.

The application can still use any successfully extracted metadata to display a link preview.

Failed extraction

An extraction failure occurred.

The application can handle this separately, perhaps by informing the user that the article couldn't be processed.

These distinctions give developers more control over how their applications respond to different websites.

A useful extraction API shouldn't just return content when everything works. It should also communicate when it doesn't.

8. A Real Production API Response

To test the implementation, I used a real DEV Community article:

"We Keep Hearing About AI Productivity. The QA Group Chat Told a Different Story."

The page contained the main article as well as a substantial discussion section with user comments.

My initial extraction successfully retrieved the article, but it also included the comments.

The result was 6,876 words.

That wasn't the result I wanted.

I investigated the HTML structure and discovered that the comments were nested inside the broader article container, even though the main article body was a separate element.

Instead of adding a special rule for DEV Community, I improved the general discussion-region detection logic.

After implementing the change, I tested the saved HTML locally.

The extractor returned 2,755 words, retaining the main article while excluding the checked comment-section content.

I then deployed the change and tested the actual production API.

The production request returned HTTP 200, with the article extraction status set to success.

The relevant response fields were:

{
  "content": {
    "type": "article",
    "word_count": 2755,
    "status": "success",
    "diagnostics": {
      "reason": null,
      "warnings": []
    }
  }
}

The main article was retained, the checked comment content was excluded, and the extraction diagnostics contained no warnings.

This was an important test because it demonstrated that the improvement worked against a real webpage, not just a synthetic HTML example.

It also reinforced an important principle in building extraction software: a successful response isn't necessarily a good extraction.

You have to look at what was actually returned.

This is one verified production example, not a guarantee that comments will be excluded from every website. Different websites structure their pages differently, and extraction quality needs to be evaluated across a variety of real-world pages.

9. Conclusion

Building GrabApe has made me think differently about what happens when an application receives a URL.

Initially, I was focused on extracting the information needed to create link previews.

But I discovered that there is another layer of useful information behind a link: the actual content of the webpage.

Metadata extraction and article extraction solve related but different problems.

Metadata helps applications identify and present webpages. Article extraction attempts to retrieve the main readable content.

The difficult part isn't simply downloading HTML and collecting text. It's understanding the structure of a webpage, identifying the content that matters, and avoiding unrelated material.

There isn't one extraction technique that works perfectly everywhere. A layered approach, conservative content filtering and clear extraction outcomes make the process more useful for developers.

That's the direction I'm taking with GrabApe: making it easier for applications to turn URLs into structured, usable information.

Because sometimes, a link preview just isn't enough.

The real value is getting to the content behind the link.

DEV Community: Arthur Weale