I Built a URL Metadata API After Wasting Days on Manual Scraping

root — Mon, 10 Nov 2025 16:58:31 +0000

The Problem That Wouldn't Go Away

I was building a bookmark manager (because apparently that's what developers do when they can't find the perfect one). Everything was going smoothly until I hit the "add bookmark" feature.

Users paste a URL, and I needed to show them a nice preview card with:

Title
Description
Image
Favicon

Simple, right? Wrong.

First Attempt: The BeautifulSoup Nightmare

Started with Python and BeautifulSoup. Wrote some code to fetch HTML and parse meta tags:

from bs4 import BeautifulSoup
import requests

def get_metadata(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.text, 'html.parser')

    title = soup.find('meta', property='og:title')
    description = soup.find('meta', property='og:description')
    image = soup.find('meta', property='og:image')

    return {
        'title': title['content'] if title else None,
        'description': description['content'] if description else None,
        'image': image['content'] if image else None
    }

Issues I immediately ran into:

JavaScript-rendered sites - Half the modern web doesn't have meta tags in the initial HTML. They're added client-side.
Inconsistent meta tag formats - Some sites use og:title, others use twitter:title, some just use <title>. You need fallback logic for everything.
Bot detection - Cloudflare and other services block obvious scrapers. My IP got banned testing on my own URLs.
Timeouts - Some sites take 10+ seconds to respond. Had to implement aggressive timeouts and retry logic.
Malformed HTML - So many sites have broken HTML that crashes parsers.

After three days of fighting edge cases, I gave up on the DIY approach.

Second Attempt: Existing APIs

Found a few metadata extraction APIs. The cheapest one was $49/month for 10,000 requests. For a side project that might get 100 users? No thanks.

The Solution: Build My Own

I realized this problem isn't going away. Every developer building bookmarks, chat apps, or content aggregators hits this wall.

So I built Scrapix - a production-ready metadata extraction API that handles all the annoying edge cases.

How It Works

Simple API Call

const response = await fetch('https://scrapix-api.com/extract', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-RapidAPI-Key': 'your-api-key'
  },
  body: JSON.stringify({ url: 'https://example.com' })
});

const metadata = await response.json();

Response Format

{
  "url": "https://dev.to",
  "title": "DEV Community",
  "description": "Where programmers share ideas and help each other grow",
  "image": "https://dev.to/social-preview.png",
  "favicon": "https://dev.to/favicon.ico",
  "site_name": "DEV Community"
}

Clean, consistent, predictable.

Real-World React Example

Here's how I use it in my bookmark manager:

import { useState } from 'react';

function BookmarkForm() {
  const [url, setUrl] = useState('');
  const [metadata, setMetadata] = useState(null);
  const [loading, setLoading] = useState(false);

  const fetchMetadata = async () => {
    setLoading(true);

    const response = await fetch('https://scrapix-api.com/extract', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-RapidAPI-Key': process.env.RAPIDAPI_KEY
      },
      body: JSON.stringify({ url })
    });

    const data = await response.json();
    setMetadata(data);
    setLoading(false);
  };

  return (
    <div>
      <input 
        value={url} 
        onChange={(e) => setUrl(e.target.value)}
        placeholder="Paste URL..."
      />
      <button onClick={fetchMetadata}>
        {loading ? 'Loading...' : 'Fetch Preview'}
      </button>

      {metadata && (
        <div className="preview-card">
          <img src={metadata.image} alt={metadata.title} />
          <h3>{metadata.title}</h3>
          <p>{metadata.description}</p>
        </div>
      )}
    </div>
  );
}

Under the Hood: How I Solved the Hard Problems

1. JavaScript-Rendered Content

Used headless browsers (Puppeteer) for sites that require client-side rendering. The API detects whether a page needs JS execution and routes accordingly.

2. Smart Caching

5-minute cache on all metadata requests. Most link previews don't change that often, so this cuts redundant requests by ~70%.

3. Multiple Fallback Strategies

The parser tries in order:

OpenGraph tags (og:*)
Twitter Cards (twitter:*)
Standard meta tags
HTML tags (<title>, <h1>)
URL parsing (domain name as fallback)

4. Global Performance

Deployed on Vercel's edge network. Average response time: 800ms. Cached responses: <100ms.

5. Batch Processing

Need metadata for 10 URLs at once? One API call handles it.

When Should You Use This vs DIY?

Build it yourself if:

You have one specific site to scrape
You need very custom data extraction
You're building a scraping tool as the core product

Use an API if:

You need it to work across any website
You don't want to maintain parsing logic
Your time is better spent on your actual product
You need reliable uptime and performance

Common Use Cases

I've seen developers use Scrapix for:

Bookmark managers - Auto-fill card previews
Chat apps - Rich link previews (like Slack/Discord)
Content aggregators - Pull article metadata
Social schedulers - Preview posts before publishing
CMS platforms - Auto-populate article details
RSS readers - Enhance feed items with images

Performance Matters

For my bookmark app, metadata extraction went from:

DIY approach: 3-15 seconds (with failures)
With Scrapix: 800ms average, 100ms cached

That's the difference between users waiting and users not noticing.

Pricing That Makes Sense

Free tier: 100 requests/month (perfect for testing)

Basic: 1,000 requests/month

Pro: 10,000 requests/month

(Check RapidAPI for current pricing)

Try It Yourself

The API is live on RapidAPI: Scrapix API

Free tier means you can test it with zero risk. If it doesn't solve your problem, you've lost nothing.

What I Learned Building This

Edge cases are infinite - Every site has its own quirks
Performance beats features - Developers want fast, reliable, simple
Good APIs solve one problem perfectly - Don't try to do everything
Caching is magic - 5-minute TTL made costs drop dramatically

The Bottom Line

If you're building anything that needs link previews, don't waste three days like I did. Your time is worth more than that.

Focus on what makes your product unique. Let specialized tools handle the boring infrastructure stuff.

Have you built something similar? What challenges did you face? Drop a comment below - always curious to hear other developers' experiences with web scraping and metadata extraction.

P.S. - If you found this helpful, I'd love to hear what you're building! Connect with me on [https://github.com/fiston-user]

support images

root — Wed, 26 Feb 2025 21:47:50 +0000

support images

slsl

root — Wed, 26 Feb 2025 21:46:29 +0000

slsl

kqkl

root — Wed, 26 Feb 2025 21:46:10 +0000

kqkl

post on all textxs platforms

root — Wed, 26 Feb 2025 21:44:12 +0000

post on all textxs platforms

reddit tell me

root — Tue, 25 Feb 2025 16:59:00 +0000

reddit tell me

yh

root — Mon, 24 Feb 2025 21:55:23 +0000

test

root — Mon, 24 Feb 2025 21:52:56 +0000

test

root — Mon, 24 Feb 2025 16:30:51 +0000

test

qoqo

root — Mon, 24 Feb 2025 16:29:54 +0000

qoqo

testing

root — Sat, 22 Feb 2025 16:42:27 +0000

testing

test schduled

root — Sat, 22 Feb 2025 13:25:00 +0000

test schduled

DEV Community: root

I Built a URL Metadata API After Wasting Days on Manual Scraping

The Problem That Wouldn't Go Away

First Attempt: The BeautifulSoup Nightmare

Second Attempt: Existing APIs

The Solution: Build My Own

How It Works

Simple API Call

Response Format

Real-World React Example

Under the Hood: How I Solved the Hard Problems

1. JavaScript-Rendered Content

2. Smart Caching

3. Multiple Fallback Strategies

4. Global Performance

5. Batch Processing

When Should You Use This vs DIY?

Common Use Cases

Performance Matters

Pricing That Makes Sense

Try It Yourself

What I Learned Building This

The Bottom Line

support images

slsl

kqkl

post on all textxs platforms

reddit tell me

yh

test

test

qoqo

testing

test schduled