DEV Community

Daniel Igel
Daniel Igel

Posted on

Scrape any URL with one POST — Cloudflare bypass included

Most scraping libraries fall apart the moment they hit a Cloudflare-protected page — you get a 403 or an infinite challenge loop. Running a headless browser locally isn't an option in a serverless or edge environment, and maintaining your own bypass infrastructure is a project in itself.

One POST fetches any URL — Cloudflare-protected or not — and returns clean HTML, stripped text, or both in a single response:

curl --request POST \
  --url 'https://web-scraping-api.p.rapidapi.com/api/v1/scrape' \
  --header 'Content-Type: application/json' \
  --header 'x-rapidapi-key: YOUR_RAPIDAPI_KEY' \
  --header 'x-rapidapi-host: web-scraping-api.p.rapidapi.com' \
  --data '{"url":"https://example.com","output":"text"}'
Enter fullscreen mode Exit fullscreen mode

Set output to html for the raw markup, text for the stripped content (scripts and styles removed), or omit it to get both fields in one response alongside the final URL and HTTP status code.

When you only need specific data points, use the /extract endpoint with a CSS selector map:

const res = await fetch(
  'https://web-scraping-api.p.rapidapi.com/api/v1/extract',
  {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-rapidapi-key': process.env.RAPIDAPI_KEY,
      'x-rapidapi-host': 'web-scraping-api.p.rapidapi.com',
    },
    body: JSON.stringify({
      url: 'https://example.com/products',
      selectors: {
        title: 'h1',
        prices: '.price[]',
        thumbnail: 'img.hero@src',
      },
    }),
  }
);
const { data } = await res.json();
// { title: "Widget Pro", prices: ["$9.99", "$19.99"], thumbnail: "https://..." }
Enter fullscreen mode Exit fullscreen mode

Append [] to a selector to collect all matching elements as an array; add @attr to pull an attribute value instead of text content. No headless browser on your end — the bypass runs server-side.

Free tier on RapidAPI: https://rapidapi.com/danieligel/api/web-scraping-api

What's the site type that's been hardest to scrape reliably in your projects — Cloudflare, heavy SPAs, or login-gated content?

Top comments (0)