Build a Web Data Extractor API with Puppeteer and Next.js

Modern websites are built with JavaScript frameworks (React, Vue, Angular) that render content dynamically. Traditional HTTP requests with axios or fetch can't extract data from these sites because the content doesn't exist in the initial HTML.

The solution? A headless browser.

What We're Building

A web data extraction API that loads any URL with a full Chromium browser via Puppeteer, waits for JavaScript to render completely, and returns structured JSON: title, text content, links, and images.

Core Architecture

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle2' });

const data = await page.evaluate(() => ({
  title: document.title,
  text: document.body.innerText,
  links: Array.from(document.querySelectorAll('a[href]'))
    .map(a => ({ text: a.innerText, href: a.href })),
  images: Array.from(document.querySelectorAll('img[src]'))
    .map(img => ({ alt: img.alt, src: img.src })),
}));

Why Puppeteer Over Simple HTTP Requests

Feature	HTTP Request	Puppeteer
JavaScript rendering	No	Full JS execution
SPA support (React/Vue)	No	Waits for hydration
Dynamic content	No	networkidle2 waits
Authentication flows	No	Full browser session

Production Considerations

Memory Management

Puppeteer launches a full Chromium instance per request. For production:

• Use a browser pool (reuse browser instances)
• Set memory limits (512MB per instance)
• Implement request queuing

Timeouts

Always set timeouts to avoid hanging requests:

await page.goto(url, {
  waitUntil: 'networkidle2',
  timeout: 30000
});

Error Handling

Websites can fail: network errors, CAPTCHAs, bot detection.

• Wrap in try/catch
• Implement retry logic
• Use stealth plugins to avoid detection

The API

POST /api/extract
{
  "url": "https://example.com/products",
  "prompt": "Get all product names and prices"
}

Response:
{
  "ok": true,
  "data": {
    "title": "Products - Example",
    "text": "... page content ...",
    "links": [...],
    "images": [...]
  },
  "meta": {
    "elapsed": 4234,
    "contentLength": 15234
  }
}

Conclusion

Building a web data extraction API with Puppeteer and Next.js is straightforward. The key insight is that modern web scraping requires a real browser engine, not just HTTP requests. Puppeteer gives you that capability with a clean, well-documented API.

The complete source is open. Contributions and feedback welcome.