DEV Community

Kabeer Ahmed
Kabeer Ahmed

Posted on

Building 8 Production Web Scrapers & Manifest V3 Browser Extensions in Node.js

#javascript #node #webdev #opensource

Over the past few weeks, I built and shipped 8 production cloud scrapers on the Apify Store alongside two cross-browser Manifest V3 extensions on the Mozilla Add-ons Store.

Most scraping tutorials default to spinning up full headless Chromium instances for every URL. While Playwright and Puppeteer are essential when dealing with heavy client-side hydration, they are often overkill for structured B2B extraction, e-commerce catalog indexing, and technical SEO auditing. By combining CheerioCrawler in Crawlee with hidden platform JSON endpoints and regex-backed DOM parsers, you can cut memory overhead and runtime costs by over 85%.

Here is a practical look at the architecture behind these tools and where you can try them out.

1. Fast E-Commerce Catalog Extraction Without Headless Browsers

When scraping Shopify or WooCommerce stores, rendering product grids in a headless browser is slow and fragile. Both platforms expose structured JSON APIs out of the box on almost every storefront:

  • Shopify: GET /products.json?limit=250&page=1 returns full product variants, SKUs, pricing, tags, and image URLs.
  • WooCommerce: GET /wp-json/wc/store/v1/products?per_page=100 (Store API v1) exposes catalog items, stock status, and pricing metadata, with HTML fallback to Schema.org Product JSON-LD when the REST route is restricted. 
async function fetchShopifyProducts(baseUrl, maxProducts = 250) {
  const url = new URL('/products.json?limit=250&page=1', baseUrl).toString();
  const res = await fetch(url, {
    headers: { 'User-Agent': 'Mozilla/5.0 (compatible; CatalogBot/1.0)' }
  });
  if (!res.ok) return [];
  const data = await res.json();
  return (data.products || []).slice(0, maxProducts).map(p => ({
    id: p.id,
    title: "p.title,"
    handle: p.handle,
    vendor: p.vendor,
    productType: p.product_type,
    price: p.variants?.[0]?.price || null,
    available: p.variants?.some(v => v.available) ?? false
  }));
}
Enter fullscreen mode Exit fullscreen mode

I packaged these patterns into two dedicated cloud actors:

2. Converting Arbitrary Websites into Token-Sized Markdown Chunks for AI & RAG

Feeding raw HTML into an LLM wastes context window tokens on navigation bars, cookie banners, inline SVGs, and tracking scripts. When building the Website to Clean Markdown & AI RAG Content Extractor, I structured the pipeline in three stages:

  1. Boilerplate Stripping: Remove script, style, noscript, svg, nav, footer, header, and aside nodes via Cheerio before inspecting text nodes.
  2. Semantic Markdown Conversion: Convert headings (h1 to h6), lists, tables, code blocks, and anchor links into clean Markdown while preserving document hierarchy.
  3. Token-Aware Chunking: Split long documents on heading boundaries into configurable chunk sizes (e.g. 500 to 1,000 words) with metadata (url, title, chunkIndex, estimatedTokens) ready for direct ingestion into vector databases like Pinecone, Qdrant, or pgvector.

3. Detecting 40+ Tech Stack Signatures, Ad Pixels & B2B Contact Signals

For sales intelligence and agency prospecting, knowing whether a company runs Shopify Plus, Next.js, HubSpot, Klaviyo, Meta Pixel, or Google Tag Manager is just as important as finding their contact email.

In the Website Tech Stack, Pixel & CMS Detector, each page response is evaluated across four lightweight inspection layers:

  • Script src & Link href patterns: Detects CDN signatures (cdn.shopify.com, /_next/static/, js.hs-scripts.com, static.klaviyo.com).
  • Inline Script Globals: Matches initialization snippets such as fbq('init', gtag('config', lintrk, and ttq.load.
  • Meta Generator & DOM Attributes: Checks meta[name="generator"], data-wf-site (Webflow), and ng-version (Angular).
  • HTTP Response Headers: Inspects x-powered-by, server, and x-shopify-stage.

Here are the rest of the B2B and SEO actors in the suite:

4. Shipping Cross-Browser Manifest V3 Extensions (Chrome & Firefox)

Alongside the cloud scrapers, I wanted instant in-browser inspection tools when reviewing client websites. I built and published two Manifest V3 browser extensions on the official Mozilla Add-ons Store:

  1. LeadSnap Pro - B2B Contact & Email Extractor: Extracts verified emails, phone numbers, WhatsApp links, and social profiles directly from the active tab DOM and mailto: / tel: links, with one-click CSV and JSON export.
  2. MetaPeek Pro - Technical SEO & Schema Auditor: Audits canonical URLs, title/description pixel lengths, heading hierarchy (H1 to H6), OpenGraph/Twitter cards, image alt coverage, and Schema.org JSON-LD blocks in real time.

Tip for Cross-Browser MV3 Manifests

When packaging a single Manifest V3 codebase for both Chromium and Firefox, remember that Firefox requires browser_specific_settings.gecko.id and currently uses background scripts or event pages alongside standard scripting and activeTab permissions. Keeping your DOM analyzer inside a self-contained content script injected via chrome.scripting.executeScript (with browser.* fallback) makes the extension work seamlessly across both engines with zero external runtime dependencies.

Wrap-Up & Links

All 8 cloud actors and both browser extensions are live right now:

If you are building data pipelines, e-commerce integrations, or custom web applications and want to chat, feel free to reach out through my portfolio or GitHub!

Top comments (0)