DEV Community: Charles

I stopped fighting Amazon's anti-bot wall — and built a scoring engine instead

Charles — Wed, 24 Jun 2026 03:20:59 +0000

I stopped fighting Amazon's anti-bot wall — and built a scoring engine instead

I spent way too long trying to scrape Amazon product pages in real time. Proxies got CAPTCHA'd. Search pages returned 503s. eBay blocked me too. The usual arms race.

Then I realized I was solving the wrong problem.

The insight: scraping and analysis are different products

The scraping is fragile and commoditized — proxies rotate, layouts change, anti-bot vendors (DataDome, PerimeterX, Cloudflare) escalate. Everyone fights the same war.

The analysis — "which of these 100k products is actually worth sourcing?" — is durable. The scoring logic doesn't care whether the data came from a scraper, an API, a CSV export, or a partner feed.

So I split them. The scoring engine is the asset. The data source is a swappable adapter.

The engine

ecom-intel-engine (MIT) takes normalized product records and returns a 0–100 opportunity score across four dimensions:

demand — log-scaled review count (proven sales)
margin — price headroom (sweet spot ~$20–60)
quality — rating fit (4.0–4.8 ideal; suspiciously perfect penalized)
competition — reviews vs. category median (below median = an opening)

const { analyzeDataset } = require('ecom-intel-engine');

const products = [
  { asin: 'B01', title: 'Vitamin C Serum', price: 24.99, rating: 4.6, reviews: 8200, category: 'Skincare' },
  { asin: 'B02', title: 'Jade Roller', price: 9.99, rating: 4.3, reviews: 120, category: 'BeautyTools' },
];

const { summary } = analyzeDataset(products);
console.log(summary.topOpportunities);
// [{ id: 'B01', title: 'Vitamin C Serum', score: 82, category: 'Skincare' }, ...]

No scraping inside the package. Zero runtime dependencies for the core scoring path.

Trend detection needs history, not a single scrape

A one-shot scrape gives you a snapshot. To know what's trending, you need a time series. So the package ships a tiny snapshot store that accumulates review/price history across runs and de-dupes by day:

const { store, analyzeDataset } = require('ecom-intel-engine');

let s = store.loadStore('store.json');
store.ingest(s, freshBatch, { source: 'my-source' });   // run daily
store.saveStore('store.json', s);

const { products } = analyzeDataset(store.toDataset(s));
// products now carry trendingScore + reviewVelocityPerDay

A product only gets flagged isTrending if it had a real baseline (≥100 reviews) and grew ≥50% — so you don't get fooled by "5 → 12 reviews = +140%!" noise.

Where the data comes from (when Amazon won't cooperate)

Since real-time scraping kept getting walled, I pointed the adapter at a public, CC-licensed Amazon products dataset (117k rows on HuggingFace) instead. Same engine, stable input, no proxy war. When you do have a working scraper or a paid API (Keepa, Rainforest, PA-API), you just write a 20-line adapter and the engine doesn't change.

Takeaway

If your scraper keeps getting blocked, ask whether scraping is even the valuable part. Often the durable, sellable thing is the analysis layer — and that layer should never be coupled to where the bytes came from.

Engine: npm · MIT licensed.

I Scanned 8 Popular Sites for Bot Protection — Here's What Actually Stops Scrapers

Charles — Mon, 22 Jun 2026 08:04:21 +0000

If you've ever built a web scraper, you know the pain: your code works perfectly in dev, then the moment you point it at a real production site — blocked. 403s, CAPTCHA walls, infinite "checking your browser" loops.

After spending an embarrassing number of hours fighting anti-bot systems (and writing about it here), I got curious about the other side of the table:

If I were defending a site, would I actually know how strong my bot protection is?

Most engineers don't. So I built a tool to find out — and ran it against 8 well-known sites. The results were more revealing than I expected.

What "anti-bot protection" actually means

When people say a site is "protected," they usually mean one of these layers is in play:

Edge WAF / CDN — Cloudflare, Akamai, AWS WAF, Imperva. These fingerprint your TLS handshake and request headers before your code even reaches the app.
Behavioral bot managers — PerimeterX (now HUMAN), DataDome, F5. These score your behavior: mouse movement, timing, navigation patterns.
Challenge layers — reCAPTCHA, hCaptcha, JS challenges. The visible "prove you're human" walls.

Here's the thing most people miss: these leave fingerprints. A site using Cloudflare leaks cf-ray headers. PerimeterX injects telltale cookies. DataDome sets specific response signatures. You don't need to bypass the protection to detect it — the defense announces itself.

What I found scanning real sites

I checked response headers and body signatures across a handful of sites. A few patterns stood out:

E-commerce marketplaces were all over the map. Some had aggressive PerimeterX/DataDome behavioral scoring. Others? Basically wide open — full product data, prices, seller info, all extractable with a standard browser fingerprint.
Sites that felt secure often weren't. A polished UI says nothing about backend protection. I saw consumer brands with zero detectable bot defense.
The strongest defenders layered edge WAF + behavioral scoring + challenges. That combination is genuinely hard — it triggers human-verification friction even for legitimate automation.

The uncomfortable takeaway: a huge number of production sites have no meaningful bot protection at all. If you run one, "we haven't been scraped yet" is not a strategy.

How detection works (the useful part)

You can build a basic detector yourself. The core idea:

function detectFromHeaders(headers) {
  const h = {};
  for (const [k, v] of Object.entries(headers)) h[k.toLowerCase()] = v;

  const found = [];
  if (h['cf-ray'] || h['server']?.includes('cloudflare'))
    found.push('Cloudflare');
  if (h['x-amz-cf-id'] || h['server']?.includes('awselb'))
    found.push('AWS WAF/Shield');
  if (h['server']?.includes('Incapsula'))
    found.push('Imperva/Incapsula');
  // PerimeterX, DataDome, Akamai, F5 each have their own tells...
  return found;
}

Layer on top of that:

Body analysis — challenge pages have recognizable markers (_px, datadome, cf-challenge).
Status patterns — a 403 with a challenge body vs. a clean 200 tells you a lot.
Confidence scoring — combine signals into a single 0–100 "protection score" instead of a binary yes/no.

This is exactly the logic I packaged into a tool.

The free tool

I put this into a small service: paste a URL, get back which anti-bot systems it's running, a protection score (0–100), and prioritized recommendations if you're underprotected.

It's free for a single scan — no signup wall to see the verdict:

👉 Run a free bot-protection scan

It tests for the big nine: Cloudflare, AWS WAF, Imperva, Akamai, PerimeterX, DataDome, F5, reCAPTCHA, and hCaptcha — then tells you where you stand.

If you run a site with real data worth protecting (pricing, inventory, user content), it's worth 30 seconds to see what an attacker would see first.

What I'd love to know

If you scan your own site — what did you get? Drop the protection score in the comments. I'm especially curious how many "0 / no protection" results show up, because in my testing it was way more than I'd have guessed.

And if you're on the offensive side building scrapers: detecting the defense layer first saves enormous time. Know what you're up against before you write a single line of bypass logic.

Built on top of XCrawl for the fetch layer. Detection logic is open to feedback — tell me what signatures I'm missing.

How I Built a Real-Time Amazon Price Tracker That Actually Works

Charles — Mon, 08 Jun 2026 06:39:04 +0000

How I Built a Real-Time Amazon Price Tracker That Actually Works

Most Amazon price tracking tools break within days. Amazon blocks datacenter IPs, changes HTML structure randomly, and throws CAPTCHAs at anything that looks like a bot. After building dozens of scraping systems for clients, I've learned what actually works.

Here's the architecture I've used for 2+ years without a single day of downtime.

The Problem with Basic Scrapers

The typical approach: run a Python script via cron job, scrape the price, store it in a CSV. Simple, right? Except:

Amazon detects datacenter IPs → 403 errors within hours
HTML structure changes without notice → your parser breaks silently
Rate limits hit → your IP gets temp-banned for 24 hours
No error handling → you don't even know it's broken until you check manually

The fix isn't a better parser. It's a complete architecture rethink.

Architecture Overview

┌─────────────┐    ┌──────────────┐    ┌────────────────┐
│  Scheduler  │───▶│  XCrawl API   │───▶│  Data Store     │
│  (cron/utc) │    │  (residential │    │  (Airtable/     │
│             │    │   proxies)   │    │   Notion)       │
└─────────────┘    └──────────────┘    └────────────────┘
                          │
                          ▼
                   ┌──────────────┐
                   │  Alert Logic  │
                   │  (price drop │───▶ Email/Telegram
                   │   detection) │
                   └──────────────┘

Step 1: Residential Proxies (Non-Negotiable)

Datacenter IPs are immediately flagged. Residential IPs rotate through real home connections — Amazon sees them as normal shoppers.

With XCrawl, every request automatically rotates through residential proxies across 50+ countries:

const { XCrawlScraper } = require('xcrawl-scraper');

const xcrawl = new XCrawlScraper({
  apiKey: process.env.XCRAWL_API_KEY,
  // Automatically rotates residential proxies
});

async function getPrice(url) {
  const result = await xcrawl.scrapeMarkdown(url, {
    render: true, // Handle JavaScript-rendered pages
  });

  // Extract price from markdown
  const priceMatch = result.data.markdown.match(/\$[\d,]+\.?\d*/);
  return priceMatch ? priceMatch[0] : null;
}

Step 2: Handle Structure Changes Gracefully

Amazon changes HTML constantly. The solution: extract multiple possible formats and validate.

async function parsePrice(markdown) {
  const patterns = [
    /\$[\d,]+\.?\d{2}/,        // $129.99
    /USD\s*([\d.]+)/i,          // USD 129.99
    /price.*?([\d.]+)/i,        // "price":129.99
    /class="a-price-whole">(\d+)/ // HTML structure
  ];

  for (const pattern of patterns) {
    const match = markdown.match(pattern);
    if (match) return parseFloat(match[1].replace(/,/g, ''));
  }

  // Don't fail silently — alert yourself
  sendAlert('Price parse failed for: ' + url);
  return null;
}

Step 3: Reliable Scheduling

I run price checks every 6 hours (not more — to avoid detection). Use UTC cron to avoid timezone bugs:

# Every 6 hours, UTC
0 */6 * * * /usr/bin/node /app/price-tracker.js >> /var/log/cron.log 2>&1

Key trick: always store checked_at timestamp in UTC. When comparing prices across timezones, this prevents false alerts.

Step 4: Alert Logic

A price drop is only interesting if it's significant. Track the 30-day average and alert only on meaningful changes:

function shouldAlert(currentPrice, history) {
  if (history.length < 7) return false; // Need at least a week of data

  const avg30d = history.slice(-30).reduce((a, b) => a + b.price, 0) / Math.min(history.length, 30);
  const change = ((currentPrice - avg30d) / avg30d) * 100;

  return change <= -5; // Only alert on 5%+ drop
}

Results After 6 Months

Metric	Value
Uptime	99.4%
False alerts	3 (out of 140+)
Successful scrapes	2,847
Countries rotated through	23

The Secret Nobody Tells You

The most important part isn't the scraping — it's the data storage. Store raw markdown (not just parsed prices) so you can re-parse if your extraction logic changes. I learned this the hard way when a major Amazon UI change broke my price parser for 3 weeks before I realized I had all the raw data and could rebuild the parser retroactively.

Want the full working code? I open-sourced the tracker on GitHub — link in bio. Questions? Drop them below.

The Anti-Bot Detection Checklist I Use Before Every Scraping Project

Charles — Mon, 08 Jun 2026 06:36:11 +0000

The Anti-Bot Detection Checklist I Use Before Every Scraping Project

Every scraping project I take on starts with this checklist. Not because I'm paranoid — but because I've learned the hard way that production scrapers fail silently. They return 200 OK with garbage data, or they get rate-limited so gradually you don't notice for days.

This is the systematic approach I've refined over 50+ scraping projects.

Pre-Scraping: Know Your Target

1. Identify the CDN and Protection Stack

Before writing a single line of code, check what you're up against:

# Check CDN and headers
curl -I https://target-site.com

# Look for these common protection headers:
# X-Engine: akamai-html-protection
# X-Served-By: DataDome
# cf-ray: Cloudflare
# X-Bot-Status: blocked

Common protection platforms:

Cloudflare → Look for cf-ray and __cfduid cookies
DataDome → Look for datadome in headers or scripts
PerimeterX → Look for _pxff cookies
Akamai → Look for akamai-html-protection headers

2. Check Robots.txt Respectfully

curl https://target-site.com/robots.txt | grep -v "^#"

Don't take this as gospel — but it's a good signal. If they explicitly disallow your use case, that's a flag.

3. Map the Site's JavaScript Rendering

Some sites are fully static (fast, easy). Others render everything with JavaScript (need Playwright/Puppeteer). Check:

// Quick check - fetch raw HTML vs rendered content
// If they differ significantly, you need JS rendering

const https = require('https');
const html = await fetch('https://target.com').then(r => r.text());
const hasAngularVueReact = /ng-app|vue|react|__NEXT_DATA__/i.test(html);
console.log('Needs JS rendering:', hasAngularVueReact);

Code-Time: Defensive Patterns

4. Rotate User Agents

const USER_AGENTS = [
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120 Safari',
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120 Edge/120',
  'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/120 Firefox/120',
  // Add 10-15 more realistic user agents
];

function randomUA() {
  return USER_AGENTS[Math.floor(Math.random() * USER_AGENTS.length)];
}

Never use a single UA string. Rotate through 10+ realistic ones.

5. Respect Retry-After Headers

async function scrapeWithRetry(url, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    const response = await fetch(url, {
      headers: { 'User-Agent': randomUA() }
    });

    if (response.status === 429) {
      const retryAfter = response.headers.get('Retry-After') || 60;
      console.log(`Rate limited. Waiting ${retryAfter}s...`);
      await sleep(retryAfter * 1000);
      continue;
    }

    return response;
  }
  throw new Error('Max retries exceeded');
}

6. Detect Block Patterns Early

function detectBlock(response) {
  const html = response.text();

  // Common block signals
  if (html.includes('Access Denied')) return 'aws-waf';
  if (html.includes('captcha')) return 'captcha';
  if (html.includes('Please enable cookies')) return 'cloudflare';
  if (html.length < 1000 && html.includes('checking your browser')) return 'cloudflare-js';
  if (html.includes('datadome')) return 'datadome';

  return null;
}

7. Add Random Delays Between Requests

function randomDelay(min = 2000, max = 7000) {
  return Math.floor(Math.random() * (max - min) + min);
}

async function scrapeWithDelay(url) {
  await sleep(randomDelay());
  return scrape(url);
}

Infrastructure: Proxy Rotation

8. Use Residential Proxies (Not Datacenter)

This is the most impactful single change you can make:

Proxy Type	Block Rate	Cost	Speed
Datacenter	70-90% on protected sites	Cheap	Fast
Rotating Residential	5-15% on protected sites	$$	Medium
ISP Static	<5%	$$$	Fast

For anything beyond hobby projects, residential proxy rotation is worth the cost. With XCrawl's residential network:

// One line change — everything else stays the same
const xcrawl = new XCrawlScraper({ apiKey: process.env.XCRAWL_API_KEY });
// No more managing proxy lists, rotations, or bans

9. Sticky Sessions for Batching

When scraping a single site multiple times, use sticky sessions so you appear as the same user:

// XCrawl handles this automatically
const result = await xcrawl.scrape('https://site.com/page', {
  stickySession: true // Same proxy for 2 minutes
});

Validation: Before Going Live

10. Validate Data Quality

Never assume a 200 response means good data:

function validateData(data) {
  const required = ['title', 'price', 'url'];
  const missing = required.filter(f => !data[f]);

  if (missing.length > 0) {
    console.warn('Missing fields:', missing.join(', '));
    return false;
  }

  if (data.price && typeof data.price !== 'number') {
    console.warn('Invalid price type');
    return false;
  }

  return true;
}

11. Health Check Monitoring

Set up automated health checks that alert you when your scraper starts returning garbage:

// Run this every hour
async function healthCheck() {
  const testUrl = 'https://target-site.com/product-page';
  const result = await scrape(testUrl);

  const blockType = detectBlock(result);
  if (blockType) {
    sendAlert(`Scraper blocked by ${blockType}!`);
    return false;
  }

  if (!validateData(result.parsed)) {
    sendAlert('Scraper returning invalid data!');
    return false;
  }

  return true;
}

12. Always Store Raw HTML

This is the most overlooked step. Store every response as raw HTML before parsing:

async function scrapeAndStore(url) {
  const response = await fetch(url);
  const raw = await response.text();

  // Store raw for debugging
  await db.rawResponses.insert({
    url,
    raw_html: raw,
    timestamp: new Date(),
    status: response.status
  });

  // Then parse
  const parsed = parseHTML(raw);
  return parsed;
}

When your parser breaks (and it will), you'll thank yourself for the raw data.

The Full Picture

A production-ready scraper isn't just code — it's a system:

Monitoring → Alerting → Health Checks → Data Validation → Backup Parser
     ↑           ↑              ↑                ↑
  Residential Proxies ──────── Sticky Sessions ──── Error Handling

Quick Wins

If you only implement three things from this list:

Residential proxies (biggest win)
Block detection (prevents silent failures)
Store raw HTML (enables debugging)

Everything else is incremental improvement.

Questions about specific anti-bot systems? I've dealt with all of them — drop a comment.

Why Your Web Scraper Gets Blocked: A Production Architecture That Survives

Charles — Mon, 08 Jun 2026 06:36:06 +0000

Why Your Web Scraper Gets Blocked: A Production Architecture That Survives

Most web scrapers do not fail because the CSS selector was wrong.

They fail because the system was designed as a script:

Fetch a page
Parse the HTML
Retry if it fails
Hope tomorrow looks like today

That works for demos. It breaks in production.

If you are scraping protected sites, blocking is not a random runtime event. It is a design constraint. You need an architecture that assumes failure, measures data quality, and adapts before the whole pipeline goes dark.

This is the architecture pattern I use now for production scraping systems.

The real reason scrapers get blocked

Most teams focus on the visible error:

403 Forbidden
429 Too Many Requests
CAPTCHA page
Cloudflare / Akamai / PerimeterX challenge
Empty HTML shell from a JavaScript app

But those are symptoms. The actual causes usually sit one layer deeper.

1. Your traffic identity looks wrong

A datacenter IP sending repeated requests to product pages, search pages, or profile pages does not look like normal human traffic.

Even if your headers are perfect, the IP reputation may already be enough to classify the request as automation.

Common signals include:

IP ASN / hosting provider
request velocity
repeated URL patterns
missing browser behavior
inconsistent geo / language / timezone hints
known proxy ranges

2. Your scraper has no pacing model

A human does not open 500 product pages in 90 seconds.

Many scrapers use a simple concurrency value like this:

const concurrency = 20;

That is easy to reason about, but it is not how websites experience your traffic. The site sees request bursts, repeated paths, and abnormal timing.

A production scraper needs pacing rules per domain, per route type, and sometimes per account/session.

3. Your parser trusts successful responses

A 200 OK response can still be useless.

It might be:

a login page
a CAPTCHA page
a soft-block message
an empty JavaScript shell
a localized version missing the fields you expect
a page template that changed yesterday

If your pipeline treats every 200 as valid data, your database will slowly fill with garbage.

4. Your system cannot explain failures

A scraper that fails loudly is annoying.

A scraper that fails silently is expensive.

When the business asks, "Why did yesterday's pricing data drop by 40%?", you need more than "the script ran successfully." You need request logs, response samples, validation scores, and failure categories.

The production architecture

Here is the high-level design:

URL Queue
   │
   ▼
Scheduler / Rate Policy
   │
   ▼
Fetch Layer ──► Residential proxy / browser rendering / retries
   │
   ▼
Raw Response Store
   │
   ▼
Parser
   │
   ▼
Validation Layer
   │
   ├── valid data ──► Database / API / export
   │
   └── invalid data ──► Review queue + alerting
   │
   ▼
Monitoring Dashboard

The important idea is separation of concerns.

Your parser should not manage proxies. Your retry logic should not decide whether a product is valid. Your database should not be the first place you discover a block page.

Each layer has one job.

Layer 1: URL queue

Do not throw thousands of URLs directly at a worker.

Use a queue with metadata:

const job = {
  url: 'https://example.com/product/123',
  domain: 'example.com',
  type: 'product',
  priority: 'normal',
  lastScrapedAt: null,
  attempts: 0,
};

That metadata allows you to make better decisions:

search pages can be slower than product pages
failed URLs can be retried later instead of immediately
high-value URLs can get better proxy/rendering options
stale URLs can be refreshed without scraping everything

A queue also makes the system restartable. If a worker crashes, the job is not lost.

Layer 2: scheduler and pacing policy

This layer decides when a URL should be fetched.

A simple version:

const policy = {
  'example.com': {
    concurrency: 2,
    minDelayMs: 3000,
    maxDelayMs: 9000,
    maxRetries: 3,
  },
};

function randomDelay(min, max) {
  return min + Math.floor(Math.random() * (max - min));
}

A better version also tracks recent failures:

function shouldSlowDown(stats) {
  return stats.rateLimitRate > 0.05 || stats.captchaRate > 0.02;
}

function nextDelay(baseDelay, stats) {
  if (shouldSlowDown(stats)) return baseDelay * 4;
  if (stats.successRate > 0.95) return baseDelay;
  return baseDelay * 2;
}

The scheduler should adapt to what the site is telling you.

If the block rate rises, slow down. If validation starts failing, sample and inspect. Do not keep hammering the site with the same pattern.

Layer 3: fetch layer

The fetch layer is where identity and rendering are handled.

For simple static pages, an HTTP fetch may be enough. For JavaScript-heavy pages, you may need browser rendering. For protected sites, you may need residential proxies.

The key is to hide that complexity behind one interface:

async function fetchPage(url, options = {}) {
  const result = await xcrawl.scrapeMarkdown(url, {
    render: options.render ?? false,
    country: options.country,
  });

  return {
    status: result.status,
    finalUrl: result.finalUrl,
    markdown: result.data?.markdown || '',
    html: result.data?.html || '',
    fetchedAt: new Date().toISOString(),
  };
}

Your business logic should not care whether the request used a proxy, a rendered browser, or a fallback route. It should receive a normalized response.

Layer 4: block detection

Before parsing, classify the response.

function detectBlock(response) {
  const text = `${response.markdown}\n${response.html}`.toLowerCase();

  if (response.status === 429) return 'rate_limit';
  if (response.status === 403) return 'forbidden';
  if (text.includes('captcha')) return 'captcha';
  if (text.includes('access denied')) return 'access_denied';
  if (text.includes('enable javascript')) return 'needs_rendering';
  if (response.markdown.length < 300) return 'too_little_content';

  return null;
}

This does not need to be perfect. It just needs to be good enough to route failures correctly.

For example:

async function fetchWithFallback(url) {
  let response = await fetchPage(url, { render: false });
  let block = detectBlock(response);

  if (block === 'needs_rendering' || block === 'too_little_content') {
    response = await fetchPage(url, { render: true });
    block = detectBlock(response);
  }

  if (block) {
    return { ok: false, block, response };
  }

  return { ok: true, response };
}

This is much better than blindly retrying the same blocked request three times.

Layer 5: raw response storage

Always store the raw response before parsing.

await db.raw_pages.insert({
  url,
  status: response.status,
  finalUrl: response.finalUrl,
  markdown: response.markdown,
  html: response.html,
  fetchedAt: response.fetchedAt,
});

This feels boring until the day the target site changes its layout.

If you stored raw pages, you can fix the parser and reprocess yesterday's data without hitting the site again. If you only stored extracted fields, the data is gone.

Raw storage also gives you evidence when debugging:

Was the page blocked?
Did the selector fail?
Did the price format change?
Did the site serve a different layout by region?

Layer 6: parser

The parser should be defensive.

Bad parser:

const price = $('.price').text();

Better parser:

function extractPrice(markdown) {
  const patterns = [
    /\$\s?([0-9,]+(?:\.\d{2})?)/,
    /USD\s?([0-9,]+(?:\.\d{2})?)/i,
    /price[^0-9]{0,20}([0-9,]+(?:\.\d{2})?)/i,
  ];

  for (const pattern of patterns) {
    const match = markdown.match(pattern);
    if (match) {
      return Number(match[1].replace(/,/g, ''));
    }
  }

  return null;
}

Returning null is often better than throwing. The validation layer can decide whether missing data is acceptable.

Layer 7: validation

Validation is where you protect the database.

function validateProduct(product) {
  const checks = {
    hasTitle: Boolean(product.title && product.title.length > 8),
    hasPrice: typeof product.price === 'number' && product.price > 0,
    hasUrl: Boolean(product.url && product.url.startsWith('https://')),
    priceLooksReasonable: product.price == null || product.price < 10000,
  };

  const passed = Object.values(checks).filter(Boolean).length;
  const confidence = passed / Object.keys(checks).length;

  return {
    valid: confidence >= 0.75,
    confidence,
    checks,
  };
}

The database should only receive records that pass validation.

Everything else should go to a review queue:

if (!validation.valid) {
  await db.review_queue.insert({ url, product, validation, rawPageId });
  return;
}

await db.products.upsert(product);

This one layer prevents a huge class of silent data quality problems.

Layer 8: monitoring

Track the metrics that actually matter:

Effective scrape rate = valid records / total attempts

Also track:

success rate by domain
block rate by domain
validation failure rate
average response time
retry count per URL
rendering usage rate
cost per valid record

Requests per second is not the goal. Valid, fresh, usable data is the goal.

A slower scraper with a 96% valid-data rate is usually better than a fast scraper that silently returns garbage 40% of the time.

A practical retry strategy

Retries should be based on failure type.

async function handleFailure(job, failure) {
  switch (failure.block) {
    case 'rate_limit':
      return requeue(job, { delayMs: 30 * 60 * 1000 });

    case 'captcha':
    case 'access_denied':
      return requeue(job, { delayMs: 2 * 60 * 60 * 1000, useNewIdentity: true });

    case 'needs_rendering':
      return requeue(job, { delayMs: 0, render: true });

    case 'too_little_content':
      return requeue(job, { delayMs: 10 * 60 * 1000, render: true });

    default:
      return markFailed(job, failure);
  }
}

Blind retries create more suspicious traffic. Typed retries reduce damage.

Production checklist

Before calling a scraper production-ready, I want these boxes checked:

[ ] per-domain pacing rules
[ ] proxy / identity strategy
[ ] rendering fallback for JavaScript pages
[ ] block detection
[ ] raw response storage
[ ] defensive parser
[ ] validation score
[ ] review queue for low-confidence records
[ ] dashboard / alerts
[ ] cost per valid record tracked

If any of these are missing, the system might still work — but you will not know why it fails when it fails.

Final thought

Scraping is not just parsing HTML. Production scraping is distributed systems work with adversarial constraints.

The winning architecture is not the one that sends the most requests. It is the one that turns the highest percentage of attempts into valid, explainable, reusable data.

If your scraper keeps getting blocked, do not start by adding another retry loop.

Start by asking: which layer of the architecture is missing?

Node.js Web Scraping Best Practices: Lessons From Production Pipelines

Charles — Fri, 05 Jun 2026 01:59:29 +0000

Node.js Web Scraping Best Practices: Lessons From Production Pipelines

Node.js is a great choice for web scraping.

The ecosystem is mature, async I/O is a natural fit, and you can move quickly with tools like fetch, cheerio, Playwright, Puppeteer, and scraping APIs.

But most Node.js scrapers fail for the same reason: they are written like one-off scripts and then quietly promoted into production.

This post is a practical checklist for building scrapers that survive real usage.

1. Start with a queue, not a loop

The first version of a scraper often looks like this:

for (const url of urls) {
  const html = await fetch(url).then(r => r.text());
  const data = parse(html);
  await save(data);
}

That is fine for a test. It is fragile in production.

Use jobs instead:

const job = {
  url: 'https://example.com/products/123',
  type: 'product',
  priority: 'normal',
  attempts: 0,
  createdAt: new Date().toISOString(),
};

A queue gives you:

retries without losing work
priority handling
restartability
deduplication
visibility into failures
safer concurrency control

You can start simple with a database table or a JSON-backed queue. You do not need Kafka on day one. You just need to stop treating URLs as a temporary in-memory array.

2. Control concurrency per domain

Global concurrency is too blunt.

// Too simple for production
const concurrency = 20;

Different domains tolerate different request patterns. Even within one domain, product pages, search pages, and detail pages may need different pacing.

A better structure:

const domainPolicy = {
  'example.com': {
    concurrency: 2,
    minDelayMs: 3000,
    maxDelayMs: 9000,
  },
  'another-site.com': {
    concurrency: 1,
    minDelayMs: 8000,
    maxDelayMs: 20000,
  },
};

function randomDelay(min, max) {
  return min + Math.floor(Math.random() * (max - min));
}

Then make your workers respect the policy:

async function waitForDomainPolicy(domain) {
  const policy = domainPolicy[domain] || domainPolicy.default;
  await sleep(randomDelay(policy.minDelayMs, policy.maxDelayMs));
}

Your goal is not maximum requests per second. Your goal is stable, valid data over time.

3. Use the right fetch strategy for the page

Not every page needs a browser.

A practical hierarchy:

HTTP fetch for static pages
HTML parsing with Cheerio when the markup is available
Browser rendering when JavaScript is required
Residential proxy / scraping API when the site is protected

Example wrapper:

async function fetchPage(url, options = {}) {
  if (options.render) {
    return await fetchWithBrowser(url);
  }

  return await fetchWithHttp(url);
}

If you use a scraping API such as XCrawl, keep it behind the same wrapper:

async function fetchWithScrapingApi(url, options = {}) {
  const result = await xcrawl.scrapeMarkdown(url, {
    render: options.render ?? false,
    country: options.country,
  });

  return {
    status: result.status,
    markdown: result.data?.markdown || '',
    html: result.data?.html || '',
    finalUrl: result.finalUrl,
  };
}

The rest of your application should not care which fetch method was used.

4. Detect blocks before parsing

Do not send every response directly into the parser.

A blocked response can still be 200 OK.

function detectBlock(response) {
  const text = `${response.html || ''}\n${response.markdown || ''}`.toLowerCase();

  if (response.status === 403) return 'forbidden';
  if (response.status === 429) return 'rate_limited';
  if (text.includes('captcha')) return 'captcha';
  if (text.includes('access denied')) return 'access_denied';
  if (text.includes('enable javascript')) return 'needs_rendering';
  if (text.length < 500) return 'too_little_content';

  return null;
}

Then route failures by type:

const response = await fetchPage(job.url);
const block = detectBlock(response);

if (block === 'needs_rendering') {
  return requeue(job, { render: true });
}

if (block === 'rate_limited') {
  return requeue(job, { delayMs: 30 * 60 * 1000 });
}

if (block) {
  return markBlocked(job, block);
}

Blind retries are a common way to make blocking worse. Typed retries are safer.

5. Store raw responses first

This is one of the highest-ROI habits in scraping.

async function storeRawPage(job, response) {
  return db.rawPages.insert({
    url: job.url,
    status: response.status,
    html: response.html,
    markdown: response.markdown,
    finalUrl: response.finalUrl,
    fetchedAt: new Date(),
  });
}

Why it matters:

you can re-parse old pages after fixing a parser
you can inspect block pages later
you can prove whether missing data came from the site or your code
you reduce repeated requests during debugging

Storage is usually cheaper than re-scraping.

6. Write parsers that degrade gracefully

A brittle parser:

const price = Number($('.price').text().replace('$', ''));

A more defensive parser:

function extractPrice(text) {
  const patterns = [
    /\$\s?([0-9,]+(?:\.\d{2})?)/,
    /USD\s?([0-9,]+(?:\.\d{2})?)/i,
    /price[^0-9]{0,20}([0-9,]+(?:\.\d{2})?)/i,
  ];

  for (const pattern of patterns) {
    const match = text.match(pattern);
    if (match) {
      return Number(match[1].replace(/,/g, ''));
    }
  }

  return null;
}

Returning null is often better than throwing. The validation layer can decide whether the record is good enough.

7. Validate before writing to your main tables

A successful parse is not the same as valid data.

function validateProduct(product) {
  const checks = {
    hasTitle: Boolean(product.title && product.title.length >= 8),
    hasPrice: typeof product.price === 'number' && product.price > 0,
    hasUrl: Boolean(product.url?.startsWith('https://')),
    priceReasonable: product.price == null || product.price < 10000,
  };

  const passed = Object.values(checks).filter(Boolean).length;
  const confidence = passed / Object.keys(checks).length;

  return {
    valid: confidence >= 0.75,
    confidence,
    checks,
  };
}

Only write valid records to the main table:

const validation = validateProduct(product);

if (!validation.valid) {
  await db.reviewQueue.insert({ product, validation, rawPageId });
  return;
}

await db.products.upsert(product);

This prevents silent corruption. Your scraper will still fail sometimes, but bad data will not quietly become business data.

8. Deduplicate aggressively

Do not scrape the same fresh URL again and again.

async function planJobs(urls) {
  const existing = await db.pages.findByUrls(urls);
  const sixHoursAgo = Date.now() - 6 * 60 * 60 * 1000;

  return urls.filter(url => {
    const record = existing.get(url);
    if (!record) return true;
    return new Date(record.fetchedAt).getTime() < sixHoursAgo;
  });
}

Deduplication reduces:

cost
block risk
processing time
noisy logs

Freshness should be a business decision. A stock price page and a company profile page do not need the same refresh interval.

9. Stream large jobs

If you process thousands of URLs, avoid loading everything into memory.

async function* scrapeStream(jobs) {
  for await (const job of jobs) {
    const response = await fetchPage(job.url, job.options);
    yield { job, response };
  }
}

for await (const { job, response } of scrapeStream(jobSource())) {
  await processResponse(job, response);
}

Generators make long-running jobs easier to control. They also pair well with backpressure, batching, and incremental checkpointing.

10. Add health checks

A scraper can look healthy while returning junk.

Create known test cases:

const healthChecks = [
  {
    url: 'https://example.com/products/test-product',
    expect: page => page.includes('Test Product'),
  },
];

async function runHealthChecks() {
  for (const check of healthChecks) {
    const response = await fetchPage(check.url, { render: true });
    const text = response.markdown || response.html || '';

    if (!check.expect(text)) {
      await sendAlert(`Health check failed for ${check.url}`);
      return false;
    }
  }

  return true;
}

Run health checks on a schedule. You want to know about broken extraction before your users or clients notice stale data.

11. Track the metrics that matter

Do not obsess over requests per second.

Track:

valid records per run
validation failure rate
block rate by domain
retry count by failure type
render fallback rate
cost per valid record
freshness lag
parser error rate

The most important metric is:

effective scrape rate = valid records / total attempts

A slower scraper with a high effective scrape rate is usually better than a fast scraper that fills your database with partial records.

12. Keep secrets out of code

Do not hardcode API keys.

const xcrawl = new XCrawlScraper({
  apiKey: process.env.XCRAWL_API_KEY,
});

Use environment variables, secret managers, or platform-provided secrets.

This matters more than people think because scraping projects often start as local scripts and later get copied into workers, cron jobs, dashboards, or client repos.

A simple production skeleton

Here is the shape I like:

async function processJob(job) {
  await waitForDomainPolicy(new URL(job.url).hostname);

  const response = await fetchPage(job.url, job.options);
  const rawPageId = await storeRawPage(job, response);

  const block = detectBlock(response);
  if (block) return handleBlock(job, block, rawPageId);

  const parsed = parsePage(response);
  const validation = validate(parsed);

  if (!validation.valid) {
    return db.reviewQueue.insert({ job, parsed, validation, rawPageId });
  }

  await db.records.upsert(parsed);
  return markDone(job);
}

It is not fancy. That is the point.

Good production scrapers are usually boring systems with strong boundaries:

queue
pacing
fetch
raw storage
parse
validate
monitor

Final checklist

Before you call a Node.js scraper production-ready, check this list:

[ ] jobs are stored outside memory
[ ] concurrency is controlled per domain
[ ] delays are configurable
[ ] JavaScript rendering is optional, not default
[ ] block pages are detected
[ ] raw responses are stored
[ ] parsers return structured partial data instead of crashing
[ ] validation happens before database writes
[ ] failed records go to a review queue
[ ] large jobs stream or checkpoint progress
[ ] health checks run on a schedule
[ ] metrics track valid data, not just requests
[ ] secrets come from environment variables

If you have these pieces, your scraper can still break — but it will break in a way you can understand and recover from.

That is the difference between a scraping script and a production scraping pipeline.

How to Build a Competitor Monitoring Pipeline in 30 Minutes

Charles — Fri, 05 Jun 2026 01:58:58 +0000

How to Build a Competitor Monitoring Pipeline in 30 Minutes

Stop checking competitor prices manually. Here's the exact stack I use to monitor 50+ competitor URLs in real-time, without getting blocked.

What We're Building

A system that:

Checks competitor URLs every 6 hours
Extracts prices, titles, and availability
Alerts you only when something significant changes
Doesn't get blocked by anti-bot protection

The Stack

XCrawl — residential proxies + scraping API
Airtable — free, simple data storage with nice UI
GitHub Actions — free cron scheduling
Telegram bot — instant alerts on your phone

Total cost: under $20/month for 50 URLs.

Step 1: Define What You're Monitoring

Create a simple Airtable table with columns:

url — competitor product page
name — competitor or product name
last_price — price from last check
last_checked — timestamp
change_alert — threshold % for alerts

Start with 10-20 URLs. You can always add more later.

Step 2: The Scraping Function

const { XCrawlScraper } = require('xcrawl-scraper');

const xcrawl = new XCrawlScraper({
  apiKey: process.env.XCRAWL_API_KEY,
});

async function checkCompetitor(url) {
  try {
    const result = await xcrawl.scrapeMarkdown(url, {
      render: true,
    });

    const markdown = result.data.markdown;

    const priceMatch = markdown.match(/\$[\d,]+\.?\d*/);
    const price = priceMatch
      ? parseFloat(priceMatch[0].replace(/[$,]/g, ''))
      : null;

    const titleMatch = markdown.match(/#\s+(.+)/);
    const title = titleMatch ? titleMatch[1].trim() : 'Unknown';

    return {
      url,
      price,
      title,
      checkedAt: new Date().toISOString(),
      success: true,
    };
  } catch (err) {
    return {
      url,
      price: null,
      title: null,
      checkedAt: new Date().toISOString(),
      success: false,
      error: err.message,
    };
  }
}

Step 3: Alert Logic

You don't want to be alerted every time a price changes by $0.01. You want to know when something significant happens:

function shouldAlert(lastPrice, currentPrice, threshold = 5) {
  if (!lastPrice || !currentPrice) return false;

  const change = Math.abs(((currentPrice - lastPrice) / lastPrice) * 100);
  return change >= threshold;
}

function formatAlert(competitor, lastPrice, currentPrice) {
  const direction = currentPrice < lastPrice ? 'DROPPED' : 'INCREASED';
  const change = Math.abs(((currentPrice - lastPrice) / lastPrice) * 100);

  return direction + ' ' + competitor.name + '
' +
    'Price: $' + lastPrice + ' -> ' + currentPrice + ' (' + change.toFixed(1) + '% change)
' +
    'URL: ' + competitor.url;
}

Step 4: GitHub Actions Scheduling

Create .github/workflows/competitor-check.yml:

name: Competitor Price Monitor
on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:        # Manual trigger

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install xcrawl-scraper airtable

      - name: Run competitor checks
        env:
          XCRAWL_API_KEY: ${{ secrets.XCRAWL_API_KEY }}
          AIRTABLE_API_KEY: ${{ secrets.AIRTABLE_API_KEY }}
        run: node competitor-check.js

      - name: Send alert to Telegram
        if: matrix.changed == true
        env:
          TG_BOT_TOKEN: ${{ secrets.TG_BOT_TOKEN }}
          TG_CHAT_ID: ${{ secrets.TG_CHAT_ID }}
        run: |
          curl -s -X POST "https://api.telegram.org/bot$TG_BOT_TOKEN/sendMessage" -d "chat_id=$TG_CHAT_ID" -d "text=${{ matrix.alertMessage }}"

Step 5: The Full Check Loop

async function runChecks() {
  const competitors = await airtable.fetchCompetitors();
  const alerts = [];

  for (const competitor of competitors) {
    const result = await checkCompetitor(competitor.url);

    if (result.success) {
      await airtable.updateRecord(competitor.id, {
        last_price: result.price,
        last_checked: result.checkedAt,
        last_title: result.title,
      });

      if (shouldAlert(competitor.last_price, result.price)) {
        alerts.push(formatAlert(competitor, competitor.last_price, result.price));
      }
    } else {
      console.log('Failed: ' + competitor.url + ' - ' + result.error);
    }

    // Be respectful - don't hammer the servers
    await sleep(2000 + Math.random() * 3000);
  }

  if (alerts.length > 0) {
    await sendTelegramAlerts(alerts);
  }

  console.log('Checked ' + competitors.length + ' competitors. ' + alerts.length + ' alerts.');
}

What This Actually Looks Like in Practice

Here's the alert you'd get on your phone:

DROPPED Amazon Basics Mouse
Price: $29.99 -> $24.99 (16.7% change)
URL: amazon.com/dp/B07ZVPQR5M

That's it. No dashboards to check. No manual research. Just the signal you need to act.

Key Decisions That Made This Work

Residential proxies — datacenter IPs get blocked immediately on Amazon, Best Buy, etc.
6-hour check interval — fast enough to catch changes, slow enough to not look like a bot
Store everything in Airtable — the UI makes it easy to spot patterns over time
Telegram over email — faster, and your phone will buzz to wake you up if needed

The ROI

My client monitors 47 competitor URLs across 8 product categories. In the first month:

Caught 3 competitor price drops within hours
Adjusted their own pricing proactively
Estimated incremental revenue: $2,400 that month

System cost: $49/month (XCrawl) + free (GitHub Actions, Airtable, Telegram).

Building something similar? I've open-sourced the monitoring template. Link in bio. Questions below.

Why Your Web Scraper Gets Blocked (And the One Architecture That Doesn't)

Charles — Fri, 05 Jun 2026 01:57:23 +0000

Why Your Web Scraper Gets Blocked (And the One Architecture That Doesn't)

I've talked to dozens of developers who built what they thought was a robust scraping system — rotating IPs, retry logic, headless browsers — and still got blocked within days. They're frustrated, confused, and ready to give up.

The problem isn't their code. It's how they're thinking about the problem.

The Mental Model Mistake

Most developers approach scraping like this:

Write a parser for today's HTML structure
Add retry logic for 429 errors
Maybe use a proxy service
Run it daily and hope it works

This approach fails because it treats blocking as a runtime problem to solve. But blocking is actually a design problem to prevent.

Why Datacenter IPs Always Get Blocked

Here's the uncomfortable truth: if you're using datacenter IPs, you will get blocked on any site with even basic anti-bot measures.

Why? Because:

Google, Amazon, LinkedIn track IP ranges — they know which IPs belong to AWS, GCP, DigitalOcean, etc.
A datacenter IP doing 100 requests/day looks nothing like a human — humans browse, they don't scrape
Reputation systems — once one datacenter IP from a range gets flagged, the entire range is suspect

It's not personal. It's just math. Your scraper looks like a bot because it is one — from the target site's perspective.

The Architecture That Actually Works

After 50+ projects across Amazon, Google Maps, LinkedIn, Instagram, Twitter, and dozens of other platforms, I've converged on this:

The key insight: your code should never manage proxies. That complexity should be abstracted away. Your job is parsing logic, not IP rotation.

The Three Layers of Defense

Layer 1: Identity (Residential Proxies)

Your scraper's identity is the most important factor. One line with XCrawl:

const xcrawl = new XCrawlScraper({ apiKey: 'YOUR_KEY' });
// Every request automatically goes through residential IPs
// No proxy lists to manage, no rotation logic to write

Layer 2: Resilience (Smart Retry)

When a request fails, don't just retry. Analyze why it failed and adapt:

async function scrapeWithIntelligence(url, depth = 0) {
  if (depth > 3) return { error: 'max retries exceeded' };

  const result = await xcrawl.scrape(url);

  switch (detectBlockType(result)) {
    case 'captcha':
      await sleep(30000);
      return scrapeWithIntelligence(url, depth + 1);
    case 'rate-limit':
      const backoff = Math.pow(2, depth) * 5000;
      await sleep(backoff);
      return scrapeWithIntelligence(url, depth + 1);
    case 'cloudflare':
      return xcrawl.scrape(url, { render: true });
    case 'none':
      return result;
  }
}

Layer 3: Validation (Don't Trust the Response)

A 200 response doesn't mean valid data. Always validate:

function validateAmazonProduct(data) {
  const hasTitle = data.title && data.title.length > 10;
  const hasPrice = data.price && data.price > 0 && data.price < 10000;
  const hasASIN = data.asin && /^[A-Z0-9]{10}$/.test(data.asin);
  const hasImage = data.imageUrl && data.imageUrl.startsWith('https://');

  const confidence = [hasTitle, hasPrice, hasASIN, hasImage]
    .filter(Boolean).length / 4;

  return { valid: confidence >= 0.75, confidence };
}

Real Results

Here's data from one client's Amazon price tracking system (running 6+ months):

Metric	Before (datacenter)	After (residential)
Success rate	34%	97.3%
Daily check failures	15-20	0-2
Avg response time	2.3s	4.1s
Monthly cost	$23 (proxies)	$49 (XCrawl)

The residential proxy solution costs 2x more but achieves 3x the success rate.

The One Metric That Matters

Stop measuring "requests per second." The metric that actually matters is:

Effective Scrape Rate = Successful Valid Data / Total Attempts

Quality over quantity, every time.

What This Means For Your Project

If you're building a production scraping system:

Start with residential proxies — not because it's better, but because datacenter literally doesn't work on protected sites
Abstract away proxy management — your parser should call one function, not manage a list of IPs
Validate everything — 200 OK is not the same as valid data
Store raw responses — you will need to re-parse when structure changes

The One-Line Change

If you're using requests/cheerio/puppeteer directly and getting blocked:

// Before: every request looks like a bot
const response = await axios.get(url);

// After: every request looks like a human
const xcrawl = new XCrawlScraper({ apiKey: process.env.XCRAWL_API_KEY });
const result = await xcrawl.scrapeMarkdown(url);

Same interface for your parsing logic. Different result under the hood.

I help businesses build reliable data pipelines. If scraping is part of your business and it's giving you headaches, let's talk.

5 Things Every Developer Should Know About Web Scraping API Keys

Charles — Tue, 19 May 2026 04:04:13 +0000

5 Things Every Developer Should Know About Web Scraping API Keys

Don't leak your keys. Lessons from experience.

1. Never Hardcode

// BAD
const key = "xc-..."
// GOOD
const key = process.env.XCRAWL_API_KEY;

2. Use .env Files

Add to .gitignore immediately.

3. Rotate Monthly

Generate new key, revoke old one. Keep a lifecycle log.

4. Separate per Environment

dev-key, prod-key, ci-key. If one leaks, limited damage.

5. Monitor Usage

Check daily consumption. Spikes = possible leak.

Manage keys at dash.xcrawl.com

How to Auto-Fill Google Sheets with Web Scraped Data (No Coding Required)

Charles — Tue, 19 May 2026 04:03:40 +0000

How to Auto-Fill Google Sheets with Web Scraped Data

Want scraped data flowing into Google Sheets automatically?

Method 1: Google Apps Script

function scrapeAndFill() {
  const sheet = SpreadsheetApp.getActiveSheet();
  const options = {
    method: 'POST',
    headers: { 'X-API-Key': 'your-key' },
    payload: JSON.stringify({ url: 'https://example.com/products' })
  };
  const response = UrlFetchApp.fetch('https://run.xcrawl.com/v1/scrape', options);
  const data = JSON.parse(response);
  sheet.appendRow([data.name, data.price]);
}

Set to run on time trigger.

Method 2: n8n + Google Sheets

Build the workflow visually. HTTP node → transform → Google Sheets node.

Method 3: CLI → CSV → Import

npx xcrawl scrape "https://example.com" --output data.json

Pro Tips

Schedule off-peak hours (2-6 AM)
Always error-check before writing
Keep raw + processed sheets separate

Works with any automation tool: dash.xcrawl.com

Puppeteer vs Playwright vs Cheerio vs XCrawl: Which Web Scraping Tool to Use in 2026

Charles — Tue, 19 May 2026 04:03:07 +0000

Puppeteer vs Playwright vs Cheerio vs XCrawl: Which Web Scraping Tool to Use in 2026

Four tools, four different philosophies.

Comparison

Feature	Cheerio	Puppeteer	Playwright	XCrawl
Setup	5 min	15 min	15 min	2 min
JS rendering	❌	✅	✅	✅
Anti-bot	❌	❌ Basic	❌ Basic	✅ Built-in
CAPTCHA	❌	❌	❌	✅ Auto
Proxies	❌ DIY	❌ DIY	❌ DIY	✅ Auto
Price	Free	Free	Free	From $49/mo

When to Use What

Cheerio: Static HTML, simple parsing, you have proxy infra.

Puppeteer: Need CDP access, screenshots, already in Node.js.

Playwright: Cross-browser support, mobile emulation, E2E tests.

XCrawl: Production scraping, anti-bot bypass, want API simplicity.

Real-World Decision

One page once → Cheerio
Screenshots → Puppeteer
Testing → Playwright
1000 pages/day → XCrawl

XCrawl API — web scraping with built-in proxies, CAPTCHA solving, and JS rendering.

The Complete Guide to Real Estate Web Scraping in 2026

Charles — Tue, 19 May 2026 04:02:41 +0000

The Complete Guide to Real Estate Web Scraping in 2026

Real estate is one of the most competitive data markets. Here's how to do it right.

Why Scrape Real Estate?

Price monitoring — Track listing prices, drops, trends
Market analysis — Compare neighborhoods, cities, regions
Investment research — Identify undervalued properties
Competitive intelligence — Watch what other agents/brokers list

Major Platforms

Platform	Difficulty	Anti-Bot	Data Quality
Zillow	Hard	Cloudflare	Excellent
Realtor.com	Medium	Rate limits	Excellent
Redfin	Hard	JS rendering	Very Good
Rightmove (UK)	Easy	Basic	Good

Key Data Points

Price, address, bedrooms/bathrooms, sqft
Listing date, days on market
Price history (if available)
Tax assessment, HOA fees
Nearby schools, crime stats

Avoiding Common Pitfalls

IP Blocking: Real estate platforms are aggressive. Use residential proxies.

Data Accuracy: Some platforms hide data behind lazy loading. Ensure JS rendering.

Legal: Public listings are public information. Respect robots.txt and rate limits.

Sample Extraction

const result = await fetch('https://run.xcrawl.com/v1/ai-extract', {
  method: 'POST',
  headers: { 'X-API-Key': 'your-key' },
  body: JSON.stringify({
    url: 'https://www.zillow.com/homedetails/...',
    schema: {
      price: 'Current listing price',
      address: 'Street address',
      beds: 'Number of bedrooms',
      baths: 'Number of bathrooms',
      sqft: 'Square footage'
    }
  })
});

Built with XCrawl proxy API: dash.xcrawl.com