How to Detect and Block Price Scrapers by IP

A competitor's bot can pull your entire fare table or product catalog in the time it takes a real customer to pick a seat. It hits the same route search or category page every few seconds, reads your prices, and quietly undercuts you on a comparison site an hour later. Blocking it is harder than it sounds, because the bot you want to stop does not look like one request from one machine. It looks like a thousand requests from a thousand residential IPs.

This is a guide to the cheapest, fastest layer of that defense: scoring every incoming request by its IP before it touches your catalog. You will build middleware that flags datacenter traffic, VPNs, and residential proxies, assigns each request a risk score, and decides whether to serve, slow, challenge, or block. It runs in Node, and it is honest about the one thing IP scoring cannot do on its own.

TL;DR

• Price and fare scrapers rotate through datacenter and residential IPs, so no single signal catches them. Treat detection as layers.
• The IP layer is first because it is cheap and it kills the lazy majority: bulk scrapers running from AWS, Hetzner, and commercial proxy pools.
• Score each request with a 0-100 threat value plus specific flags (is_residential_proxy, is_cloud_provider, is_vpn), then tier your response instead of hard-blocking.
• Cache the verdict per IP. Scoring the same address on every request is a waste of budget and latency.
• Residential proxies that look like real ISP customers are where IP scoring runs out. That is a job for the behavioral layer above it, not a reason to skip the IP layer.

Datacenter and proxy detection stops the bulk scrapers immediately. Residential-proxy and VPN flags plus a risk score let you make graded decisions on the harder traffic. What you will not get is a magic boolean that ends scraping forever, and any tool that promises one is selling you something. By the end you will have a working IP-scoring layer and a clear picture of where the next layer has to take over.

Why price and fare scrapers are hard to stop

What the traffic actually looks like

Picture two real cases. A metasearch competitor wants your airline's fares, so it runs a bot that searches one popular route, say JFK to LHR, every thirty seconds, all day, from a rotating pool of IPs. Each individual request looks like a shopper checking a flight. In aggregate it is a machine reading your entire price surface. Now picture a flash sale on a retail site: an inventory-hoarding bot dumps every discounted SKU into carts to make them unavailable, then resells the reservation. Snowplow, which sees bot patterns across more than two million sites, describes the marketplace version as short-session crawlers that hit a single listing and disappear, which is exactly the shape that slips past rules built for slow human browsing.

The common thread is IP diversity. Modern scrapers do not hammer you from one address. They rent access to datacenter ranges and, increasingly, residential proxy networks that route requests through real consumer devices. DataDome, whose whole business is stopping this, puts it plainly: simple IP-based rate limiting fails against scrapers using residential proxies and rotation. That sentence is the reason this article is about scoring IPs, not banning them.

Why a single signal fails

Each obvious defense has an obvious counter, and scraper operators know all of them.

Rate limit per IP, and they rotate IPs. User-agent filtering catches the scraper that announces itself as python-requests, and misses the one that copies a real Chrome string, which is most of them. A static blocklist of known-bad IPs is stale the day after you build it. Even the biggest bot-management vendors agree on this point: HUMAN Security's framing is that no single method catches every bot, and each technique exists to cover the gaps in the others. That is not a disclaimer. It is the design principle.

The layered model

So you stack layers, cheapest and broadest first, most expensive and most precise last:

1. IP intelligence. Score the source address. Catches bulk datacenter and commercial-proxy scrapers for almost nothing. This article.
2. Rate limiting. Assume rotation and key limits on something better than a bare IP.
3. Behavioral signals. Request cadence, headers, interaction patterns. This is where residential-proxy scrapers that survived layer one get caught.
4. Honeypots. Traps a human never triggers and a naive crawler does.

The point of leading with IP is economics. A datacenter check costs one API call and a few milliseconds, and it removes the large, lazy majority of scraper traffic before any of your more expensive analysis runs. You spend your compute budget on the traffic that survives the cheap filter.

Layer 1: score the IP before you do anything else

What IP intelligence catches, and what it misses

IP scoring is good at three things. It spots datacenter and cloud origins, because a checkout request from an AWS or Hetzner range is almost never a real shopper. It spots known commercial VPNs and proxies, because those exit nodes get enumerated and catalogued. And it spots residential proxies, the hardest category, when the provider has been identified through behavioral profiling and honeypot data.

What it misses: a brand-new residential proxy exit that nobody has flagged yet, and a low-and-slow scraper running from a clean home connection at human speed. Those exist, they are a minority of scraper traffic by volume, and they are the reason layers two and three are not optional. Being clear about this up front is the difference between a defense you can reason about and a false sense of security.

The fields that matter

A useful IP intelligence response is not a single yes/no. It is a set of specific flags plus a risk score, and the specificity is the point. Here is the security block for 2.56.188.34, an IP that happens to be flagged across several categories at once, returned from ipgeolocation.io's include=security lookup:

{
  "ip": "2.56.188.34",
  "security": {
    "threat_score": 80,
    "is_anonymous": true,
    "is_known_attacker": true,
    "is_bot": false,
    "is_spam": false,
    "is_tor": false,
    "is_proxy": true,
    "proxy_provider_names": ["Zyte Proxy"],
    "proxy_confidence_score": 80,
    "proxy_last_seen": "2025-12-12",
    "is_residential_proxy": true,
    "is_vpn": true,
    "vpn_provider_names": ["Nord VPN"],
    "vpn_confidence_score": 80,
    "vpn_last_seen": "2026-01-19",
    "is_relay": false,
    "relay_provider_name": "",
    "is_cloud_provider": true,
    "cloud_provider_name": "Packethub S.A."
  }
}

Read what that gives you beyond "bad or not." is_residential_proxy is separate from is_vpn, so you can treat a NordVPN user (often a privacy-conscious real person) differently from traffic riding a residential proxy network (much more likely to be scraping or fraud). proxy_provider_names names the operator, so "Zyte Proxy" (a commercial scraping service) can trigger a harder rule than an unknown one. is_relay is broken out on its own, which matters because iCloud Private Relay is not a threat and blocking it means blocking a chunk of ordinary Apple users. And the confidence scores let you set your own bar: act on a 90 differently than a 40.

The provider attribution and confidence scores are the fields most detection APIs skip. A bare is_vpn: true forces you to guess. is_vpn: true, vpn_provider_names: ["Nord VPN"], vpn_confidence_score: 80 lets you write a rule you can defend.

Threat-score decisioning

The threat_score is a 0-100 aggregate of the individual signals. Do not wire it as a binary. The whole reason to have a graded score is to take graded action, which matters enormously when most flagged IPs on a busy site carry low-to-moderate risk and a hard block on all of them will catch real customers.

A sane starting policy, which you then tune against your own traffic:

Threat score	What it usually means	Action
0-19	No meaningful risk signal	Allow, no friction
20-44	Something worth watching	Allow, log, maybe soft rate-limit
45-79	Elevated: proxy or VPN with decent confidence	Challenge (CAPTCHA or step-up)
80-100	Datacenter proxy, known attacker, stacked signals	Block or send to manual review

The travel angle makes the tiering concrete. A logged-in customer on a corporate VPN scoring in the 40s should hit a challenge on checkout, not a wall, because blocking them loses a booking. A request from a commercial scraping proxy scoring 80 on your fare-search endpoint gets nothing. Same score field, different action, because the response is tuned to what a false block actually costs on that route.

Build it: IP-scoring middleware

Enough theory. The setup is one API call per uncached IP. Grab a key from ipgeolocation.io's IP security API; the free tier covers geolocation, but the security flags and threat score are on the paid plans, and the security module adds credits per lookup (a base lookup is 1 credit, include=security brings the request to 3), so caching is not optional at any real volume. I am using it here because it returns the datacenter, VPN, proxy, residential-proxy, and threat-score fields in one response, which keeps the example short. ipinfo, IP2Location, and IPQualityScore return overlapping data; the middleware pattern below is the same whichever you pick, so swap the client if you already have one in your stack.

See the response first

Before writing any code, confirm what you get back. One curl call, keeping the response trimmed to the security object:

curl -X GET 'https://api.ipgeolocation.io/v3/ipgeo?apiKey=API_KEY&ip=2.56.188.34&include=security&fields=security'

include=security turns on the security module, fields=security trims everything else so you are not paying attention to location data you do not need here. Check the X-Credits-Charged response header on the first call so the per-request cost is never a surprise on your bill.

The Express middleware

Here is the core: middleware that scores the client IP and attaches a verdict to the request. Every downstream handler can then read req.ipRisk and decide what to do.

// ip-risk.js
// Scores the client IP via ipgeolocation.io and attaches a verdict to req.
// Requires Node 18+ (global fetch). Set IPGEO_API_KEY in your environment.

const API_KEY = process.env.IPGEO_API_KEY;
if (!API_KEY) {
  // Fail loudly at startup, not silently on the first request.
  throw new Error("IPGEO_API_KEY is not set");
}

const ENDPOINT = "https://api.ipgeolocation.io/v3/ipgeo";

// Map a security object to an action. Tune these bands against your own traffic.
function decideAction(security) {
  if (!security) return "allow"; // No data: fail open, decide upstream.

  const score = typeof security.threat_score === "number" ? security.threat_score : 0;

  // Hard signals that justify a block regardless of the aggregate score.
  if (security.is_cloud_provider || security.is_known_attacker) return "block";

  if (score >= 80) return "block";
  if (score >= 45) return "challenge";
  if (score >= 20) return "monitor";
  return "allow";
}

async function scoreIp(ip) {
  const url = `${ENDPOINT}?apiKey=${API_KEY}&ip=${encodeURIComponent(ip)}&include=security&fields=security,asn`;

  // Time out fast. A slow lookup should never hold up a checkout page.
  const res = await fetch(url, { signal: AbortSignal.timeout(1500) });
  if (!res.ok) {
    throw new Error(`ipgeo lookup failed: ${res.status}`);
  }
  const body = await res.json();
  return {
      security: body?.security ?? null,
      asn: body?.asn ?? null,
  }; // Optional chaining: the object may be absent.
}

async function ipRisk(req, res, next) {
  const ip = clientIp(req);
  try {
    const scored = await scoreIp(ip);
req.ipRisk = {
  ip,
  security: scored.security,
  asn: scored.asn,
  action: decideAction(scored.security),
};
  } catch (err) {
    // Fail OPEN: if the risk service is down, do not take the whole site down
    // with it. Log, allow, and let a later layer catch anything egregious.
    console.error(`IP risk scoring failed for ${ip}:`, err.message);
    req.ipRisk = { ip, security: null, action: "allow" };
  }
  next();
}

module.exports = { ipRisk, decideAction };

Two decisions in there are worth calling out because getting them wrong is common. The lookup times out at 1500ms so a slow third-party call cannot stall a page a customer is waiting on. And the catch block fails open: if the scoring service is unreachable, the request is allowed rather than blocked. On a checkout path, failing closed would mean an outage at your provider becomes an outage at your store, which is almost never the trade you want. If you are protecting something where the opposite is true (an internal admin route, say), flip that default deliberately and comment why.

Getting the client IP right

clientIp() above is not a throwaway. Behind a load balancer or CDN, req.socket.remoteAddress is your proxy, not the visitor, so you have to read the forwarded header, and you have to trust it only from your own infrastructure.

// The header is trivially spoofable by the client. Only trust it when the
// request actually came through a proxy you control. In Express, configure
// app.set('trust proxy', ...) to match your infra, then use req.ip.
function clientIp(req) {
  // req.ip respects your 'trust proxy' setting. Prefer it over parsing headers by hand.
  if (req.ip) return req.ip;

  const fwd = req.headers["x-forwarded-for"];
  if (typeof fwd === "string" && fwd.length > 0) {
    // Left-most entry is the original client when the chain is trusted.
    return fwd.split(",")[0].trim();
  }
  return req.socket?.remoteAddress ?? "0.0.0.0";
}

If you skip the trust proxy configuration, one of two things breaks: you either score your own load balancer's IP on every request (and see uniform, useless verdicts), or you trust a header an attacker can set to 8.8.8.8 to look clean. Configure it once to match your actual proxy chain and read req.ip.

Cache the verdict

An IP's risk profile does not change between two requests a second apart, so scoring it twice is wasted money and latency. Cache the verdict with a sane TTL. Redis if you have it, an in-process LRU if you do not.

// cache.js — thin Redis-backed cache for IP verdicts.
const { createClient } = require("redis");

const client = createClient({ url: process.env.REDIS_URL });
client.on("error", (err) => console.error("Redis error:", err.message));
client.connect().catch((err) => console.error("Redis connect failed:", err.message));

const TTL_SECONDS = 60 * 60 * 6; // 6 hours. Long enough to save calls, short enough to stay fresh.

async function getCached(ip) {
  try {
    const raw = await client.get(`iprisk:${ip}`);
    return raw ? JSON.parse(raw) : null;
  } catch (err) {
    // Cache failure must not break scoring. Return a miss and move on.
    console.error(`cache read failed for ${ip}:`, err.message);
    return null;
  }
}

async function setCached(ip, verdict) {
  try {
    await client.set(`iprisk:${ip}`, JSON.stringify(verdict), { EX: TTL_SECONDS });
  } catch (err) {
    console.error(`cache write failed for ${ip}:`, err.message);
  }
}

module.exports = { getCached, setCached, client };

Drop the cache into the middleware by checking it before the API call and writing after. A six-hour TTL is a reasonable default for scraper defense; if you are scoring for something more time-sensitive, shorten it. The important part is that a cache outage degrades to a cache miss and a fresh lookup, never to a crash.

Layer 2: rate limiting that assumes IP rotation

IP scoring thins the herd. It does not stop the scraper that rotates through a hundred clean-looking residential IPs at a slow, human-plausible rate. That is what rate limiting is for, and the mistake most teams make is keying it on the bare IP, which is exactly the thing the scraper is rotating.

Key on something with more signal. The IP's ASN (its owning network) is a good coarse key, because a scraper rotating a hundred IPs inside one hosting provider's ASN still shares that ASN. A session token or fingerprint is better where you have one. In practice you combine them, and you feed the IP verdict from layer one into the decision so a request that already scored "monitor" gets a tighter limit than a clean one.

// A minimal sliding-window limiter keyed on ASN + path, tightened by IP risk.
// Uses the same Redis client; swap for a library like rate-limiter-flexible in production.
const { client } = require("./cache");

async function bumpWindow(key, windowSeconds) {
  const count = await client.incr(key);
  if (count === 1) {
    await client.expire(key, windowSeconds);
  }
  return count;
}

async function rateLimit(req, res, next) {
  const asn = req.ipRisk?.asn?.as_number ?? "unknown"; // coarse key that survives IP rotation
  const key = `rl:${asn}:${req.path}`;

  // Tighten the ceiling for already-suspicious traffic.
  const suspicious = ["monitor", "challenge"].includes(req.ipRisk?.action);
  const limit = suspicious ? 20 : 120; // requests per window
  const windowSeconds = 60;

  try {
    const count = await bumpWindow(key, windowSeconds); // INCR + EXPIRE, see note
    if (count > limit) {
      res.setHeader("Retry-After", windowSeconds);
      return res.status(429).json({ error: "rate limit exceeded" });
    }
  } catch (err) {
    // Limiter backed by Redis; if it is down, do not block legitimate traffic.
    console.error("rate limit check failed:", err.message);
  }
  next();
}
module.exports = { rateLimit };

Pitfall: ASN keying is coarse on purpose, and it can catch real users who share a large mobile carrier's ASN. Keep the clean-traffic ceiling generous, and reserve the tight limit for traffic the IP layer already flagged. The combination is what makes it safe.

Note that the raw security response nests the ASN under the asn object on a full lookup rather than inside security; if you trimmed the response with fields=security as shown earlier, widen it to include asn when you need the network for keying.

Layer 3: behavioral and honeypot signals

This is the layer that catches what IP scoring cannot: the patient scraper on a clean residential connection. The signals here are about behavior, not origin.

The cheap behavioral tells are request cadence (a client requesting a hundred product pages in perfect thirty-second intervals is not a person), missing or inconsistent headers (no Referer, an Accept header no real browser sends), and headless-browser markers. None of these is conclusive alone, which is the recurring theme of the whole article, but stacked with a middling IP score they add up to a confident verdict.

A honeypot is the highest-signal, lowest-effort trap in the set. Put a link on the page that is invisible to humans and irresistible to a naive crawler, and anything that follows it has outed itself.

// Render a decoy link that humans never see but a link-following bot will hit.
// Do NOT hide it with display:none — headless Chrome refuses clicks on those.
// Use opacity and off-screen positioning so it is still in the DOM and clickable.
function honeypotMarkup() {
  return `<a href="/api/catalog-export"
             aria-hidden="true" tabindex="-1"
             style="position:absolute;left:-9999px;opacity:0">
            View full catalog
          </a>`;
}

// Any hit on the decoy route is automated by definition. Record and act.
function honeypotTrap(req, res) {
  const ip = req.ipRisk?.ip ?? req.ip;
  console.warn(`Honeypot triggered by ${ip} (${req.headers["user-agent"]})`);
  // Add the IP to your own blocklist / feed it back into scoring here.
  res.status(404).end(); // 404, not 403: do not tell the bot it found a trap.
}

Two details earn their comments. Hiding the link with display:none backfires because headless Chrome will not click those, so the trap uses opacity:0 and off-screen positioning to stay clickable. And the trap returns a 404, not a 403, because a 403 tells the operator they hit a honeypot and should adjust; a 404 looks like a dead link and teaches them nothing.

Putting the layers together

Wired in order, the layers form a single decision. IP scoring runs first and cheaply, rate limiting keys on network and tightens for flagged traffic, behavioral checks and the honeypot catch what got through.

const express = require("express");
const { ipRisk } = require("./ip-risk");
const { rateLimit } = require("./rate-limit");

const app = express();
app.set("trust proxy", 1); // match this to your real proxy depth

app.use(ipRisk);      // layer 1: attach req.ipRisk
app.use(rateLimit);   // layer 2: ASN-keyed, risk-aware

// layer 1 enforcement on the endpoints scrapers actually want
app.get("/api/fares", (req, res, next) => {
  const action = req.ipRisk?.action ?? "allow";
  if (action === "block") return res.status(404).end(); // 404 leaks less than 403
  if (action === "challenge") return res.status(403).json({ challenge: "captcha" });
  next(); // allow + monitor fall through; "monitor" is already logged
}, serveFares);

function serveFares(req, res) {
  res.json({ route: "JFK-LHR", fares: [/* ... */] });
}

What you log here decides whether you can tune later. Record the IP, the score, the individual flags, the action taken, and the endpoint, for every non-allow decision. After a week you will see which score bands catch real scrapers on which routes, and you will move your thresholds off the starting defaults to fit your own traffic. Detection without logging is a guess you can never check.

Where this stops working

Be clear-eyed about the limits, because pretending they do not exist is how you get a false sense of safety.

Residential proxies that route through genuine ISP customers are the hard ceiling. When a scraper's exit IP belongs to a real Comcast subscriber and has not been flagged yet, the IP layer sees an ordinary residential address. That traffic is caught, if at all, by behavior: cadence, session shape, the honeypot. The IP score narrows the field; it does not close it.

False positives are the other side of the coin, and on a revenue path they cost you directly. Corporate VPNs, university networks, and iCloud Private Relay all carry real customers who will trip a naive rule. This is why the flags are broken out and why you tier by score instead of blocking on any anonymizer signal. Treat is_relay gently. Do not put a hard block on a VPN flag alone.

And do not block the good bots. Googlebot and Bingbot need in, or you disappear from search. Verify legitimate crawlers (reverse DNS on the claimed identity, not the user-agent string, which is trivially forged) and allowlist them ahead of any of this.

When the scraping is sophisticated, persistent, and expensive enough to be worth an arms race, that is the point where a dedicated bot-management platform earns its cost. The IP layer you just built is the right first move for most travel and e-commerce sites, and it is often enough on its own for the bulk, lazy scraping that makes up the majority of the problem. It is a floor, not a ceiling.

A few extra notes

robots.txt is not enforcement. It is a request that polite crawlers honor and scrapers ignore. Keep it accurate for Googlebot; do not mistake it for a control.

Logging IPs has compliance weight. Under GDPR an IP address is personal data. If you are storing verdicts and IPs, have a retention policy and a reason, and keep the window short.

IPv6 changes the blocklist math. A single IPv6 subscriber can have a /64 (that is 18 quintillion addresses), so blocking individual v6 addresses is pointless. Score and rate-limit on the prefix, not the full address.

Self-hosted is an option at scale. If per-request lookups become a latency or budget problem, the same detection data ships as a downloadable database you query locally. Worth it once you are past a few million lookups a month or you need air-gapped scoring.

Start with the IP layer on your fare-search and category endpoints, because that is where the scrapers go and where the cheap filter buys you the most. Add rate limiting the same day; it is a few lines and it covers the rotation case. Get behavioral detection in once you have a week of logs telling you what the IP layer missed. If I were doing this from scratch on a busy catalog, I would wire scoring and the honeypot on day one and treat everything past that as tuning.