Detect sitemap, feed, and tech-stack changes and get them as signed webhooks

Crawlwise — Sat, 24 Oct 2026 23:57:59 +0000

A common pattern for tracking a competitor's pages, a vendor's changelog, or the stack behind a domain is a cron job that fetches something, diffs it against last time, and pings you. It works until it doesn't: you store snapshots, handle sitemap index files, normalize three feed formats, and decide what "changed" means.

This post walks through a push-based version of that loop: tell a service what to watch, and receive a signed webhook when something changes. I'll use Crawlwise Data, which I work on (disclosure at the end). The pattern is the same whichever tool you use.

Three kinds of change worth watching

Sitemap changes. Diffing a site's XML sitemap shows new, updated (lastmod), and removed URLs. One catch: if you only fetched part of a large sitemap, a URL missing from your partial result is not a removal. A diff that treats truncation as deletion will send you false "removed" alerts, so be careful here.
Feed items. RSS, Atom, RDF, and JSON Feed all describe "new post" differently. Normalize them, remember which item IDs you've seen, and only report new ones.
Tech-stack changes. Fingerprint a page (headers, HTML, DNS records) and diff the list of detected technologies. Something like "a CDN appeared" or "a framework disappeared" is often a useful signal.

Be a polite crawler

Before any of this, a few rules that apply whether you build it or buy it:

Check robots.txt first (RFC 9309) and respect it. If the site disallows you, don't fetch.
Identify yourself with a descriptive User-Agent that links to a page explaining who you are and how to reach you. Crawlwise sends Mozilla/5.0 (compatible; CrawlwiseBot/1.0; +https://crawlwisedata.com/bot.html) and uses the product token CrawlwiseBot for robots.txt.
Back off on 429 and 503. Honor Retry-After when it's present, and use exponential backoff when it isn't. Don't retry in a tight loop.
Don't poll harder than the change rate. Sitemaps and feeds rarely change every minute. Crawlwise's minimum monitor interval is 15 minutes.
Stick to public data. No logins, no form submissions, no getting around bot protection.

Setting up a monitor

With Crawlwise you create a monitor that points at a site, feed, or sitemap and at your HTTPS webhook URL:

curl -X POST "https://crawlwisedata.com/v1/monitors" \
  -H "Authorization: Bearer $CRAWLWISE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "endpoint": "sitemap",
    "name": "competitor",
    "url": "https://example.com",
    "webhookUrl": "https://your.app/hooks/crawlwise",
    "schedule": true,
    "intervalSec": 900
  }'

endpoint can be feed, sitemap, or techstack. You get your signing secret from GET /v1/webhooks/secret. Scheduled checks usually run within an hour, and you can trigger a check on demand with POST /v1/monitors/run.

What a delivery looks like

When a check finds a change, your endpoint gets a POST with Content-Type: application/json and a body shaped like this (values are illustrative, and the changes entries are trimmed):

{
  "id": "evt_...",
  "type": "monitor.change",
  "created_at": "2026-10-23T15:00:00.000Z",
  "detected_at": "2026-10-23T14:58:12.000Z",
  "account_id": "acct_...",
  "monitor": { "endpoint": "sitemap", "name": "competitor", "url": "https://example.com" },
  "change": {
    "kind": "sitemap.diff",
    "counts": { "new": 1, "updated": 1, "removed": 1, "unchanged": 950 },
    "changes": [ ... ]
  },
  "credits": { "charged": 13 }
}

change.kind tells you which shape to expect:

feed.new_items: newItems and an items array
sitemap.diff: counts and a changes array
techstack.diff: added, removed, and technologyNames

The request also carries these headers: X-Crawlwise-Signature, X-Crawlwise-Delivery, X-Crawlwise-Timestamp, and X-Crawlwise-Event.

Verify the signature

The signature header looks like t=<unix>,v1=<hex>. The v1 value is an HMAC-SHA256, keyed with your webhook secret, over the string `${t}.${rawBody}`. Two details matter: use the raw request body (not re-serialized JSON), and compare in constant time.

Node (Express):

import express from 'express';
import { createHmac, timingSafeEqual } from 'node:crypto';

const TOLERANCE_SEC = 300;

function verify(secret, rawBody, header) {
  const m = /^t=(\d+),v1=([0-9a-f]{64})$/i.exec(header || '');
  if (!m) return false;
  if (Math.abs(Date.now() / 1000 - Number(m[1])) > TOLERANCE_SEC) return false;
  const expected = createHmac('sha256', secret).update(`${m[1]}.${rawBody}`).digest();
  const got = Buffer.from(m[2], 'hex');
  return got.length === expected.length && timingSafeEqual(got, expected);
}

const app = express();
app.post('/hooks/crawlwise', express.raw({ type: 'application/json' }), (req, res) => {
  const raw = req.body.toString('utf8');
  if (!verify(process.env.CRAWLWISE_WEBHOOK_SECRET, raw, req.get('X-Crawlwise-Signature'))) {
    return res.sendStatus(401);
  }
  const event = JSON.parse(raw);
  // event.type === 'monitor.change'; branch on event.change.kind
  res.sendStatus(200);
});

Python equivalent of the check:

import hashlib, hmac, re, time

def verify(secret: str, raw_body: bytes, header: str) -> bool:
    m = re.match(r'^t=(\d+),v1=([0-9a-f]{64})$', header or '', re.I)
    if not m or abs(time.time() - int(m[1])) > 300:
        return False
    signed = m[1].encode() + b'.' + raw_body
    expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, m[2].lower())

The 300-second tolerance is a suggestion from the docs, and it also protects you against replayed requests.

Handle retries and duplicates

Failed deliveries are retried with the same event ID and the same X-Crawlwise-Delivery value, and a fresh signed timestamp, so your tolerance check still passes. Deduplicate on X-Crawlwise-Delivery (or the event id) and make your handler idempotent.

Testing without waiting

You don't need a site that happens to change. POST /v1/monitors/test sends a free signed sample event of type monitor.test to your webhook, with an X-Crawlwise-Test: 1 header. The response echoes the exact body and signature that were sent, so you can check your verifier. There is also a built-in demo target whose feed and sitemap change every 5 minutes; scheduled checks on it use credits, like any other monitor. Don't treat monitor.test events as real changes.

Limits worth knowing

This is an early product, so here's what it doesn't do: it only reads public pages, it doesn't run JavaScript, and it checks robots.txt before fetching and honors its rules. Scheduled checks are usually within an hour rather than instant, and intervalSec is a lower bound, not a guarantee.

Cost

It's credit-based (1 credit = $0.0001). A monitor check that finds no change costs 2 credits (5 for tech-stack); when a change is found, the normal endpoint price applies (a sitemap change is 10 credits + 1 per changed URL). Failed calls, including robots-disallowed ones, cost nothing. Test events are free.

To try it: a free trial key with 1,000 credits, no card needed (accept the Terms to get it). Monthly plans are Starter $19, Pro $49, and Business $149. Launch offer (FIRST5): the first 5 subscribers get 14 days free on any monthly plan. A card is required, and it renews at the plan price unless you cancel before the 14 days end.

If you work from an AI agent, there's also a local MCP server (npx crawlwise-mcp) that uses the same API key and credits.

Disclosure: We built Crawlwise Data. It's new, and feedback on the API shape and docs is welcome at crawlwise.support@gmail.com. Docs and trial key: crawlwisedata.com

DEV Community: Crawlwise