DEV Community

coreclaw
coreclaw

Posted on

Webhook Data Delivery: How to Build a Reliable Webhook Receiver for Web Data with Python

#webscraping #api #python #data

Most web data platforms that run scraping jobs — including CoreClaw Workers — can push results to your HTTP endpoint instead of forcing you to poll for completion. The reliable way to receive that push is to build a small webhook receiver that verifies incoming signatures, deduplicates by idempotency key, acknowledges within the timeout window, and retries failures with exponential backoff. This article shows the full pattern in runnable Python.

TL;DR

  • A webhook receiver is an HTTP endpoint that accepts POST requests containing structured web data results.
  • You need four components: signature verification, idempotency, fast acknowledgement, and a background retry queue.
  • Use environment variables for endpoint secrets and API keys — never hardcode them.
  • Test locally with a Flask or FastAPI server, then deploy behind a reverse proxy that handles TLS.
  • The CoreClaw Workers platform lets you deploy scrapers that can deliver results via webhook, and the CoreClaw store offers ready-made Workers you can configure without writing a scraper from scratch.

Why Polling Falls Short

When you run scraping jobs at scale — whether you are monitoring Amazon prices, tracking Google SERP positions, or collecting Google Maps business data — you eventually hit a choice: poll the API for job status, or receive a push notification when results are ready.

Polling works for small volumes. But it creates three problems as you scale:

  1. Wasted requests. If jobs average 45 seconds and you poll every 5 seconds, you issue 9 requests that return "still processing" for every 1 that returns results. At 500 jobs per day, that is 4,500 wasted API calls.
  2. Latency floor. You only learn about results on your next poll cycle. If you poll every 30 seconds, your downstream pipeline has a built-in 15-second average delay — even if the job finished immediately.
  3. Rate-limit pressure. Polling consumes your API quota for status checks, leaving less headroom for actual data requests.

Webhook delivery flips the model: the scraping platform pushes results to your endpoint the moment they are ready. No wasted requests, no latency floor, no quota pressure from status checks.

What a Webhook Receiver Must Do

A production webhook receiver is not just @app.post("/webhook") and a print(request.json). It must handle four concerns:

Concern Why It Matters Failure Mode If Ignored
Signature verification Confirm the payload came from the platform, not an attacker Spoofed data injected into your pipeline
Idempotency The platform may retry delivery if your server is slow to respond Duplicate records in your database
Fast acknowledgement Most platforms time out within 5–10 seconds Legitimate deliveries marked as failed
Retry / dead-letter queue Transient database or downstream failures should not lose data Data lost on the first error

The Architecture 

Scraping Platform
       │
       ▼
  POST /webhook/data  (TLS terminated by reverse proxy)
       │
       ▼
  ┌─────────────────────┐
  │ Signature verifier   │  ← HMAC-SHA256 using shared secret
  └──────────┬──────────┘
             ▼
  ┌─────────────────────┐
  │ Idempotency check    │  ← Redis or DB: have we seen this event_id?
  └──────────┬──────────┘
             ▼
  ┌─────────────────────┐
  │ 202 Accepted         │  ← Respond immediately, before processing
  └──────────┬──────────┘
             ▼
  ┌─────────────────────┐
  │ Background worker    │  ← Process payload, write to DB/storage
  └──────────┬──────────┘
             ▼
         Success? 
         ├─ Yes → mark event processed
         └─ No  → enqueue retry with backoff
                    └─ after N retries → dead-letter queue
Enter fullscreen mode Exit fullscreen mode

The key insight: acknowledge before processing. Your HTTP handler should verify the signature, check idempotency, enqueue the payload, and return 202 — all in under one second. The actual data processing happens in a background worker.

Runnable Python Implementation

This example uses FastAPI for the webhook endpoint and an in-process background task for processing. In production, replace the in-process queue with Redis, Celery, or a message broker.

"""
Webhook receiver for web data delivery.

Environment variables:
  WEBHOOK_SECRET  - shared secret for HMAC signature verification
  WEBHOOK_PORT    - port to listen on (default 8000)

The scraping platform should be configured to send POST requests
to https://your-domain.com/webhook/data with:
  - Header: X-Signature: sha256=<hex-hmac>
  - Body: JSON payload containing job results
  - Header: X-Event-Id: unique event identifier (for idempotency)
"""
import os
import hmac
import hashlib
import json
import asyncio
import logging
from collections import OrderedDict
from typing import Dict, Any

from fastapi import FastAPI, Request, HTTPException, BackgroundTasks
from fastapi.responses import JSONResponse

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
logger = logging.getLogger(__name__)

WEBHOOK_SECRET = os.environ.get("WEBHOOK_SECRET", "dev-secret-change-me")
WEBHOOK_PORT = int(os.environ.get("WEBHOOK_PORT", "8000"))

app = FastAPI(title="Web Data Webhook Receiver")

# --- LRU cache for idempotency (use Redis in production) ---
class LRUCache:
    """Simple ordered dict-based LRU cache for seen event IDs."""
    def __init__(self, capacity: int = 10000):
        self.capacity = capacity
        self._cache: OrderedDict[str, bool] = OrderedDict()

    def seen(self, key: str) -> bool:
        if key in self._cache:
            self._cache.move_to_end(key)
            return True
        return False

    def mark(self, key: str) -> None:
        self._cache[key] = True
        self._cache.move_to_end(key)
        if len(self._cache) > self.capacity:
            self._cache.popitem(last=False)

seen_events = LRUCache(capacity=10000)

# --- Retry queue (use Celery/RabbitMQ in production) ---
retry_queue: list[dict] = []
MAX_RETRIES = 5
BASE_BACKOFF_SECONDS = 2

def verify_signature(payload: bytes, signature_header: str, secret: str) -> bool:
    """
    Verify HMAC-SHA256 signature.

    The platform sends: X-Signature: sha256=<hex>
    This function recomputes the HMAC and compares in constant time.
    """
    if not signature_header.startswith("sha256="):
        return False
    expected = signature_header.removeprefix("sha256=")
    computed = hmac.new(
        secret.encode("utf-8"),
        payload,
        hashlib.sha256,
    ).hexdigest()
    return hmac.compare_digest(expected, computed)


async def process_payload(event_id: str, payload: dict) -> None:
    """
    Background task: process the webhook payload.

    In production, this writes to your database, triggers downstream
    pipelines, or forwards to a message queue. Here we simulate work.
    """
    try:
        logger.info(f"Processing event {event_id}, keys: {list(payload.keys())}")

        # --- Your business logic here ---
        # Example: store results, trigger enrichment, notify Slack
        records = payload.get("results", [])
        logger.info(f"Received {len(records)} records in event {event_id}")

        # Simulate processing time
        await asyncio.sleep(0.5)
        logger.info(f"Successfully processed event {event_id}")

    except Exception as e:
        logger.error(f"Failed to process event {event_id}: {e}")
        retry_queue.append({
            "event_id": event_id,
            "payload": payload,
            "attempt": 1,
        })


async def process_retries() -> None:
    """Background loop: retry failed payloads with exponential backoff."""
    while True:
        if retry_queue:
            item = retry_queue.pop(0)
            attempt = item["attempt"]
            if attempt > MAX_RETRIES:
                logger.error(
                    f"Event {item['event_id']} exceeded max retries — sending to dead-letter"
                )
                # In production: write to a dead-letter table or queue
                continue

            backoff = BASE_BACKOFF_SECONDS * (2 ** (attempt - 1))
            logger.info(f"Retrying event {item['event_id']} in {backoff}s (attempt {attempt})")
            await asyncio.sleep(backoff)

            try:
                await process_payload(item["event_id"], item["payload"])
            except Exception as e:
                logger.error(f"Retry {attempt} failed for {item['event_id']}: {e}")
                item["attempt"] += 1
                retry_queue.append(item)
        else:
            await asyncio.sleep(1)


@app.on_event("startup")
async def startup_event():
    asyncio.create_task(process_retries())


@app.post("/webhook/data")
async def receive_webhook(
    request: Request,
    background_tasks: BackgroundTasks,
):
    """
    Main webhook endpoint.

    Verifies signature, checks idempotency, enqueues background
    processing, and returns 202 within ~100ms.
    """
    raw_body = await request.body()

    # 1. Verify signature
    signature = request.headers.get("x-signature", "")
    if not verify_signature(raw_body, signature, WEBHOOK_SECRET):
        logger.warning("Invalid signature — rejecting")
        raise HTTPException(status_code=401, detail="Invalid signature")

    # 2. Parse JSON
    try:
        payload = json.loads(raw_body)
    except json.JSONDecodeError:
        raise HTTPException(status_code=400, detail="Invalid JSON")

    # 3. Idempotency check
    event_id = request.headers.get("x-event-id", "")
    if not event_id:
        # Fall back to a payload hash if no event ID is provided
        event_id = hashlib.sha256(raw_body).hexdigest()[:16]

    if seen_events.seen(event_id):
        logger.info(f"Duplicate event {event_id} — acknowledging without processing")
        return JSONResponse(status_code=202, content={"status": "duplicate", "event_id": event_id})

    seen_events.mark(event_id)

    # 4. Enqueue background processing
    background_tasks.add_task(process_payload, event_id, payload)

    # 5. Acknowledge immediately
    return JSONResponse(
        status_code=202,
        content={"status": "accepted", "event_id": event_id},
    )


@app.get("/health")
async def health():
    return {"status": "ok", "queue_size": len(retry_queue)}


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=WEBHOOK_PORT)
Enter fullscreen mode Exit fullscreen mode

Expected Payload Shape

The exact payload depends on the platform you use. A typical structured web data webhook looks like this:

{
  "job_id": "job_a1b2c3d4",
  "source": "google_maps_scraper",
  "status": "completed",
  "completed_at": "2026-09-15T01:05:00Z",
  "results": [
    {
      "name": "Example Business",
      "address": "123 Main St, Anytown, USA",
      "phone": "+1-555-0100",
      "rating": 4.5,
      "reviews_count": 127,
      "category": "Restaurant"
    }
  ],
  "result_count": 1,
  "metadata": {
    "query": "restaurants in Anytown",
    "pages_processed": 1
  }
}
Enter fullscreen mode Exit fullscreen mode

Your receiver should not assume a fixed schema across all data sources. A Google Maps job returns business records; an Amazon job returns product records; a SERP job returns ranking entries. Use the source field to route payloads to the correct handler.

Testing Your Receiver Locally

You can test without a live platform by sending mock requests:

"""
Send a test webhook to your local receiver.
"""
import os
import hmac
import hashlib
import json
import requests

WEBHOOK_SECRET = os.environ.get("WEBHOOK_SECRET", "dev-secret-change-me")
WEBHOOK_URL = os.environ.get("WEBHOOK_URL", "http://localhost:8000/webhook/data")

payload = {
    "job_id": "test_job_001",
    "source": "google_maps_scraper",
    "status": "completed",
    "results": [
        {"name": "Test Business", "rating": 4.0, "reviews_count": 50}
    ],
    "result_count": 1,
}

body = json.dumps(payload).encode("utf-8")
signature = hmac.new(
    WEBHOOK_SECRET.encode("utf-8"),
    body,
    hashlib.sha256,
).hexdigest()

response = requests.post(
    WEBHOOK_URL,
    data=body,
    headers={
        "Content-Type": "application/json",
        "X-Signature": f"sha256={signature}",
        "X-Event-Id": "evt_test_001",
    },
    timeout=10,
)

print(f"Status: {response.status_code}")
print(f"Response: {response.json()}")
Enter fullscreen mode Exit fullscreen mode

Business Use Cases

Use Case What the Webhook Delivers Downstream Action
Price monitoring Amazon product price + stock for tracked ASINs Trigger repricing engine, alert if drop > 5%
SERP rank tracking Position + URL for tracked keywords Update dashboard, notify SEO team on position changes
Lead enrichment Google Maps business records matching criteria Push to CRM, trigger sales outreach sequence
Creator analytics YouTube/TikTok engagement metrics Update influencer database, flag viral content
Competitor monitoring Social media post data for competitor accounts Feed to sentiment analysis pipeline

Limitations and Compliance

  • Endpoint availability is your responsibility. If your server is down, the platform may retry a limited number of times before dropping the delivery. Use a managed queue or serverless function for high availability.
  • Signature schemes vary. The HMAC-SHA256 pattern shown here is common, but some platforms use a different header name or algorithm. Always check the platform's documentation for the exact verification method.
  • Idempotency requires durable storage. The LRU cache in the example loses state on restart. In production, use Redis or a database table to track processed event IDs.
  • Rate limits apply. A platform that can deliver 1,000 webhooks per second will overwhelm an under-provisioned receiver. Use a queue between the HTTP handler and your database to absorb bursts.
  • TLS is mandatory. Webhooks carry structured data over the public internet. Your endpoint must use HTTPS. Most platforms refuse to deliver to plain HTTP.
  • Respect data source terms. Web data delivered via webhook is still subject to the target website's terms of service, robots directives, and applicable privacy law. Receiving data via push does not change your compliance obligations. For a broader discussion of public web data compliance, consult the CoreClaw pricing page to understand pay-per-result models, and review the terms of any data source you monitor.

Self-Hosted vs Managed Webhook Delivery

Dimension Self-Hosted Polling Managed Webhook Delivery
Latency Bounded by poll interval Near-immediate on job completion
API quota Consumed by status checks Used only for data requests
Infrastructure Client-side scheduler + retry logic Receiver endpoint + queue
Failure modes Missed polls, scheduler crashes Endpoint down, signature mismatch
Best for Low-volume, infrequent jobs High-volume, time-sensitive pipelines
Scaling Linear with poll frequency Horizontal: add receiver replicas

Checklist Before Production

  • [ ] Endpoint is behind HTTPS (TLS terminated by reverse proxy or load balancer)
  • [ ] Signature verification uses hmac.compare_digest (constant-time comparison)
  • [ ] Idempotency store is durable (Redis or database, not in-memory)
  • [ ] HTTP handler returns 202 within 2 seconds of receiving the request
  • [ ] Background processing has a retry mechanism with exponential backoff
  • [ ] Dead-letter queue or table captures payloads that exceed max retries
  • [ ] Monitoring alerts on queue depth and retry failure rate
  • [ ] Health check endpoint exists and is monitored
  • [ ] Secrets are loaded from environment variables, not hardcoded

FAQ

1. Do I need a webhook receiver if I only run a few jobs per day?

No. If you run fewer than 20–30 jobs per day and latency is not critical, polling is simpler and perfectly adequate. Webhook receivers become worthwhile when volume, speed, or API quota pressure makes polling expensive.

2. What happens if my server is down when the platform sends a webhook?

Most platforms retry delivery after a delay (commonly 1, 5, and 30 minutes). If your server remains down after the final retry, the delivery is dropped. A durable queue (Redis, SQS, or similar) ensures you do not lose data during outages — the platform retries until your receiver is back.

3. How do I verify that the webhook came from the platform and not an attacker?

Use HMAC signature verification. The platform computes HMAC-SHA256(payload, shared_secret) and sends the result in a header. Your receiver recomputes the HMAC and compares it in constant time using hmac.compare_digest. An attacker without the shared secret cannot produce a valid signature.

4. Can I use the same receiver for multiple data sources?

Yes. Route by the source field in the payload or by a custom header. Each source can have its own handler function. Just make sure your idempotency key includes the source identifier if event IDs are only unique within a single source.

5. How do I deploy a scraper that delivers results to my webhook?

On the CoreClaw Workers platform, you can deploy a custom Worker or choose a ready-made one from the store. Configure the webhook URL and shared secret in the Worker settings. The platform handles job execution and pushes results to your endpoint when the job completes.

6. Should I store raw webhook payloads or only processed results?

Store both, at least temporarily. Raw payloads serve as an audit trail and let you reprocess if your handler logic changes. A common pattern: write the raw payload to an append-only log (S3, GCS, or a database JSON column), then process it into structured tables.

7. What is the difference between a webhook and an API callback?

They are often used interchangeably. A webhook is an HTTP callback: the platform makes an HTTP POST to your URL when an event occurs. Some platforms also support long-polling or Server-Sent Events (SSE), but webhooks are the most common push-delivery mechanism because they work across firewalls and do not require a persistent connection.

Summary

Webhook delivery eliminates the latency, waste, and quota pressure of polling — but only if your receiver is built to handle the realities of network-delivered data. The four pillars are signature verification, idempotency, fast acknowledgement, and a durable retry queue. Start with the FastAPI implementation above, test locally with mock payloads, and move the queue and idempotency store to Redis before production traffic.

If you want to deploy a scraper that pushes results to your endpoint, explore the CoreClaw console for custom deployments or browse the CoreClaw store for ready-made Workers. You can check pay-per-result pricing to estimate delivery costs before you scale.

Related Reading

Top comments (0)