Enriching 10,000 NPIs with physician emails: batching, webhooks and HMAC verification on the Dmand AI API

#python #api #webhooks #healthtech

Last week I showed how to turn one NPI into a verified physician email with a single call (part 1). One NPI is a demo. Real jobs look like "here are 10,000 NPIs from last quarter's referral data, get me emails by Monday."

This post covers the three things that matter at that scale on the Dmand AI API: batching, webhooks, and verifying the webhook signature so nobody can feed your pipeline fake results.

The limits you are working with

From the rate limits page:

45 NPIs per bulk request
45 NPIs per minute

So 10,000 NPIs is about 223 batches, and at one batch a minute that is roughly 3.7 hours of wall time. That is fine for a background job. It is not fine for a request handler, which is why the rest of this post is about queues and webhooks.

Credits work the way you would want: they are held when you submit and only charged when an email is found. NPIs with no result are refunded. Check GET /api/v1/credits before a big run; the response has balance, total and used.

Step 1: chunk and submit

The bulk endpoint is POST https://ext-api.dmand.ai/api/v1/email/bulk. Send up to 45 NPIs, a webhook object, and optional custom_data that gets echoed back on every result.

import os, time, uuid, requests

API_KEY = os.environ["DMAND_API_KEY"]
BASE = "https://ext-api.dmand.ai/api/v1"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

def chunks(items, size=45):
    for i in range(0, len(items), size):
        yield items[i:i + size]

def submit_batch(npis, job_id):
    body = {
        "npis": npis,
        "webhook": {
            "url": "https://yourapp.example.com/hooks/dmand",
            "mode": "batch",           # one callback when the whole batch is done
        },
        "custom_data": {"job_id": job_id},
    }
    r = requests.post(
        f"{BASE}/email/bulk",
        json=body,
        headers={**HEADERS, "Idempotency-Key": str(uuid.uuid4())},
        timeout=30,
    )
    if r.status_code == 429:
        return None                    # rate limited: re-queue and try later
    if r.status_code in (403, 422):
        r.raise_for_status()           # auth or validation: fix the request, do not retry
    r.raise_for_status()               # 5xx: let your worker retry with backoff
    return r.json()                    # 202 with batch_id and one request_id per NPI

Two details worth copying:

mode: "batch" means one webhook per 45 NPIs instead of 45 separate callbacks. Use real-time only when a person is watching a screen and wants rows to fill in one by one.
The Idempotency-Key header makes retries safe. If your worker dies after the request lands but before it saves the batch_id, resending with the same key does not double-submit.

Check the exact request and response shapes in the bulk submit reference and the data dictionary; I am only showing the fields I use.

Step 2: pace it with a queue

Do not loop over chunks() in a web request. Put each batch on your own queue (Celery, Sidekiq, SQS, whatever you have) and let a single worker drain it. The worker pattern from the volume guide:

On 202, store batch_id and the request_id for each NPI, mark the batch as submitted.
On 429, sleep briefly and put the batch back on the queue.
On 5xx, retry with exponential backoff (1, 2, 4, 8 seconds) up to a cap.
On 403 or 422, stop and alert. Retrying a bad request just burns time.

Sleeping about 60 seconds between full batches keeps you under the per-minute limit without any clever math.

Step 3: receive results and verify the signature

Every webhook carries an X-LNZ-Signature header: a hex HMAC-SHA256 of the raw request body, keyed with the webhook secret you got when the API key was issued. Hash the raw bytes before parsing JSON. Re-serialising the body changes whitespace and the digest will never match.

import hmac, hashlib
from flask import Flask, request, abort

app = Flask(__name__)
SECRET = os.environ["DMAND_WEBHOOK_SECRET"].encode()

@app.post("/hooks/dmand")
def dmand_hook():
    sig = request.headers.get("X-LNZ-Signature", "")
    expected = hmac.new(SECRET, request.get_data(), hashlib.sha256).hexdigest()
    if not hmac.compare_digest(sig, expected):
        abort(401)

    payload = request.get_json()
    # payload["event"] is "enrich.completed"
    # payload["custom_data"] carries your job_id
    # payload["data"] is a list of per-NPI results
    for item in payload["data"]:
        upsert_result(item)            # dedupe on item["request_id"]
    return "", 200

def upsert_result(item):
    # INSERT ... ON CONFLICT (request_id) DO UPDATE ...
    ...

Three rules keep this handler boring, which is what you want:

Return 2xx fast and do real work in a background job. Non-2xx responses get retried.
Deduplicate on request_id. Retries and the lack of a timestamp in the signature mean the same payload can arrive twice.
Use hmac.compare_digest, never ==, for the comparison.

What comes back

Each result has a status: completed, not_found or failed, plus the email fields when found. Emails validated in the last 30 days come back from storage within seconds. Anything older triggers a live lookup that can take 5 to 10 minutes, so show a pending state in your UI rather than a spinner. The status reference explains each state and what "fresh" means.

Why NPI as the key

Most B2B enrichment keys on name plus company domain, and in healthcare that breaks constantly: 200 Dr. Patels, hospitalists who rotate across three sites, physicians whose "company" is a 40,000-person health system. The NPI is the one identifier every US clinician has exactly once, and it is the same key you already have in referral, billing and credentialing data. Keying on it means the enrichment job you build today still joins cleanly next year.

If you want the rest of the picture around the NPI (where the physician actually practices, which parent organization that site rolls up to, who the executives are), that is what the full Dmand AI platform does on top of the same data. Pricing starts at $249 per user per month with a 30-day trial, and the API runs on the same credits.

Questions on the API: hello@dmand.ai or the comments.

DEV Community