How to Build a Polite Restock Checker: Polling, Caching and Backoff

WeBot — Sun, 11 Oct 2026 00:54:19 +0000

Restock alerts are useful. Flooding a retailer's origin with aggressive scrapers is not. If you are watching Pokémon TCG product pages—or any collectible SKU that vanishes in minutes—you can still get timely signals while staying a good citizen of the web.

This article walks through the engineering patterns that keep a desktop restock checker polite: conditional HTTP, sensible intervals with jitter, backoff on rate limits, structured stock detection, and de-duplicated notifications.

Start with permission, not code

Before you write a loop, read the retailer's robots.txt and terms of use. If crawling product pages is disallowed, or the site offers an official stock API or feed, prefer that path. A checker that ignores published rules will eventually get blocked—and it may also be a terms violation.

Treat every poll as a cost the retailer pays. Design so that cost stays low.

Prefer conditional requests over full downloads

Most product pages support caching headers. Send ETag / If-None-Match and Last-Modified / If-Modified-Since so unchanged pages return 304 Not Modified with an empty body. You learn "nothing changed" without transferring HTML.

import httpx

headers = {}
if etag:
    headers["If-None-Match"] = etag
if last_modified:
    headers["If-Modified-Since"] = last_modified

r = httpx.get(url, headers=headers, timeout=20)
if r.status_code == 304:
    return None  # unchanged
# Persist r.headers.get("ETag") and r.headers.get("Last-Modified") for next poll
return r.text

Cache those validators per URL. On a quiet night, most of your traffic should be 304s.

Poll on a human cadence—with jitter

Fixed 500 ms loops look like bots because they are bots. For restock monitoring of a handful of SKUs, intervals measured in tens of seconds to a few minutes are usually enough. Spread load across time with jitter so many clients do not align on the same second.

function nextDelay(baseMs, jitterRatio = 0.3) {
  const jitter = baseMs * jitterRatio * (Math.random() * 2 - 1);
  return Math.max(5_000, Math.round(baseMs + jitter));
}

// e.g. base 45s → roughly 31.5s–58.5s between polls
setTimeout(poll, nextDelay(45_000));

Tune baseMs to how fast stock actually turns over for that retailer—and to what their infrastructure can reasonably absorb. Faster is not always better if it gets you throttled.

Back off hard on 429 and 503

When a server says 429 Too Many Requests or 503 Service Unavailable, stop. Honour Retry-After if present; otherwise use exponential backoff with a ceiling.

A simple policy:

On 2xx/304: reset the failure streak; schedule the next poll with normal jitter.
On 429/503: multiply the delay (e.g. 2× each time), cap at something like 15–30 minutes, and never spin.
On repeated failures: pause that URL and surface a clear status in the UI instead of hammering.

This protects both you and the store during sale events when everyone is watching the same page.

Detect stock from structured data, not layout

Scraping CSS class names breaks whenever marketing redesigns a button. Prefer machine-readable product data—especially JSON-LD Product / Offer markup with availability values such as InStock or OutOfStock.

Parse the JSON-LD blocks, normalise the offer fields you care about (SKU, price, availability, URL), and diff against the previous snapshot. An alert fires only when availability flips in a way you care about (for example, OutOfStock → InStock), not when a banner image changes.

If JSON-LD is missing, fall back to other stable signals the site already exposes for SEO or accessibility—still prefer structured fields over brittle DOM paths. Persist the last known offer state so restarts do not re-alert for the same stock level.

De-duplicate alerts and notify once

Desktop notifications are easy to spam. Keep a small alert ledger keyed by something like (retailer, sku, availability_state) with a short TTL or a "already notified for this in-stock window" flag. When stock goes out again and later returns, you can allow a new alert; while it stays in stock, stay quiet.

For the notification itself, use the platform APIs (Windows toast / macOS User Notifications) with a concise title, product name, and a deep link the user can open. One clear ping beats five redundant ones during a restock scramble.

Put it together

A respectful checker looks roughly like this:

Confirm the URL is allowed under robots/terms.
Conditional GET with stored validators.
On 304: sleep with jitter; on 200: parse JSON-LD offers and diff.
On stock transition of interest: de-dupe, then desktop notify.
On 429/503: exponential backoff; never tight-loop.

None of this requires defeating bot protection or jumping queues. The goal is a lightweight monitor that behaves like a careful browser refresh—just automated, cached, and rate-aware.

We use these patterns in WeBot for AU/NZ collectibles restock alerts. If you build your own checker, keep the retailers' infrastructure in mind: polite defaults scale better than aggressive ones—for you and for everyone else watching the same aisle.

— The WeBot team

DEV Community: WeBot