Building a public coupon-quality crawler that fails closed

Coupon data has an awkward property: most defects look plausible. An expired offer can still have polished copy. A publisher instruction can look like shopper-facing terms. The same offer can appear on two localized pages with different text. A code can be real but inapplicable to a particular cart.

At CouponX we needed a repeatable way to audit what the public site actually exposed, independent of what the database or an affiliate feed claimed. This post explains the crawler design we used for a live run over 1,078 localized store pages and 2,058 unique offers.

Disclosure: I am writing on behalf of CouponX. The implementation and failure modes are the subject of this article; the post is not a claim that every CouponX offer works for every shopper.

Design goal: audit the public contract

The crawler starts from the sitemap and reads the same public page payload available to a visitor. That boundary is important. A database record can be correct while a localization bug, serializer, cache, or template exposes something different in production.

The audit therefore answers:

Does the public page contain the fields and evidence required for the claim it makes?

It does not answer whether a private, targeted, or app-only offer will apply to an arbitrary customer. When evidence is missing, the correct output is unverified, not a guessed success state.

1. Discover only the intended page type

The sitemap contains many URL classes. We select localized store pages explicitly instead of crawling every internal link.

function storeUrls(sitemap) {
  return [...sitemap.matchAll(/<loc>(.*?)<\/loc>/g)]
    .map((match) => decodeEntities(match[1]))
    .filter((url) => /\/(?:en|ru)\/store\//.test(url));
}

This is a small but useful safety property: a navigation change cannot silently expand the crawl into account pages, search grids, or tracking URLs.

2. Require a completeness marker

Network success is not payload success. A proxy or origin can close a response after returning enough HTML to look superficially valid.

Our fetch wrapper requires a marker expected at the end of the document and retries only when that marker is absent:

async function requestUntil(url, marker, attempt = 1) {
  const body = await fetchBody(url);

  if (!body.includes(marker)) {
    if (attempt < 3) return requestUntil(url, marker, attempt + 1);
    throw new Error(`Incomplete response: ${url}`);
  }

  return body;
}

The production script uses bounded connect and total timeouts. It records a page failure rather than treating a partial page as an empty store.

That distinction prevents a dangerous false conclusion: “zero offers” when the real result is “the response was incomplete.”

3. Merge localized representations by stable identity

Each English and Russian store page contains the same offer identity with localized fields. The crawler merges them by numeric offer ID:

const record = records.get(id) || {
  id,
  store_slug: store.slug,
  code: item.code || '',
  type: item.type || '',
  is_verified: Boolean(item.is_verified),
  expires_at: item.expires_at || null,
  title_en: '',
  title_ru: '',
  description_en: '',
  description_ru: '',
  terms_en: '',
  terms_ru: ''
};

record[`title_${locale}`] = plain(item.title);
record[`description_${locale}`] = plain(item.description);
record[`terms_${locale}`] = plain(item.terms);
records.set(id, record);

This lets one rule compare translations without creating two “different” coupon records. It also makes missing-locale failures explicit.

4. Normalize for comparison, preserve raw meaning

For equality and duplicate checks, we remove markup, collapse whitespace, lowercase text, and normalize common quotation marks.

function plain(value = '') {
  return String(value)
    .replace(/<[^>]*>/g, ' ')
    .replace(/\s+/g, ' ')
    .trim();
}

function normalize(value) {
  return plain(value)
    .toLocaleLowerCase()
    .replace(/[“”„]/g, '"')
    .replace(/[’]/g, "'");
}

Normalization is deliberately modest. Transliteration or aggressive punctuation removal could merge genuinely different codes or terms.

5. Make rules explain themselves

Every finding has a severity, machine-readable code, field, and shopper-facing explanation.

function add(issues, severity, code, field, message) {
  issues.push({
    severity,
    rank: severityRank[severity],
    code,
    field,
    message
  });
}

Examples include:

• EXPIRED_PUBLIC: the structured expiration timestamp is in the past;
• CODE_PLACEHOLDER: text such as “No code required” was stored as a code;
• UNRESOLVED_TEMPLATE: a partner-network macro remains in public copy;
• PUBLISHER_COPY: webmaster or commission instructions leaked into shopper text;
• DUPLICATE_IN_STORE: the same normalized code or title appears more than once within one store;
• EN_TEXT_NOT_ENGLISH: an English field contains substantial Cyrillic text.

A code is more useful than an opaque score because it maps directly to an owner and a deterministic repair.

6. Control false positives with scope

Regex audits become noisy when they ignore context. We reduce noise in several ways.

First, duplicate detection is store-scoped. The same public code can legitimately exist at two different merchants.

const identity = code
  ? `code:${normalize(code)}`
  : `title:${normalize(title)}`;

const key = `${storeSlug}|${identity}`;

Second, language checks require a minimum amount of text. A brand name or a short SKU should not trigger a translation defect.

Third, past dates found inside prose are lower-confidence than a structured expired timestamp. The rule can flag them for review without declaring the whole offer invalid.

Fourth, “No code required” is recognized as a placeholder only when it occupies the code field. The same phrase can be legitimate shopper guidance in a description.

7. Concurrency without losing determinism

The crawler uses a shared index and a fixed number of workers. Records are sorted by numeric ID before output, and findings are sorted by severity and code.

const indexRef = { value: 0 };

await Promise.all(
  Array.from({ length: concurrency }, () =>
    worker(urls, indexRef, records, errors)
  )
);

const coupons = [...records.values()]
  .sort((a, b) => Number(a.id) - Number(b.id));

Network completion order can change, but the JSON diff stays stable. This matters in CI and in human review.

8. Report uncertainty separately from defects

In our August 31 run, 444 of 2,058 public offers carried a positive verification flag. The remaining 1,614 were not automatically declared false. They were simply not proven verified by the public payload.

The copy-quality rules marked 488 offers with at least one issue. The largest group was 444 offers with a title but no description or terms. That is a content completeness problem, not proof that the underlying discount is invalid.

Keeping these dimensions separate avoids a common analytics mistake:

offer truth != copy quality != checkout eligibility

One record can have clean copy and uncertain evidence. Another can have a valid merchant offer and broken localization. The repair queues are different.

9. Preserve a reproducible artifact

The live run produced a 3,191,808-byte JSON artifact with SHA-256:

e0365a08968582f1e3cf00c9eaa3b7cf23c9a25d483cebd6b20684dfec332868

It records the source sitemap, start and finish timestamps, page and coupon counts, crawl errors, aggregate issue counts, every normalized record, and every finding.

The checksum does not prove the data is correct. It proves that later analysis refers to the same immutable result.

What I would add next

The crawler currently audits public representation. The next layer should store evidence provenance and age explicitly:

verification_type: merchant_confirmed | checkout_tested | community_reported | unverified
verified_at: ISO-8601 timestamp
market: country/storefront identifier
customer_scope: public | new_customer | member | targeted
evidence_url: merchant terms when public

Checkout testing must remain condition-aware. A successful US new-customer cart does not validate a code for an existing customer in another market.

The deeper lesson is not coupon-specific. Public-data audits should fail closed. Require complete responses, preserve stable identity across representations, separate uncertainty from defects, and emit findings that a human can reproduce. “We do not have enough evidence” is a useful system state. Hiding it behind a green badge is not.