DEV Community

Mikhail Nikitenkov
Mikhail Nikitenkov

Posted on Fully Autonomous

Building a public coupon-quality crawler that fails closed

#javascript #webdev #automation #tutorial

Coupon data has an awkward property: most defects look plausible. An expired offer can still have polished copy. A publisher instruction can look like shopper-facing terms. The same offer can appear on two localized pages with different text. A code can be real but inapplicable to a particular cart.

At CouponX we needed a repeatable way to audit what the public site actually exposed, independent of what the database or an affiliate feed claimed. This post explains the crawler design we used for a live run over 1,078 localized store pages and 2,058 unique offers.

Disclosure: I am writing on behalf of CouponX. The implementation and failure modes are the subject of this article; the post is not a claim that every CouponX offer works for every shopper.

Design goal: audit the public contract

The crawler starts from the sitemap and reads the same public page payload available to a visitor. That boundary is important. A database record can be correct while a localization bug, serializer, cache, or template exposes something different in production.

The audit therefore answers:

Does the public page contain the fields and evidence required for the claim it makes?

It does not answer whether a private, targeted, or app-only offer will apply to an arbitrary customer. When evidence is missing, the correct output is unverified, not a guessed success state.

1. Discover only the intended page type

The sitemap contains many URL classes. We select localized store pages explicitly instead of crawling every internal link.

function storeUrls(sitemap) {
  return [...sitemap.matchAll(/<loc>(.*?)<\/loc>/g)]
    .map((match) => decodeEntities(match[1]))
    .filter((url) => /\/(?:en|ru)\/store\//.test(url));
}
Enter fullscreen mode Exit fullscreen mode

This is a small but useful safety property: a navigation change cannot silently expand the crawl into account pages, search grids, or tracking URLs.

2. Require a completeness marker

Network success is not payload success. A proxy or origin can close a response after returning enough HTML to look superficially valid.

Our fetch wrapper requires a marker expected at the end of the document and retries only when that marker is absent:

async function requestUntil(url, marker, attempt = 1) {
  const body = await fetchBody(url);

  if (!body.includes(marker)) {
    if (attempt < 3) return requestUntil(url, marker, attempt + 1);
    throw new Error(`Incomplete response: ${url}`);
  }

  return body;
}
Enter fullscreen mode Exit fullscreen mode

The production script uses bounded connect and total timeouts. It records a page failure rather than treating a partial page as an empty store.

That distinction prevents a dangerous false conclusion: “zero offers” when the real result is “the response was incomplete.”

3. Merge localized representations by stable identity

Each English and Russian store page contains the same offer identity with localized fields. The crawler merges them by numeric offer ID:

const record = records.get(id) || {
  id,
  store_slug: store.slug,
  code: item.code || '',
  type: item.type || '',
  is_verified: Boolean(item.is_verified),
  expires_at: item.expires_at || null,
  title_en: '',
  title_ru: '',
  description_en: '',
  description_ru: '',
  terms_en: '',
  terms_ru: ''
};

record[`title_${locale}`] = plain(item.title);
record[`description_${locale}`] = plain(item.description);
record[`terms_${locale}`] = plain(item.terms);
records.set(id, record);
Enter fullscreen mode Exit fullscreen mode

This lets one rule compare translations without creating two “different” coupon records. It also makes missing-locale failures explicit.

4. Normalize for comparison, preserve raw meaning

For equality and duplicate checks, we remove markup, collapse whitespace, lowercase text, and normalize common quotation marks.

function plain(value = '') {
  return String(value)
    .replace(/<[^>]*>/g, ' ')
    .replace(/\s+/g, ' ')
    .trim();
}

function normalize(value) {
  return plain(value)
    .toLocaleLowerCase()
    .replace(/[“”„]/g, '"')
    .replace(/[’]/g, "'");
}
Enter fullscreen mode Exit fullscreen mode

Normalization is deliberately modest. Transliteration or aggressive punctuation removal could merge genuinely different codes or terms.

5. Make rules explain themselves

Every finding has a severity, machine-readable code, field, and shopper-facing explanation.

function add(issues, severity, code, field, message) {
  issues.push({
    severity,
    rank: severityRank[severity],
    code,
    field,
    message
  });
}
Enter fullscreen mode Exit fullscreen mode

Examples include:

  • EXPIRED_PUBLIC: the structured expiration timestamp is in the past;
  • CODE_PLACEHOLDER: text such as “No code required” was stored as a code;
  • UNRESOLVED_TEMPLATE: a partner-network macro remains in public copy;
  • PUBLISHER_COPY: webmaster or commission instructions leaked into shopper text;
  • DUPLICATE_IN_STORE: the same normalized code or title appears more than once within one store;
  • EN_TEXT_NOT_ENGLISH: an English field contains substantial Cyrillic text.

A code is more useful than an opaque score because it maps directly to an owner and a deterministic repair.

6. Control false positives with scope

Regex audits become noisy when they ignore context. We reduce noise in several ways.

First, duplicate detection is store-scoped. The same public code can legitimately exist at two different merchants.

const identity = code
  ? `code:${normalize(code)}`
  : `title:${normalize(title)}`;

const key = `${storeSlug}|${identity}`;
Enter fullscreen mode Exit fullscreen mode

Second, language checks require a minimum amount of text. A brand name or a short SKU should not trigger a translation defect.

Third, past dates found inside prose are lower-confidence than a structured expired timestamp. The rule can flag them for review without declaring the whole offer invalid.

Fourth, “No code required” is recognized as a placeholder only when it occupies the code field. The same phrase can be legitimate shopper guidance in a description.

7. Concurrency without losing determinism

The crawler uses a shared index and a fixed number of workers. Records are sorted by numeric ID before output, and findings are sorted by severity and code.

const indexRef = { value: 0 };

await Promise.all(
  Array.from({ length: concurrency }, () =>
    worker(urls, indexRef, records, errors)
  )
);

const coupons = [...records.values()]
  .sort((a, b) => Number(a.id) - Number(b.id));
Enter fullscreen mode Exit fullscreen mode

Network completion order can change, but the JSON diff stays stable. This matters in CI and in human review.

8. Report uncertainty separately from defects

In our August 31 run, 444 of 2,058 public offers carried a positive verification flag. The remaining 1,614 were not automatically declared false. They were simply not proven verified by the public payload.

The copy-quality rules marked 488 offers with at least one issue. The largest group was 444 offers with a title but no description or terms. That is a content completeness problem, not proof that the underlying discount is invalid.

Keeping these dimensions separate avoids a common analytics mistake:

offer truth != copy quality != checkout eligibility
Enter fullscreen mode Exit fullscreen mode

One record can have clean copy and uncertain evidence. Another can have a valid merchant offer and broken localization. The repair queues are different.

9. Preserve a reproducible artifact

The live run produced a 3,191,808-byte JSON artifact with SHA-256:

e0365a08968582f1e3cf00c9eaa3b7cf23c9a25d483cebd6b20684dfec332868
Enter fullscreen mode Exit fullscreen mode

It records the source sitemap, start and finish timestamps, page and coupon counts, crawl errors, aggregate issue counts, every normalized record, and every finding.

The checksum does not prove the data is correct. It proves that later analysis refers to the same immutable result.

What I would add next

The crawler currently audits public representation. The next layer should store evidence provenance and age explicitly:

verification_type: merchant_confirmed | checkout_tested | community_reported | unverified
verified_at: ISO-8601 timestamp
market: country/storefront identifier
customer_scope: public | new_customer | member | targeted
evidence_url: merchant terms when public
Enter fullscreen mode Exit fullscreen mode

Checkout testing must remain condition-aware. A successful US new-customer cart does not validate a code for an existing customer in another market.

The deeper lesson is not coupon-specific. Public-data audits should fail closed. Require complete responses, preserve stable identity across representations, separate uncertainty from defects, and emit findings that a human can reproduce. “We do not have enough evidence” is a useful system state. Hiding it behind a green badge is not.

Top comments (1)

Collapse
 
crdtcto profile image
Kane Lim •

Hello Glad to see you, I am Kane Lim from Hong Kong. I have over 10 years of development experience. I am writing this because your post was interesting.

I really like the fail closed philosophy here, especially the distinction between payload integrity, content quality, and actual offer eligibility. That separation is what makes this architecture reliable rather than just another scraping pipeline.

I would push this further by treating every crawl result as an evidence graph instead of a flat audit record. Each offer could reference its source URL, locale, extraction timestamp, normalization version, rule version, and evidence confidence. This makes historical comparisons and regression analysis much stronger.

I would also introduce schema validation before semantic rules. A malformed response, missing completeness marker, stale cache response, or unexpected DOM structure should produce an explicit UNKNOWN state rather than entering the validation pipeline as empty data.

For duplicate detection, stable identity plus content fingerprints would help detect subtle localization drift. A hash of normalized terms, expiration metadata, and eligibility scope can reveal when two localized representations silently diverge.

The next major improvement would be differential crawling. Compare the current public representation against the previous known good artifact and trigger alerts only for meaningful semantic changes. That dramatically reduces noisy CI failures while catching serializer, localization, cache, and template regressions.

Your point about “not enough evidence” being a valid state is particularly important. In distributed data systems, UNKNOWN is often far safer than FALSE or TRUE.

Excellent engineering approach. I would enjoy exchanging ideas around evidence driven crawlers, deterministic audits, and resilient data pipelines.