Your scraper isn't broken: a decision table for recurring data changes

Recurring scraping jobs rarely fail because the selector is wrong. They fail because the pipeline cannot tell the difference between a value that changed, a value that was not observed, and a value that was observed badly.

Those three cases need different handling. Most implementations collapse them into one: "the new value is different, so overwrite the old one." That is how a temporary timeout becomes a permanent null, and how a layout change quietly corrupts a catalog.

The three states

For every field, record the observation before deciding what it means:

• Changed: the source returned a complete response, the expected element was present, and the normalized value differs from the last accepted value.
• Unchanged: the source returned a complete response and the normalized value matches.
• Unknown: the request timed out, returned a partial page, hit an auth wall, or the element was missing. The previous value is still the best available value; it is not evidence that the field is now empty.

The important rule is simple: an unknown observation must never overwrite a known value.

A decision table that survives real sites

Previous value	Current observation	Action
Known	Complete + same	Skip write; update last_seen_at
Known	Complete + different	Write new value with evidence
Known	Timeout / partial / missing element	Keep previous value; mark unknown; retry
Missing	Complete + value	Write value with evidence
Missing	Complete + absent	Keep missing; do not invent a default
Any	Parser version changed	Re-validate a sample before writing

That last row matters more than it looks. A parser deploy can change normalization without changing the site. If you write every row immediately, a bug in the parser looks exactly like a change in the source.

Store evidence, not just values

A value without provenance is hard to defend later. At minimum, keep:

observed_at
source_url
http_status
content_hash
parser_version
previous_value
decision          -- changed | unchanged | unknown
reason            -- timeout | partial | selector_missing | ok

With those fields you can answer the two questions that matter during an incident: when did this value change? and was the source actually observed, or did we simply stop seeing it?

Make unknown visible

A job can return 200 OK and still learn nothing. Track two separate metrics:

• Job success rate: did the runner finish?
• Observation coverage: for each required field, what percentage of runs produced a complete observation?

Coverage is the one that catches silent data loss. A scraper with 99% job success and 60% field coverage is not healthy; it is just quiet.

A small implementation sketch

type Observation =
  | { state: "complete"; value: string | null; evidence: Evidence }
  | { state: "unknown"; reason: "timeout" | "partial" | "missing"; evidence: Evidence };

function decide(previous: string | null, current: Observation) {
  if (current.state === "unknown") {
    return { write: false, next: previous, flag: "unknown" as const };
  }
  if (current.value === previous) {
    return { write: false, next: previous, flag: "unchanged" as const };
  }
  return { write: true, next: current.value, flag: "changed" as const };
}

The function is deliberately boring. The useful part is the contract: callers cannot write unless the observation is complete, and every decision carries a reason.

The operational checklist

Before adding another retry or proxy, check these first:

1. Does a failed observation preserve the last accepted value?
2. Is unknown a first-class state, or is it stored as null?
3. Can you trace a disputed value back to its source observation?
4. Do you alert on field coverage, not only job status?
5. Does a parser change require re-validation before writes?

Recurring collection gets easier when the pipeline stops pretending it always knows the answer. Separating changed, unchanged, and unknown removes a whole class of data corruption before it reaches the warehouse.

I wrote a longer field guide with the handoff and review steps here: Incremental product data collection: detect changes without losing freshness.