DEV Community

NorbertChristensen3183
NorbertChristensen3183

Posted on

Marketplace Health Errors — Tracking Failed Checks Without Unsafe Rollbacks

The central trade-off is diagnostic precision versus rollback safety: a marketplace should alert when a scheduled import stops producing results, but a timeout, connection refusal, and DNS failure must not be collapsed into one generic fetch error or allowed to trigger an automatic rollback without evidence that the deployed application caused it. Short answer: record the probe phase, a bounded failure class, the checked target's stable identity, the import schedule, the last successful result time, and the deployment revision; group on stable fields, search the volatile details, and require both loss of output and deployment correlation before treating rollback as a defensible response.

This separates two questions that operators often ask under pressure. "Why could the health endpoint not be reached?" is a network diagnosis. "Why did the import stop producing marketplace results?" is a business-progress diagnosis. They may share a cause, but neither proves the other.

How should error tracking classify failed health endpoint checks?

A health request is an observation from one origin at one instant. DNS may fail before any connection exists; a connection may be refused after resolution; a request may exceed its deadline after a connection is established. Recording each outcome as health check failed destroys the phase information needed for triage. Alerting on every raw error, however, turns a transient probe result into an operational verdict.

Scheduled imports add a quieter failure mode. The process can answer its health endpoint while repeatedly importing zero usable records, waiting on a queue, or failing before committing a result. Conversely, the health endpoint can be unreachable from one probe location while an already-running import continues to commit valid results. The decisive signal is therefore a joined condition: expected work is overdue and successful output has not advanced. Probe failures raise confidence and narrow the search; they do not define success.

Healthy is ambiguous.

The schedule supplies the compliance limit. If an import is due every 15 minutes and the operating policy allows one missed run, the alert boundary must be expressed as that explicit policy, including clock-skew and execution allowances, rather than inferred from a convenient dashboard window. Do not invent an availability promise after the page fires.

Build an audit trail before choosing a group key

The event model should preserve evidence without creating an unbounded metric dimension. Prometheus's instrumentation guidance warns against labels whose cardinality can grow without bound and suggests investigating alternate designs when a metric could exceed roughly 100 label values. That rules out URLs with arbitrary query strings, raw exception messages, import IDs, seller IDs, and stack traces as metric labels. Those values still belong in logs or traces, where operators can search them under appropriate retention and access controls.

A compact event can carry both bounded classification and searchable detail:

package probe

import "time"

type FailureClass string

const (
    DNSFailure        FailureClass = "dns"
    ConnectionRefused FailureClass = "connection_refused"
    DeadlineExceeded  FailureClass = "deadline_exceeded"
    HTTPFailure       FailureClass = "http_status"
)

type CheckEvent struct {
    ObservedAt       time.Time
    TargetKey        string // Stable service identity, never a raw URL.
    ProbeRegion      string
    FailureClass     FailureClass
    StatusCode       int
    Duration         time.Duration
    DeploymentRev    string
    ScheduleKey      string
    LastResultAt     time.Time
    AttemptID        string // Searchable audit field, not a metric label.
    Detail           string // Sanitized diagnostic text for restricted logs.
}
Enter fullscreen mode Exit fullscreen mode

The small taxonomy is intentional. dns, connection_refused, deadline_exceeded, and http_status answer different operational questions, while the exact hostname resolution error or fetch message remains searchable. For a Node.js probe using fetch, normalize the runtime's error and its cause at the collection boundary; do not use the full message as the group identity. Runtime messages can contain unstable target data, and two wording variants should not split one incident.

The tempting first design is to fingerprint the complete exception because it appears to preserve precision. The correction is to separate identity from evidence. Consider an import due at 10:00 with one missed run allowed: a timeout at 10:01 is evidence about one attempt, not proof of a stopped schedule; another timeout at 10:16, no committed result since the prior permitted window, and the same failure class from several probe regions form a materially different case. The incident key remains stable across those observations, the individual attempt IDs retain their exact chronology, and a deployment revision can be tested as a correlation rather than silently promoted to a cause. If resolution fails in only one region while committed results continue, the same model records the fault but does not authorize rollback. This example is deliberately asymmetric because false rollback and delayed diagnosis do not carry the same operational cost.

Keep the original event immutable. If a later classifier version changes the category, append the derived classification with its version rather than rewriting history. This is the same discipline that makes ledger reconciliation possible: raw evidence, deterministic derivation, and an audit trail linking the two.

Group for action; search for evidence

Grouping should correspond to an action boundary. The useful default key is target_key + probe_region + failure_class, optionally partitioned by deployment revision during an active rollout. It should exclude attempt IDs, timestamps, durations, raw hostnames, and exception text. A deadline exceeded from three probe regions is meaningfully different from a refusal in one region, yet 400 identical deadline events do not deserve 400 incidents.

Search has the opposite purpose. Preserve the fields needed to reconstruct a sequence: schedule key, attempt ID, deployment revision, observed time, last successful result time, bounded class, status code, and sanitized detail. A practical investigation starts with the affected schedule and a time range, then pivots to the target and deployment revision. Only then should an operator inspect raw errors.

Signal Group dimension? Search field? Reason
Failure class Yes Yes Bounded category maps to a diagnostic path
Probe region Yes Yes Distinguishes local reachability from broad failure
Deployment revision Temporarily Yes Supports rollout correlation without becoming permanent incident identity
Attempt ID No Yes Unique values preserve auditability but explode cardinality
Raw error message No Yes, sanitized Valuable evidence, unstable identity
Last successful result time No Yes Establishes business-progress loss

This division also constrains sensitive data. Marketplace import records may contain merchant or catalog identifiers; retain only an opaque, stable schedule key in general operational telemetry, and apply the organization's access and retention rules to detailed logs. Observability data is not exempt from compliance boundaries merely because engineers produced it.

Retries can obscure the outage

Retries are appropriate only when the operation is safe to repeat and the caller has a finite budget. AWS's guidance on timeouts, retries, exponential backoff, and jitter explains two relevant hazards: retries can amplify load on a failing dependency, and retry timing should include jitter so clients do not synchronize. A probe should therefore have an explicit overall deadline and a small attempt budget; its telemetry should record every attempt while the alert evaluator counts the check cycle once.

Import execution is stricter. A retry that can duplicate marketplace results or advance a cursor twice is not recovery. The write path needs an idempotency key or equivalent uniqueness constraint, and the audit record must distinguish attempted, committed, and reconciled. Exactly-once delivery is not something a health endpoint can establish; the practical goal is an idempotent effect with a durable record that reconciliation can verify.

Stop there. More retries can make the dashboard greener while delaying the evidence that the schedule has missed its permitted completion window.

Rollback logic needs the same restraint. A new revision followed by widespread refusals and stalled output is strong correlation. A DNS failure isolated to one probe region is not. Automatic rollback should require a recent revision, a defined breadth of failed checks, and lack of committed results; otherwise page an operator and preserve the current state for diagnosis. This bias costs a little response time, but it avoids replacing healthy code because a resolver or probe path failed elsewhere.

Roll out the classifier without losing evidence

Begin in shadow mode: emit the new bounded class and progress fields alongside the existing alert, but do not change paging or rollback behavior. Compare incident counts, unknown classifications, region distribution, and the link between missed schedules and absent commits. The important acceptance test is replayable: given the same stored events and classifier version, the evaluator must reach the same incident decision.

Next, route alerts from the joined business-progress rule while keeping rollback manual. Exercise DNS failure, refused connection, fetch timeout, healthy-but-stalled import, and duplicate delivery in controlled tests. Verify that a retry creates a new attempt record without creating a second committed result.

Only after the audit trail survives reconciliation should automated rollback be enabled for the narrow, deployment-correlated case. Keep a switch that disables the automation without disabling detection, and deploy classifier changes independently from the probe. The durable design is conservative: output progress decides urgency, bounded network classes guide diagnosis, and immutable attempt records make every rollback decision reviewable.

Evidence first.

Sources

Top comments (0)