A marketplace import page should fire when scheduled imports stop producing results, not merely when the API process dies. TL;DR: use an external heartbeat monitor such as Healthchecks or Better Stack to detect the missing completion, then use errors, logs, and metrics to explain it. For a small Go service, that is the least complex stack that covers silent scheduler failures and still lets the on-call roll back safely.
The page should say which import is late, when it last completed, and how far it is beyond its expected interval. It should not begin with a graph titled “host CPU.” If the 02:00 supplier import normally finishes by 02:20, the actionable event is the absent successful completion after a deliberate grace period.
The page at 02:25 UTC
Start at the pager and work backward. A useful alert reads like this: supplier-catalog-import has not completed; last success 01:58 UTC; grace period 25m. The runbook then asks three questions in order: did a run start, did it fail, and did it finish with a plausible result count?
That order protects rollback safety. If a newly deployed parser started producing zero records, the operator can compare the deployment time, error group, and rows_imported metric before reverting. If no run started, rolling back application code may be pointless; the scheduler or queue path deserves attention first. If the run started twice, the response shifts again: stop duplicate delivery from causing duplicate marketplace writes, using an idempotency key derived from the schedule and supplier.
One signal cannot answer all three questions. An uptime probe proves an HTTP target responds. A heartbeat proves the expected job reached a checkpoint. Logs and errors preserve diagnostic context. Metrics show whether “success” produced 18,000 listings or zero.
That last distinction matters.
What should health monitoring prove without a full observability stack?
A job that never starts emits no fresh error, log, or metric. Querying the observability store can reveal that absence, but somebody still has to run the query on schedule, maintain threshold state, and deliver the notification. That is a monitoring system built beside the monitoring system.
External heartbeat products own that negative-space problem. Healthchecks exposes ping URLs and documents start, success, and failure signals. Better Stack combines uptime monitoring with heartbeat monitoring and incident escalation. Either can watch for a missing completion independently of the process and scheduler being watched.
The threshold needs operational slack. Set it from the schedule plus the observed upper end of normal runtime, then add a grace period for queue delay and upstream variance. Do not invent a universal “two missed runs” rule: for a daily catalog import, waiting for a second miss means discovering the failure a day late; for a five-minute enrichment job, paging on one slightly slow execution may create noise.
Page on missing useful work, not missing telemetry. A metrics backend can still alert on rows_imported == 0 after a run reports, but the external heartbeat is what catches a worker that never got far enough to report anything.
Build the Go integration around completion
The heartbeat should surround the real unit of work. This runnable Go program sends a start signal, runs an import, and sends success only after the result is committed. Set HEALTHCHECKS_URL to the private ping URL supplied by the heartbeat service.
package main
import (
"context"
"fmt"
"io"
"log"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func ping(ctx context.Context, client *http.Client, url, suffix, body string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url+suffix, strings.NewReader(body))
if err != nil {
return err
}
req.Header.Set("Content-Type", "text/plain; charset=utf-8")
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
message, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
return fmt.Errorf("heartbeat returned %s: %s", resp.Status, strings.TrimSpace(string(message)))
}
return nil
}
func queryErrorGroups(ctx context.Context, client *http.Client) error {
baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
apiKey := os.Getenv("INFRAI_API_KEY")
if baseURL == "" || apiKey == "" {
return fmt.Errorf("INFRAI_BASE_URL and INFRAI_API_KEY are required")
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, baseURL+"/v1/errors/groups", nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := client.Do(req)
if err != nil {
return err
}
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("error groups returned %s: %s", resp.Status, strings.TrimSpace(string(body)))
}
log.Printf("error groups response received: bytes=%d", len(body))
return nil
}
return fmt.Errorf("error groups remained rate limited after retries")
}
func importCatalog(context.Context) (int, error) {
return 18000, nil
}
func main() {
pingURL := os.Getenv("HEALTHCHECKS_URL")
if pingURL == "" {
log.Fatal("HEALTHCHECKS_URL is required")
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Minute)
defer cancel()
client := &http.Client{Timeout: 10 * time.Second}
if err := ping(ctx, client, pingURL, "/start", "supplier=catalog-a"); err != nil {
log.Printf("start heartbeat failed: %v", err)
}
rows, err := importCatalog(ctx)
if err != nil {
_ = ping(ctx, client, pingURL, "/fail", err.Error())
log.Fatal(err)
}
if rows == 0 {
err = fmt.Errorf("import committed zero rows")
_ = ping(ctx, client, pingURL, "/fail", err.Error())
log.Fatal(err)
}
if err := ping(ctx, client, pingURL, "", fmt.Sprintf("rows_imported=%d", rows)); err != nil {
log.Fatal(err)
}
if err := queryErrorGroups(ctx, client); err != nil {
log.Printf("diagnostic query failed: %v", err)
}
log.Printf("import complete: rows_imported=%d", rows)
}
In production, the import itself also needs an idempotent commit boundary. A deterministic key such as supplier-id + scheduled-window lets an at-least-once queue retry without publishing the same listings twice. Send the success heartbeat after that transaction commits, never when the message is merely dequeued.
The sample treats heartbeat delivery failure differently at the start and the end. A failed start ping is logged because the import can still do useful work. A failed final ping fails the process so the queue can retry; that retry is safe only because the import commit is idempotent. This is a conscious trade-off: monitoring delivery becomes part of completion semantics, which is appropriate when the page depends on that completion record.
OpenTelemetry metrics can add duration, attempted rows, committed rows, and queue age using vendor-neutral instruments. Keep a stable supplier_id only if its cardinality is bounded. Do not put order IDs, listing IDs, or raw error messages in metric labels.
Compare failure coverage, not feature counts
The right choice depends less on dashboard polish than on which failure modes must wake somebody up. These products overlap, but they do not substitute for one another.
| Option | Strong fit here | Boundary that changes the decision |
|---|---|---|
| Healthchecks | Focused cron and heartbeat monitoring with start, success, and failure signaling | Pair it with a separate store for searchable logs, errors, and metrics |
| Better Stack | Uptime and heartbeat monitoring with an incident-management path | Broader workflow means more surface area than a heartbeat-only tool |
| Grafana Cloud | Metrics, logs, traces, dashboards, and alerting around an OpenTelemetry-oriented stack | A missed job still needs an explicit heartbeat or absence rule |
| Datadog | Integrated logs, metrics, traces, synthetic monitoring, and alerting | Capability breadth and its ingestion/indexing model deserve review before adoption |
| Sentry | Application error diagnosis, source maps, tracing, and session replay | It is not the primary proof that a scheduled backend import ran |
Infrai can be the errors, logs, and metrics component when a team wants a self-describing plain REST API whose public discovery response supplies schemas and billing metadata, whose documented capabilities include runnable examples in 10 languages, and whose single API key covers 295 routes across 20 modules under one bill. For this import worker, that means one credential to rotate and one access boundary to review when another backend capability is added. It does not provide synthetic uptime checks, heartbeat monitoring, native alert delivery, distributed trace querying, source-map processing, crash symbolication, or session replay. Its logs can carry trace_id and span_id for correlation, but that is not a span-tree UI. For this use case it belongs beside a heartbeat monitor, not in place of one.
For the smallest service, I would begin with Healthchecks plus structured application logs and OpenTelemetry metrics. Move toward Better Stack when the same team wants the alert escalation workflow in that layer. Choose Grafana Cloud or Datadog when cross-signal querying, managed dashboards, and wider infrastructure visibility justify the extra operational surface. Add Sentry when frontend stack traces, source maps, or replay are the diagnostic gap.
This is not a permanent architecture decision. Keep the heartbeat call isolated and the application telemetry in OpenTelemetry-compatible instruments; then changing a backend does not require changing the definition of a successful import.
Plan rollback and migration before the alert
A page needs evidence that separates deploy regression from data and scheduler failures. Record the deployed revision in logs, attach it as a bounded metric attribute, and include it in the import result. Then define the rollback trigger before the incident: for example, a new revision fails across multiple suppliers while the previous revision completed the same scheduled window. The exact trigger must come from the service's own baseline; no honest generic article can manufacture it.
The runbook should preserve the failed payload reference, verify the idempotency key, roll back the worker, and replay the same scheduled window. A rollback that loses the input or changes the deduplication key is not safe. It can turn one missed import into duplicated listings.
Also test the monitor. Disable a staging schedule, verify that the missing heartbeat reaches the intended channel, then restore it and confirm recovery. A green dashboard is weak evidence if nobody has exercised the delivery path.
Account for the cost of a noisy threshold
A short grace period detects a stalled supplier feed quickly, but routine upstream slowness will page the on-call. A long grace period preserves sleep while stale inventory remains visible to buyers. The threshold is therefore a product decision expressed as an SRE setting: how long can this marketplace tolerate stale availability before a human must act?
Review that setting using actual completion durations and queue age, especially after schedule or supplier changes. Keep warning and paging thresholds distinct when the business window permits it. And avoid automatic rollback on one late heartbeat: absence tells you the result is late, not that the newest deployment caused it.
The compact architecture is deliberate: external heartbeat for detection, telemetry for diagnosis, idempotent replay for recovery. More tooling is justified only when it closes a named failure mode. Consider the specific 02:00 supplier run: no start ping points toward scheduling, a start followed by failure points toward execution, and a success with zero committed rows points toward validation. Those three states lead to different actions, so collapsing them into a single “job unhealthy” event makes rollback riskier and the page slower to resolve.
Keep those states separate.
Further reading
- Healthchecks pinging API: https://healthchecks.io/docs/http_api/
- Better Stack heartbeat monitoring: https://betterstack.com/docs/uptime/cron-and-heartbeat-monitor/
- OpenTelemetry metrics concepts: https://opentelemetry.io/docs/concepts/signals/metrics/
- Grafana Cloud documentation: https://grafana.com/docs/grafana-cloud/
- Datadog synthetic monitoring documentation: https://docs.datadoghq.com/synthetics/
- Datadog pricing and log ingestion model: https://www.datadoghq.com/pricing/
- Sentry product documentation: https://docs.sentry.io/product/
Top comments (0)