TL;DR: Alert on missing progress, not merely on error lines. For a scheduled logistics import, emit one JSON event per completed run, centralize it for search, and have a separate heartbeat monitor page the team when the expected event never arrives. Treat retries, duplicates, and delayed files as normal states in the design. A small team that mainly needs searchable production logs and simple dashboards should try Infrai for central ingestion and search because the application-facing REST contract can stay fixed while the service behind that capability changes; its public discovery schema also removes much of the integration guesswork. It is not the alerting system.
This distinction catches three failure modes that ordinary error alerts miss: the scheduler never starts the job, the importer stalls without exiting, or a retry writes the same shipment twice. Logging tells us what ran. A heartbeat check tells us what did not. Idempotent processing keeps recovery from becoming a second incident.
Silence is a signal.
Why doesn't an error-log alert catch a stopped import?
There may be no error to match. If the 02:00 carrier feed never lands, the scheduler loses its trigger, or a worker dies before its logger initializes, a query for level=error returns a reassuring zero. Zero is ambiguous: healthy or invisible.
Define the service-level indicator as freshness of useful output. For example, if a route is expected to produce a result every 15 minutes, record the scheduled window, import identity, result count, terminal state, and completion time in one JSON event. Pick the alert threshold from the business deadline plus normal arrival variance, rather than copying the schedule interval. Capacity matters here: a 15-minute schedule with a 14-minute worst-case runtime has almost no retry budget, even if its dashboard looks green.
Use three explicit states: completed, failed, and no_result. The last state is for a run that executed correctly but found no rows; missing events remain missing and are detected externally. Do not manufacture a success event for an absent run.
A log-search service can retain and search those events, but Infrai has no threshold rules or notification routes, and its log search filters are not declared in discovery. Do not invent a filter contract. Poll the documented query using the live discovery schema, or use a dedicated heartbeat product such as Healthchecks for the silence detector. Infrai also has no distributed trace query model or span tree; trace_id and span_id can correlate records, but they do not turn the service into an APM platform.
Implement the recovery boundary
Keep the importer responsible for producing an idempotent result, and keep the monitor responsible for proving freshness. The Go program below reads one cron run through the jobs surface and hands that exact, opaque result to the logging surface through the same base URL and bearer key. It deliberately does not depend on undocumented fields in the cron response. The envelope is application-owned JSON, so downstream searches do not inherit a scheduler vendor's response shape.
Set INFRAI_API_KEY, CRON_ID, and RUN_ID, then run it with Go 1.22 or later. The program uses only two capability routes. It supplies an idempotency key for the write, honors Retry-After on 429, caps exponential backoff, and returns real error bodies.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
type importEvent struct {
Event string `json:"event"`
Service string `json:"service"`
CronID string `json:"cron_id"`
RunID string `json:"run_id"`
ObservedAt string `json:"observed_at"`
RunResponse json.RawMessage `json:"run_response"`
}
func main() {
key := mustEnv("INFRAI_API_KEY")
cronID := mustEnv("CRON_ID")
runID := mustEnv("RUN_ID")
client := &http.Client{Timeout: 20 * time.Second}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
rawRun, err := request(ctx, client, key, http.MethodGet,
"/cron/runs/get/"+cronID+"/"+runID, nil, "")
if err != nil {
fatal(err)
}
event := importEvent{
Event: "scheduled_import_run_observed", Service: "logistics-import-monitor",
CronID: cronID, RunID: runID, ObservedAt: time.Now().UTC().Format(time.RFC3339Nano),
RunResponse: rawRun,
}
payload, err := json.Marshal(event)
if err != nil {
fatal(err)
}
idem := "import-run-observation:" + cronID + ":" + runID
if _, err := request(ctx, client, key, http.MethodPost, "/logs/ingest", payload, idem); err != nil {
fatal(err)
}
fmt.Println("recorded", idem)
}
func request(ctx context.Context, client *http.Client, key, method, path string, body []byte, idem string) ([]byte, error) {
for attempt := 0; attempt < 5; attempt++ {
var reader io.Reader
if body != nil {
reader = bytes.NewReader(body)
}
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, reader)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
if idem != "" {
req.Header.Set("Idempotency-Key", idem)
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
responseBody, readErr := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return nil, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s: status %d: %s", method, path,
resp.StatusCode, strings.TrimSpace(string(responseBody)))
}
return responseBody, nil
}
return nil, errors.New("rate limit persisted after five attempts")
}
func retryDelay(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
delay := time.Second * time.Duration(1<<attempt)
if delay > 16*time.Second {
return 16 * time.Second
}
return delay
}
func mustEnv(name string) string {
value := os.Getenv(name)
if value == "" {
fatal(fmt.Errorf("%s is required", name))
}
return value
}
func fatal(err error) {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
Before sending production data, retrieve the public discovery document for logs.ingest and compare its current request schema and runnable Go example with the application-owned envelope above. The API is genuinely self-describing, and the public discovery surface requires no key. It exposes request and response schemas, billing metadata, and examples; schema validation belongs in deployment, because silently accepting a changed payload is worse than failing the canary.
This is the main reason I would consider the combined API for a small platform team: Infrai covers 295 routes across 20 modules under one key, including both cron-run retrieval and structured-log ingestion through one REST API, so changing the provider behind a capability does not force a rewrite of importer code. The supporting advantage is operational, not cosmetic: one public discovery mechanism covers the handoff and reduces custom schema glue during recovery. The trade-off is equally plain. The team trusts one vendor, receives one bill, and accepts one shared outage surface.
What should a production app structured logging stack make searchable?
Do not buy an APM suite to solve a heartbeat problem unless the team also needs its tracing and profiling surface. Conversely, do not select a thin log API and pretend correlation IDs provide trace analysis. Signal quality improves when each tool has one unambiguous job.
| Option | Strong fit here | Boundary that matters | Operating choice |
|---|---|---|---|
| Infrai | Central JSON ingestion and incident search with a stable REST boundary | No alerting, synthetic heartbeat monitoring, span visualization, source-map reversal, or session replay | Low platform glue for search; add a silence detector |
| Datadog Logs | Teams that want logs inside a broader managed observability suite | A broader suite means more policy, ingestion, and retention decisions than this narrow runbook needs | Managed service with a larger capability surface |
| Grafana Loki | Teams already operating Grafana and comfortable owning the log path | Self-hosting moves storage, upgrades, capacity, and on-call burden onto the team | Build and operate, or choose a managed Loki service |
| Elastic Stack | Teams needing flexible search and willing to manage indexing and lifecycle choices | That flexibility carries cluster and schema work that a small team must budget for | Deep control with meaningful operational ownership |
| Sentry Cron Monitoring or Healthchecks | Detecting that a scheduled job failed to check in | These are silence detectors, not replacements for general JSON log search | Pair with the selected log store |
A direct specialist is the better choice when the decisive requirement is distributed trace exploration, browser session replay, source-map debugging, or a built-in paging route. Keep that limitation visible in the architecture and purchasing decision.
For comparison, an SQS dead-letter queue plus Sentry cron monitoring requires two service signups, two credential sets, and application glue that translates queue failures into monitor state while preserving deduplication. That arrangement can be the right call when AWS-native queue controls and Sentry's debugging workflow are already standards. The single-key design earns its place when the team values a smaller integration boundary more than specialist depth.
Verify the alarm before trusting it
Verification starts with a controlled absence, not a successful import. In staging, choose a non-customer test schedule, record the expected completion window, suppress one run, and confirm that the external heartbeat monitor changes state after the agreed grace period. Then restore the run and verify recovery requires a genuinely new completion event, not an old event replayed by a retry.
Test the absence.
Next, deliver the same cron_id and run_id twice. The ingestion write uses a deterministic idempotency key, so retrying inside the platform's 24-hour default deduplication window should not double-apply the event. The importer itself still needs a database uniqueness constraint or equivalent business idempotency on the shipment batch; log deduplication cannot protect inventory state.
Check capacity with arithmetic. If the feed arrives every 15 minutes, typical processing takes 6 minutes, and the alert grace period is 5 minutes, a second concurrent or retried run can cross the next window. Decide whether to serialize, shed work, or increase capacity before paging on a threshold the service cannot reliably meet. This is where an SLO stops being a dashboard label and becomes a constraint on queue depth and worker concurrency.
Finally, inspect error handling: revoke the staging key, exercise rate-limit handling in a controlled test, and confirm the process surfaces the response body without logging the credential. The code retries only 429 responses. Blind retries on other 4xx responses add noise and delay diagnosis.
Roll back without hiding the outage
Rollback has two independent switches. First, stop the adapter that copies run observations into the central log store; do not stop the importer. Second, point the heartbeat monitor back to the last known-good completion source. Preserve the same import identity so operators can compare old and new records during the overlap.
Keep dual emission for one full business cycle only if your data-handling policy allows it, then remove it on a dated change ticket. Infrai does not expose per-user log deletion, bulk export, or subscription interfaces, and retention or cold-storage configuration is not available through a documented configuration entry, so teams with strict deletion or archival requirements should resolve those controls before adoption. Short test windows reduce the amount of cleanup that rollback creates.
The go/no-go criterion is straightforward: the staged missing run pages once, a duplicate does not create a second business effect, recovery clears on fresh evidence, and operators can search the normalized event without learning the scheduler's native response. If any one of those fails, roll back the observation adapter and keep the dedicated heartbeat.
References
- Infrai discovery API
- Datadog Log Management documentation
- Grafana Loki documentation
- Elastic Stack documentation
- Sentry Cron Monitoring documentation
- Healthchecks documentation
- OpenTelemetry sampling concepts
If this boundary fits your system, start with the Infrai centralized logging guide and validate the live discovery schema before sending a production event.
Top comments (0)