Short answer: treat a scheduled import as healthy only when its expected result is fresh, attributable, and usable. A free uptime check can tell you that an HTTP endpoint answered; it cannot, by itself, tell you that yesterday's patient eligibility file produced today's records. For a healthtech application, I would attach four signals to each import: schedule freshness, completion status, result count, and owner cost attribution. Alert on the missing combination, then verify the underlying job before paging someone.
That distinction matters because a green web probe can coexist with a broken import. The application may serve its login page while a queue consumer is stalled, a file parser is rejecting every row, or a retry has delivered the same batch twice. The runbook below keeps the monitoring layer small enough to operate on a free budget while preserving the evidence needed to investigate a miss.
What does an uptime check miss?
Start with the failure mode, not the tool. A web application has at least two availability surfaces: the request path a user touches and the scheduled work that changes what the user sees. They fail independently.
For an example nightly eligibility import, an HTTP 200 proves that the health handler returned a response. It does not prove that the import ran, that it finished after the last expected window, or that its output passed validation. A queue can also produce a misleading success: the producer enqueues work, the scheduler records a run, and no consumer commits a result.
I use a small event record as the contract between the import and the monitor. The record is deliberately boring. It has the schedule identity, run identifier, completion time, result count, and cost owner. That last field is operationally useful: when several teams share one monitoring budget, a page without an owner becomes a meeting instead of a fix.
The four signals are:
- Freshness: did a run complete inside the expected window?
- Status: did the job report success rather than merely start?
- Result: did it produce a plausible, non-secret count that downstream checks can use?
- Attribution: which team, pipeline, or tenant owns the check and its follow-up?
A single failed probe is usually a retry candidate. A stale completion plus a missing result is a page candidate. Make that distinction explicit.
The trade-off is real.
Keep the page small.
How should the signal contract work?
Keep the public health endpoint cheap and the import evidence private. The endpoint can expose a summary such as last_success_at, run_id, and result_count; it should not expose patient data, source filenames, or row-level errors. If the endpoint is reachable but the timestamp is old, report a dependency failure even though the HTTP request succeeded.
Here is a small Go model and evaluator. The thresholds are example policy values, not universal defaults. The important part is that the monitor evaluates a completed result rather than a scheduler heartbeat.
package main
import (
"fmt"
"time"
)
type ImportResult struct {
Schedule string
RunID string
Owner string
Status string
Finished time.Time
Rows int
}
func evaluate(r ImportResult, now time.Time, maxAge time.Duration) error {
if r.Schedule == "" || r.RunID == "" || r.Owner == "" {
return fmt.Errorf("incomplete import identity")
}
if r.Status != "success" {
return fmt.Errorf("import %s has status %q", r.RunID, r.Status)
}
if r.Finished.IsZero() || now.Sub(r.Finished) > maxAge {
return fmt.Errorf("import %s is stale", r.RunID)
}
if r.Rows == 0 {
return fmt.Errorf("import %s produced no rows", r.RunID)
}
return nil
}
The zero-row rule needs domain judgment. A legitimate empty file can exist, so encode that exception in the import contract rather than teaching the monitor to guess. The same applies to a late run: a monthly schedule should not inherit a fifteen-minute threshold from a nightly one.
Where should free monitoring sit in the runbook?
Use an external HTTP check for reachability and a separate result check for freshness. The first catches DNS, TLS, routing, and application startup problems. The second catches the scheduled-work failures that a basic uptime probe cannot see. Both checks should carry the same stable schedule name and owner label so an alert can be assigned without opening five dashboards.
Do not put credentials or sensitive payloads in a public probe. A safer arrangement is an authenticated internal evaluator that publishes a boolean or small status document to a restricted endpoint. The external check reads only that document. For a healthtech system, the response should be safe to log and safe to paste into an incident channel.
Cost attribution belongs in the event at creation time. Record the owner, schedule, environment, and reason for the check. Then aggregate alert volume by those fields. This gives a useful decision rule: keep a check when it detects a distinct failure or supports a required audit trail; remove it when it duplicates another signal without changing an operator's action. The cheapest check is the one that does not create an ambiguous page.
This design has limits. It is a poor fit for a stream that must be evaluated continuously, for a workflow whose valid result can only be known by a domain-specific reconciliation service, or for an import whose owner changes during execution. In those cases, a simple uptime check remains useful as a first alarm, but it cannot carry the whole correctness contract. The extra reconciliation signal costs operational attention, so I add it only when its failure changes the runbook.
There is a second boundary: result counts are evidence, not proof. A parser can produce 42,000 syntactically valid rows from the wrong source. If the import has a trusted source total or checksum, compare that value after the basic freshness check. If it does not, record the limitation instead of presenting the count as a complete validation result.
The monitor should also be idempotent. Alerting twice for the same stale run creates noise; suppressing every later alert hides a continuing outage. Use a deduplication key such as schedule/run_id/failure_kind, and open a new event when the run identifier changes or the failure kind changes.
How do you verify a missed import without guessing?
Verification starts with the event record, then moves toward the source of truth. Check the latest run identifier, completion timestamp, status, and result count. Compare the timestamp with the schedule's expected window. Next inspect the worker's structured error and the queue's delivery state. A scheduler log alone is insufficient because it records intent, not committed output.
Test the four important paths before deployment: a healthy run, a late run, a failed run, and a successful run with an invalid result. Add a duplicate-delivery test too. The expected behavior is concrete: one alert for one failure key, a recovery event after a valid new run, and no sensitive row data in the alert.
I initially treated the web endpoint as the simplest source of truth. That was the wrong boundary. The endpoint stayed healthy while a parser rejected an entire input, so the useful signal had to be emitted after validation and commit. That small correction also made ownership clearer: the import service owns result freshness, while the web service owns request availability.
For a concrete policy example, a nightly import may allow a 15-minute freshness window and require three consecutive empty-result exceptions before escalation. Those are policy inputs to test, not promises about every workload.
Rollback should be reversible and boring. If a new threshold pages too often, restore the previous threshold or disable only that schedule's rule while retaining the raw completion record. Do not delete the evidence to quiet the page. After the import path is repaired, run a controlled import, confirm a new run identifier and valid result count, then close the incident from the recovery event.
A practical decision rule
For a small team, start with one reachability check and one semantic result check per critical schedule. Add a separate latency or queue-depth signal only when it changes the response. Keep the checks labeled by owner and environment, and make the alert message answer three questions: what stopped, which run was expected, and who owns the next action.
This is the useful boundary for free uptime monitoring: use it to deliver a narrow, dependable signal, not to pretend that an HTTP status is an end-to-end proof. In a healthtech import pipeline, availability means that the expected data result is fresh, validated, and attributable.
References
- OpenTelemetry Logs Data Model: https://opentelemetry.io/docs/specs/otel/logs/data-model/
- Prometheus alerting practices: https://prometheus.io/docs/practices/alerting/
- HTTP Semantics, RFC 9110: https://www.rfc-editor.org/rfc/rfc9110.html
- Date and Time on the Internet, RFC 3339: https://www.rfc-editor.org/rfc/rfc3339.html
Top comments (0)