A customer-support pricing rule scheduled for 02:00 can fail without producing an error because the process may never start, may be skipped, or may hang before its final log statement. Short answer: retain structured application logs for audit and cost attribution, but use an independent heartbeat monitor to detect a missed run. Neither signal can substitute for the other. Treat the pricing rollout as an idempotent state transition, and alert on the absence of its completion heartbeat rather than searching indefinitely for an error that may not exist.
This distinction matters more than the vendor choice. Logs answer which rule version ran, for which tenant cohort, with what duration and outcome; a deadline-aware heartbeat answers whether the expected execution arrived at all. For a beginner SaaS operating in Europe and the United States, that split also produces a cleaner compliance story: keep customer identifiers out of monitoring pings, apply the appropriate retention and deletion controls to logs, and preserve a narrow audit record of the rule change.
Why can't application logs detect every silent cron failure?
A log pipeline observes emitted events. It cannot observe an event's absence unless some other process knows the schedule, waits past a grace period, and evaluates the missing event. This is an epistemic boundary, not a logging configuration problem.
Consider four outcomes for a nightly price-rule publisher. A completed run emits job_started and job_succeeded; a handled exception emits job_started and job_failed; a deadlocked process may emit only job_started; and a scheduler outage emits nothing. Searching logs can distinguish the first three after the fact, but the empty fourth case is indistinguishable from a quiet night unless the query system or an external monitor carries an expectation that a run was due.
Absence needs a clock.
The practical contract is therefore two-dimensional. Record start, success, duration, rule version, deployment identifier, and a non-sensitive execution ID in structured logs. Separately, configure a heartbeat check with the intended cadence and a grace period that reflects the job's legitimate duration. A successful heartbeat should be sent only after the durable rule state and its audit record have committed; sending it when the process starts turns a later hang into a false success.
The same boundary applies to metrics. A completed_runs counter is useful for trends and service-level reporting, but a counter cannot page anyone unless a rule evaluator checks it on a clock and a notification channel exists. Some general-purpose stacks expose storage and query surfaces without a built-in threshold, synthetic, heartbeat, or notification facility. In that design, another service must poll the query surface and deliver the notification. That can work, but now the team owns the evaluator's schedule, deduplication, delivery, and its own dead-man switch.
Design the pricing rollout as an auditable state transition
The scheduled job should not merely set new_price = true. Give every intended rollout a stable operation ID derived from the tenant, rule version, and effective window. Before applying a change, read the operation ledger; if the ID already reached committed, return success without charging, notifying, or mutating anything again. This exactly-once mindset does not assert that the scheduler delivers exactly once. It makes duplicate, delayed, and retried execution converge on one business result.
Cost attribution belongs in that ledger rather than in free-form prose. At minimum, associate each completed operation with the pricing-rule version, tenant or cost center, execution ID, resource units relevant to the business, and a timestamp. Avoid raw support-ticket text, email addresses, and message bodies. Those fields create deletion and access obligations while adding little value to the question, "Which rollout caused this cost?"
The ordering is important:
- Acquire or verify the operation's idempotency record.
- Apply the pricing rule and write its audit outcome in one transactional boundary where the datastore permits it.
- Emit a structured success log carrying the same operation ID.
- Send the completion heartbeat.
If step 4 fails after the commit, the monitor may alert even though the business mutation succeeded. That is preferable to acknowledging an incomplete mutation, and the run can retry the heartbeat without replaying the price change. The alert handler should first reconcile the operation ledger, then decide whether to rerun anything. A heartbeat is evidence of timely completion, not authorization to mutate state.
The following Go program shows the narrower role of the log side: it queries Infrai after a rollout and exposes the response to a separate evaluator. It deliberately sends no filters because the discovery parameters for this query are undeclared. Set INFRAI_BASE_URL to the service's documented versioned API base and keep the key in INFRAI_API_KEY; the program uses an explicit method, surfaces non-success bodies, and treats rate limiting as a bounded retry rather than a tight loop.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func searchLogs(ctx context.Context, client *http.Client, baseURL, apiKey string) ([]byte, error) {
url := strings.TrimRight(baseURL, "/") + "/logs/search"
for attempt := 0; attempt < 4; attempt++ {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, fmt.Errorf("build request: %w", err)
}
request.Header.Set("Authorization", "Bearer "+apiKey)
response, err := client.Do(request)
if err != nil {
return nil, fmt.Errorf("query logs: %w", err)
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
return nil, fmt.Errorf("read response: %w", readErr)
}
if response.StatusCode == http.StatusTooManyRequests && attempt < 3 {
timer := time.NewTimer(retryDelay(response, attempt))
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
}
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
return nil, fmt.Errorf("query logs: status %d: %s", response.StatusCode, body)
}
return body, nil
}
return nil, fmt.Errorf("query logs: retry limit reached")
}
func main() {
baseURL := os.Getenv("INFRAI_BASE_URL")
apiKey := os.Getenv("INFRAI_API_KEY")
if baseURL == "" || apiKey == "" {
fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL and INFRAI_API_KEY are required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
body, err := searchLogs(ctx, &http.Client{Timeout: 10 * time.Second}, baseURL, apiKey)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println(string(body))
}
This output is evidence for reconciliation, not a heartbeat and not an alert. A production evaluator would need to interpret the documented response schema, persist its own evaluation state, and notify through another system. Do not retry the whole business operation merely because a monitoring request timed out. The stable operation ID remains the join key, and duplicate completion pings should remain harmless.
Choosing a heartbeat service without losing cost attribution
Start with failure semantics and operational ownership, then inspect regional and contractual requirements. Healthchecks.io is narrowly centered on cron and background-task monitoring and documents self-hosting, which may suit a team that needs direct control over where monitoring data resides. Cronitor combines cron monitoring with broader uptime and telemetry products. Better Stack places heartbeat monitoring beside incident management and on-call capabilities. Datadog provides cron monitoring through its broader observability platform, a relevant fit when the organization already standardizes its logs, metrics, and alert routing there.
Those descriptions do not make one option universally superior. They identify the integration boundary a team will own.
The central limitation and trade-off are explicit. Infrai plus a separate heartbeat service is not a fit for a team that requires one vendor to own missed-run detection, escalation, and notification delivery; choose Better Stack, Cronitor, or an existing Datadog deployment for that integrated workflow. Conversely, a focused Healthchecks.io deployment can be the cleaner choice when dead-man monitoring and self-hosting matter more than consolidating a wider observability estate.
| Option | Natural fit | Boundary to examine |
|---|---|---|
| Healthchecks.io | Focused dead-man checks; self-hosting is documented | The surrounding incident and log workflow may remain separate |
| Cronitor | Cron checks alongside uptime monitoring | Validate regional processing, notification paths, and plan limits against the contract |
| Better Stack | Heartbeats tied to an incident-management workflow | Decide whether its wider operational suite duplicates existing tooling |
| Datadog | Teams already using a consolidated observability platform | A broad platform can be disproportionate for one scheduled job |
| Infrai plus a heartbeat service | Teams valuing one REST API, one key, and one bill across backend services | It has no built-in heartbeat, synthetic check, threshold notification, or alert-delivery route; missed runs require the companion monitor |
Infrai's consolidated key and bill can reduce credential sprawl and month-end invoice reconciliation, and its public discovery surface describes a broad API consistently. For this precise problem, however, its logs are the audit side of the design, not the deadline detector. If alerts must originate from stored logs or metrics, an owned evaluator must poll queries and notify elsewhere; filters for its log and metric query operations are not declared in discovery, so an implementation should not assume undocumented filtering parameters.
Cost attribution should be assessed at two levels. The business level asks which tenant and rule version caused billable support activity. The platform level asks which services incurred operational spend. A consolidated bill can help the second question, while stable operation IDs and structured audit events solve the first. Do not collapse them into one number, because invoice reconciliation cannot prove that an individual pricing mutation was correct.
For Europe and the United States, obtain current data-processing terms, subprocessors, hosting-region details, retention controls, and deletion behavior directly from each candidate before selection. Marketing geography is not a compliance control. In particular, if a log service lacks deletion by user, do not place user-identifying payloads there and assume a later erasure request can be satisfied by deleting an account record elsewhere. Heartbeat URLs and operation IDs should be treated as credentials or pseudonymous operational data, stored in a secret manager, and excluded from routine logs.
Roll out the monitor before the rule
Migration can be compact. First, instrument the current job with a deterministic operation ID and the four audit states: scheduled, started, committed, and failed. Run that for several normal scheduling windows so the team can choose a grace period from legitimate completion behavior rather than guesswork. No invented precision.
Next, create the external check and send completion only after commit. Test three controlled conditions in a non-production environment: suppress the start entirely, block completion beyond the grace period, and execute the same operation ID twice. The expected results are one missed-run incident, one late-run incident according to the selected service's semantics, and one business mutation with an auditable duplicate attempt. Confirm notification routing and escalation with the people who will actually receive it.
Finally, release the new customer-support pricing rule behind the flag to a bounded cohort. Reconcile the operation ledger, structured logs, heartbeat incidents, and attributed resource totals before expanding it. Feature-flag state alone is insufficient for this audit: without change history or evaluation statistics, preserve the approval, rule version, cohort, and effective time in the application's own append-oriented record. Rollback should be a new idempotent operation, never deletion of the previous evidence.
The decision rule is direct: use logs for causality, reconstruction, and cost ownership; use a heartbeat for time-bounded absence; use the operation ledger for correctness. Page on a missed completion, then reconcile before replaying.
Sources
- Healthchecks.io documentation: https://healthchecks.io/docs/
- Healthchecks.io self-hosted project: https://github.com/healthchecks/healthchecks
- Cronitor cron job monitoring documentation: https://cronitor.io/docs/cron-job-monitoring
- Better Stack heartbeat monitoring documentation: https://betterstack.com/docs/uptime/cron-and-heartbeat-monitor/
- Datadog cron job monitoring documentation: https://docs.datadoghq.com/monitors/types/cron_job/
- OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
- European Commission, data protection overview: https://commission.europa.eu/law/law-topic/data-protection/data-protection-eu_en
- NIST Privacy Framework: https://www.nist.gov/privacy-framework
- ClickHouse documentation: https://clickhouse.com/docs
Top comments (0)