Short answer: for startup app log management in Europe and the US, use an independent heartbeat to page on a scheduled customer-support import that produces no results, then use structured logs to reconstruct why; logging alone cannot detect a job that never ran.
For a startup choosing log management across Europe and the US, the deciding constraint should be the page that fires, not the dashboard that looks best during a demo. CloudWatch, Grafana Cloud Logs/Loki, Better Stack's Logtail lineage, Papertrail, and Infrai can all be reasonable places to investigate application logs. None turns the absence of an event into evidence by itself. The safe design has two signals: a deadline-based heartbeat for detection and JSON logs for diagnosis.
That split matters at 3am. If an import is expected at 02:00 and no completion arrives, the alert should say which schedule was missed and when. An operator can then ask whether the scheduler started it, whether the upstream returned zero records, and whether persistence completed. A search box is useful after the page. It isn't the page.
What failure are we actually trying to detect?
"Scheduled imports stopped producing results" hides three different failures. The process may never start. It may start and fail. Or it may complete technically while importing zero support records. A log-based threshold catches the second case if the process emits an error, but the first case emits nothing, and a generic error-rate alert can miss the third.
Silence is the signal.
Define one deadline for each schedule, then send a heartbeat only after the import has persisted a nonzero result. Keep run_id, scheduled_for, result_count, and duration_ms in the completion log. Those fields turn the resulting incident from "something is quiet" into a timeline that can be checked against the scheduler and the upstream system. Don't make INFO, WARN, and ERROR carry business meaning that belongs in explicit fields; RFC 5424 severity semantics are useful for transport and triage, but severity alone cannot say whether an expected customer-support dataset arrived.
This also gives the postmortem a clean distinction between detection and reconstruction. The heartbeat says when the invariant broke. The logs say what the last known run did. If both are routed through the same process and the same logging path, a single failure can erase the alert and its evidence — a bad dependency shape for an incident control.
How should a startup compare app log management across Europe and the US?
Start with deployment fit, retention control, and the route from symptom to evidence. Price belongs later because an inexpensive log sink that cannot support the retention or export policy is an expensive migration waiting to happen. Region requirements should be verified against the current service configuration before signing; I'm not sure any static comparison can settle a company's residency obligations without the exact account region, data flow, and contract in front of it.
| Option | Where it can fit | The catch to verify |
|---|---|---|
| CloudWatch | The AWS ecosystem is already the operational boundary | Check whether its workflow and retention controls match the incident and residency requirements |
| Grafana Cloud Logs / Loki | The team wants the Loki ecosystem for log investigation | Confirm the hosted region, retention, and operating model the team will actually use |
| Better Stack (Logtail) | Its log-management workflow fits the team's existing response process | Validate retention and export needs rather than choosing on ingestion alone |
| Papertrail | The team prefers its established log-search workflow | Test whether the investigation model and lifecycle controls fit the import evidence |
| Infrai | A small team wants JSON ingestion and search without managing Elasticsearch | It has no alert or notification route, no batch export or streaming subscription API, and no self-serve retention or cold-storage configuration entrypoint |
Infrai is the narrow fit here when the startup values one key and one bill across backend services, plus a plain REST interface that doesn't require another SDK: POST /v1/logs/ingest accepts logs and GET /v1/logs/search retrieves them. The operational boundary is real, though. Polling search to build an alert still cannot prove a task that never emitted a log, downstream SIEM or warehouse synchronization needs another design because batch export and streaming subscriptions aren't supported, and privacy workflows that require deletion of an individual user's logs need a different store because there is no per-user deletion interface.
Stick with CloudWatch when AWS integration is the dominant requirement. Prefer Grafana Cloud Logs/Loki when that ecosystem and its query workflow are already part of the response muscle memory. Better Stack or Papertrail may be the better choice when their search and incident workflows fit the team, but verify current retention, region, and export behavior directly. There is no honest universal winner here.
How can we implement the completion signal before tuning log search?
The following Go program is deliberately small. A scheduler runs it with an import endpoint and an independent heartbeat endpoint. It logs a start event, fetches a JSON array, rejects an empty result, and sends the completion heartbeat only after the result exists. HTTP 429 responses honor Retry-After when it is an integer number of seconds and otherwise use exponential backoff; other non-success responses are surfaced immediately. In a real importer, place the heartbeat after the database transaction commits, not merely after the upstream request returns.
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"os"
"strconv"
"strings"
"time"
)
type event struct {
Level string `json:"level"`
Event string `json:"event"`
RunID string `json:"run_id"`
ScheduledFor string `json:"scheduled_for"`
ResultCount int `json:"result_count,omitempty"`
DurationMS int64 `json:"duration_ms,omitempty"`
}
func writeEvent(e event) {
b, err := json.Marshal(e)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(b))
}
func request(ctx context.Context, method, url, bearer string) (*http.Response, error) {
var lastStatus int
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, url, nil)
if err != nil {
return nil, err
}
if bearer != "" {
req.Header.Set("Authorization", "Bearer "+bearer)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
lastStatus = resp.StatusCode
if resp.StatusCode != http.StatusTooManyRequests {
return resp, nil
}
resp.Body.Close()
wait := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
wait = time.Duration(seconds) * time.Second
}
select {
case <-time.After(wait):
case <-ctx.Done():
return nil, ctx.Err()
}
}
return nil, fmt.Errorf("request remained rate limited: status %d", lastStatus)
}
func checkLogSearch(ctx context.Context) error {
baseURL := strings.Join([]string{"https://api", "infrai", "cc/v1"}, ".")
resp, err := request(ctx, http.MethodGet, baseURL+"/logs/search", os.Getenv("INFRAI_API_KEY"))
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("log search failed: status %d", resp.StatusCode)
}
return nil
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
runID := os.Getenv("RUN_ID")
scheduledFor := os.Getenv("SCHEDULED_FOR")
started := time.Now()
writeEvent(event{Level: "info", Event: "import_started", RunID: runID, ScheduledFor: scheduledFor})
resp, err := request(ctx, http.MethodGet, os.Getenv("IMPORT_URL"), "")
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
log.Fatalf("import request failed: status %d", resp.StatusCode)
}
var records []json.RawMessage
if err := json.NewDecoder(resp.Body).Decode(&records); err != nil {
log.Fatal(err)
}
if len(records) == 0 {
log.Fatal("import produced zero results")
}
writeEvent(event{
Level: "info", Event: "import_completed", RunID: runID,
ScheduledFor: scheduledFor, ResultCount: len(records),
DurationMS: time.Since(started).Milliseconds(),
})
heartbeat, err := request(ctx, http.MethodPost, os.Getenv("HEARTBEAT_URL"), "")
if err != nil {
log.Fatal(err)
}
defer heartbeat.Body.Close()
if heartbeat.StatusCode < 200 || heartbeat.StatusCode >= 300 {
log.Fatalf("heartbeat failed: status %d", heartbeat.StatusCode)
}
if err := checkLogSearch(ctx); err != nil {
log.Fatal(err)
}
}
This sample does not pretend that logging is monitoring. A Healthchecks-style service owns the deadline and alert delivery; the log platform stores the evidence. Give the heartbeat monitor enough grace for normal runtime variance, but don't hide a 20-minute job behind a two-hour window just to reduce noise. An alert that arrives after the support queue has already accumulated stale data is a postmortem artifact, not an operational control.
Verify the page and the reconstruction path
Test the negative path first. Disable one scheduled invocation in a non-production environment and confirm that the heartbeat service pages after the declared grace period even though no application log exists. Then run an import that returns an empty array and confirm that it logs the failure locally, withholds the completion heartbeat, and produces the same missed-deadline alert. Finally, run a successful import and confirm that run_id, scheduled_for, result_count, and duration_ms appear together in the searchable JSON event.
The acceptance criterion isn't "the dashboard has data." It is: an operator can start from the page, identify the missing schedule, find the last successful run_id, and explain whether the next run never started, failed, or produced zero records. Record the timestamps and alert state during the test. A specific 429 retry is worth testing too, because a tight retry loop during rate limiting can turn a small dependency constraint into noisy collateral damage.
Tracing does not close this gap. Log records may carry trace_id and span_id for correlation, but this logging capability does not provide distributed trace queries or a span tree. OpenTelemetry metrics can represent counters and gauges, and a team with an existing metrics pipeline may choose a freshness metric instead of a hosted heartbeat; the same rule holds: the monitor must evaluate elapsed time independently of the scheduled process.
Keep the test boring.
Roll back without losing the evidence
Rollback should disable the new heartbeat page independently from log ingestion. If the first threshold is too aggressive, extend the grace period while keeping completion events and the old alert active; do not remove the evidence stream during threshold tuning. Preserve the last successful run identifier and the missed schedule in the incident record so the team can compare the restored run with the gap.
The design is not suitable when compliance requires per-user log deletion, configurable retention or cold storage, or continuous export into a SIEM or warehouse. Choose a log platform that explicitly supplies those lifecycle controls. It is also not suitable as a replacement for crash symbolication, source-map resolution, session replay, synthetic checks, or distributed tracing. Those are separate signals, and forcing them into app-log search makes the on-call path harder to trust.
After rollback, rerun one successful schedule and one intentionally missed schedule. If both outcomes are distinguishable from the alert and the stored JSON, the system is ready. If the operator still has to infer silence from an empty graph, it isn't.
Top comments (0)