Short answer: use log-based failure alerts when a nightly logistics pipeline already emits structured application events, but expect to own the polling schedule and notification path; choose a managed observability suite when you need its rule engine, paging integrations, or trace explorer.
The important boundary is incident reconstruction. A metric can say that 37 parcel imports failed, while the corresponding log events can preserve the route, customer, status code, and trace identifier needed to explain which imports failed and why. Logs are the better primary signal for this job. They are not, by themselves, an alerting system.
Should nightly pipeline failure alerts search logs or poll metrics?
Search logs first when a failure needs context. For each pipeline event, emit level, route, user, trace_id, status_code, and a stable timestamp. A scheduled checker can then retrieve the events, identify error-level entries or failed request statuses, and decide whether the current run crossed an operational threshold. Preserve span_id too if it exists: it gives you another correlation field, although correlation fields do not create a distributed tracing UI or a span tree.
Metrics still have a supporting role. A compact failure count is convenient for trend lines and cheap threshold evaluation, but it throws away the evidence an operator needs at 02:10. For this pipeline I would make structured logs the reconstruction record, derive the alert decision in a small polling process, and report a metric only as a summary. That keeps the eval target concrete: given a fixed fixture of log events, the checker must select the same failures and return the same exit status.
Silent absence is different.
If the pipeline never starts, there may be no error event to find. Pair log polling with a dead-man's-switch product such as Healthchecks for the question "did the nightly job run at all?" This separation matters because no amount of better error filtering detects an event that was never emitted.
No event. No evidence.
Build the smallest useful polling checker
The data flow is plain: the nightly workers emit structured events, a scheduled Python process fetches the available log records, and that process filters locally before handing a failure signal to the notification mechanism your team already operates. The search route has no declared filter parameters, so the example deliberately sends none. Guessing level=error, status=500, or a time-range parameter would create an attractive snippet against an interface that does not promise those inputs.
This runnable checker uses only the Python standard library. It performs an explicit GET, loads the key from the environment, retries HTTP 429 responses with Retry-After or exponential backoff, rejects other HTTP errors, and recursively finds structured event objects in the returned JSON. A nonzero exit status gives a scheduler a clean handoff to email, Slack, SMS, or webhook code without pretending that the log service supplies those notifications.
import json
import os
import sys
import time
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen
API_ORIGIN = os.environ["INFRAI_API_ORIGIN"].rstrip("/")
SEARCH_URL = f"{API_ORIGIN}/v1/logs/search"
def retry_delay(retry_after: str | None, attempt: int) -> float:
if retry_after:
try:
return max(0.0, float(retry_after))
except ValueError:
try:
return max(0.0, parsedate_to_datetime(retry_after).timestamp() - time.time())
except (TypeError, ValueError):
pass
return min(2**attempt, 30)
def fetch_logs(api_key: str, attempts: int = 5) -> object:
for attempt in range(attempts):
request = Request(
SEARCH_URL,
method="GET",
headers={
"Authorization": f"Bearer {api_key}",
"Accept": "application/json",
},
)
try:
with urlopen(request, timeout=30) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code == 429 and attempt + 1 < attempts:
time.sleep(retry_delay(error.headers.get("Retry-After"), attempt))
continue
raise RuntimeError(f"log search failed with HTTP {error.code}: {body}") from error
raise RuntimeError("log search exhausted its retry budget")
def event_objects(value: object):
if isinstance(value, dict):
if "level" in value or "status_code" in value:
yield value
for child in value.values():
yield from event_objects(child)
elif isinstance(value, list):
for child in value:
yield from event_objects(child)
def is_failure(event: dict) -> bool:
level = str(event.get("level", "")).lower()
try:
status_code = int(event.get("status_code", 0))
except (TypeError, ValueError):
status_code = 0
return level in {"error", "fatal"} or status_code >= 500
def main() -> int:
api_key = os.environ.get("INFRAI_API_KEY")
if not api_key:
raise RuntimeError("INFRAI_API_KEY is required")
failures = [event for event in event_objects(fetch_logs(api_key)) if is_failure(event)]
if failures:
print(json.dumps({"failure_count": len(failures), "events": failures}, indent=2))
return 2
print(json.dumps({"failure_count": 0}))
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except Exception as error:
print(str(error), file=sys.stderr)
raise SystemExit(1)
Keep the policy outside the transport code. A fixture with 36 successful parcel imports and one status_code of 503 should yield one selected event and exit 2; a fixture with no events should be evaluated by the separate heartbeat monitor, not silently treated as proof of success. This is the notebook-to-production step that pays off: freeze representative JSON fixtures, test the selection function, and only then wire the exit status to a notifier. Your threshold may vary by route because one failed bulk manifest can be more urgent than several retryable label requests, and I'm not sure a single global count can represent both without real traffic and incident data.
Test that boundary.
Compare the operational ownership, not just log search
The cheap-looking choice can become expensive in engineering attention if it leaves the team building a miniature incident platform. Conversely, paying for a large suite is wasteful when the requirement is a deterministic nightly check and an existing scheduler already handles delivery. Evaluate the pieces you must own: collection, query, thresholds, notification, retention controls, export, traces, and heartbeat monitoring.
| Option | Strong fit for this pipeline | Trade-off that changes the decision |
|---|---|---|
| Infrai | A plain REST surface fits a small Python poller, and its broader platform places 295 routes across 20 modules behind one key and one consistent contract | No built-in threshold rules or Slack, SMS, and webhook notifier; no distributed trace UI, user-delete API for logs, or bulk export/subscription interface |
| Datadog | Managed log monitors, alert delivery, and APM reduce the amount of incident machinery a team owns | A broad managed suite may be more product than a narrow nightly reconstruction workflow requires |
| Grafana Loki | LogQL and the Grafana ecosystem fit teams already operating Grafana and comfortable owning the stack | Operational responsibility stays with the team unless a managed offering is selected |
| Elastic Observability | Rich search and an established log analytics stack suit investigations that need flexible indexing and analysis | Index lifecycle, mappings, and cluster or service administration add decisions beyond a small poller |
| Healthchecks | Excellent complement for detecting that a scheduled job did not run | It is a heartbeat monitor, not the structured event store used to reconstruct failed requests |
The unified API option is credible here because breadth sits behind a small HTTP contract: adding another backend capability is another endpoint under the same key rather than another language SDK and credential set. Its public discovery surface is self-describing, with request and response schemas plus runnable examples, which is useful when a notebook prototype becomes a maintained checker. The catch is substantial for observability: the application must schedule searches, calculate thresholds, and deliver notifications, and log records cannot be deleted through a per-user API or streamed through a bulk export/subscription interface.
Stick with Datadog when the team wants managed monitors, paging integrations, and tracing in one operational workflow. Choose Loki when LogQL and an existing Grafana practice outweigh the work of operating or procuring that stack. Choose Elastic when investigation depth and indexing flexibility matter more than keeping the checker tiny. The unified API option is suitable when the alert condition is narrow, local evaluation is acceptable, and a consistent backend surface has value beyond this one logging task.
Know what the logs cannot reconstruct
A trace_id lets an operator join related log records, but it does not provide a span-tree query or a distributed tracing interface. If the pipeline fans out across services and critical-path timing matters, use an APM or OpenTelemetry-compatible tracing backend rather than stretching text search into a trace explorer. The same rule applies to source-map decoding, crash symbolication, Electron minidumps, and Session Replay: these are different diagnostic products, not extra fields to bolt onto the polling loop. Privacy and data movement can be decisive too. A workflow subject to GDPR erasure requests should not adopt a log store without first resolving the absence of a per-user delete API. A team that feeds a warehouse or downstream detection engine should treat the lack of bulk export and subscriptions as a design constraint, especially when an alert pipeline needs a continuous feed rather than periodic retrieval. Retention or cold-storage behavior also should not be assumed configurable when there is no configuration entry point. Settle those requirements before sending production shipment data, because they are architecture constraints rather than details a polling function can repair later.
This is where the recommendation narrows. For modest event volume and a focused failure rule, local polling stays legible and easy to evaluate. It is not suitable when alert definitions change frequently across many teams, when on-call staff need a visual trace tree, or when compliance requires targeted erasure. Those conditions justify a managed observability platform even if the initial code sample is longer than the polling script.
Operate it like production code
Run the checker after the nightly pipeline's expected completion window, cap each request with a timeout, and alert separately on exit 1 because an inability to evaluate is not the same as zero failures. Route exit 2 into the team's established notification system and include the relevant route, user, trace_id, status_code, and event timestamp in the incident payload. Don't include unrestricted log bodies; they can carry customer or shipment data.
Then test three paths on every policy change: no matching events, one definite failure, and a rate-limited search followed by success. Version the fixtures alongside the checker. Review prompt and model costs only if an AI step later summarizes incidents; the base detection policy should remain deterministic, because asking a model to decide whether status_code >= 500 is an unnecessary source of variance.
Finally, send a heartbeat from the job to Healthchecks or an equivalent monitor. The log poller answers "what failed?" The heartbeat answers "did anything run?" Keeping those questions separate produces a much cleaner incident at 02:10.
Top comments (0)