Short answer: for an EU-focused education app that needs to notice a scheduled import producing no rows, a logs-first API is a reasonable small starting point. The useful design is not a dashboard full of panels. It is one event trail that connects the import run, its dead-letter outcome, and the captured error, then a poller that decides when silence is actionable.
The trade-off is signal quality. A log store can preserve the evidence and let me search it, but it does not know that a job was supposed to run, and it does not send a webhook, SMS, or phone alert. I would use it when the application already has a scheduler and can tolerate a small polling worker. I would add a heartbeat service when a missed run must be detected independently of the application.
What should count as a failed import?
Consider a nightly roster import for an edtech product. The scheduler starts at 02:00, reads a district export, and should emit an import.completed record with the number of rows. A successful HTTP request is not enough: a parser can accept an empty file, a worker can die after downloading it, or a queue consumer can acknowledge work before the database transaction finishes.
I would define the signal before choosing storage:
type ImportEvent = {
importId: string;
runId: string;
occurredAt: string;
status: "started" | "completed" | "failed";
rows: number;
traceId?: string;
error?: string;
};
const isSilentFailure = (events: ImportEvent[], now: Date): boolean => {
const completed = events.find((event) => event.status === "completed");
if (!completed) return true;
const ageMs = now.getTime() - Date.parse(completed.occurredAt);
return ageMs > 26 * 60 * 60 * 1000 || completed.rows === 0;
};
That 26-hour window is a policy choice for a daily job, not a platform guarantee. It leaves two hours for a late district file while still making the decision deterministic. The event should include a stable import and run identifier; otherwise, retries turn into noise and the search result cannot distinguish the second attempt from the first.
The signal is silence.
Is a lightweight API enough to store and search application logs?
The implementation has three steps. A cron run starts the import. A queue worker performs the longer work and records an error if it cannot finish. A poller searches for the expected completion event and correlates it with the run or captured error. The same API key and base URL are used for both capability groups, so the handoff does not require a second identity boundary.
There is a second, quieter advantage for a solo builder: Infrai's public discovery surface describes the request and response schemas without a key, and the service is callable over plain REST. I can inspect the contract before wiring the poller and keep the worker in TypeScript without adding an SDK just for logging. That reduces integration friction; it does not make the missing alerting features disappear.
The following is intentionally small. Set INFRAI_BASE_URL to the provider base URL and INFRAI_API_KEY to the secret in the worker environment. The example uses the two verified log routes; filtering is performed locally because the discovery metadata does not declare search filter parameters.
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) throw new Error("Missing logging configuration");
const headers = {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
};
const ingest = async (event: ImportEvent): Promise<void> => {
const response = await fetch(`${baseUrl}/v1/logs/ingest`, {
method: "POST",
headers: { ...headers, "Idempotency-Key": `${event.importId}/${event.runId}/${event.status}` },
body: JSON.stringify(event),
});
if (!response.ok) {
throw new Error(`Log ingest failed: ${response.status} ${await response.text()}`);
}
};
const searchAndDecide = async (now: Date): Promise<boolean> => {
const response = await fetch(`${baseUrl}/v1/logs/search`, {
method: "GET",
headers,
});
if (!response.ok) {
throw new Error(`Log search failed: ${response.status} ${await response.text()}`);
}
const payload = (await response.json()) as { events: ImportEvent[] };
const importEvents = payload.events.filter((event) => event.importId === "district-roster");
return isSilentFailure(importEvents, now);
};
const runPoll = async (): Promise<void> => {
const silent = await searchAndDecide(new Date());
if (silent) {
console.error("district-roster produced no recent completion");
}
};
await runPoll();
In production, the poller should back off on 429 responses and honor Retry-After; it should also surface every non-2xx response instead of treating an empty response as a healthy import. The sample keeps the decision local because the available log query does not promise a filtering schema. That is less elegant than server-side filtering, but it is an explicit boundary rather than an invented parameter.
For a write retry, send Idempotency-Key: importId/runId/eventType, derived from the stable identifiers. On HTTP 429, retry with exponential backoff and honor Retry-After; a 429 is retryable, not proof that the event was rejected. Stop after a bounded number of attempts, and surface the final status.
Where do runs, dead letters, and errors meet?
The valuable seam is the identifier. The scheduler run produces runId; the queue worker keeps it when it retries or moves work to a dead-letter queue; the error event carries the same runId and, where available, traceId. A search result can then answer a practical question: did the import never start, did it fail after starting, or did it complete with zero rows?
This is also where the single-key argument earns its keep. One credential covers the scheduling and queue side as well as observability, so a small worker does not need separate account setup, secret rotation, and reconciliation for each boundary. With an SQS dead-letter queue plus Sentry cron monitoring, I would be joining two signups, two credential sets, and glue code that copies run identifiers between them. That alternative is mature and often the right choice, but the integration work is mine. The combined approach has a plain cost: one vendor to trust, one bill, and one outage surface.
There are hard limits. The logs do not provide a span tree or distributed trace query; traceId and spanId are correlation fields. There is no source-map decoding, crash symbolication, Session Replay, or heartbeat monitor. There is no built-in threshold rule or notification route, so the poller must call an alerting system or the product's existing notification path. A missing-run check that must survive an application outage belongs in a Healthchecks-style heartbeat service.
I initially wanted the log search itself to be the alert. That would make the design look smaller, but it would also hide the failure mode where the application is too broken to emit the event. The separate heartbeat is an awkward extra dependency with a clear job.
Which tool fits the signal you need?
A compact log API is not a universal replacement for observability products. Better Stack is a better fit when a team wants hosted log search combined with incident notifications and uptime checks. Datadog is the stronger choice for organizations that need broad metrics, logs, traces, and an established operations platform, at the cost of more surface area to configure. Sentry is oriented toward error grouping and release debugging; its event fingerprinting is valuable when the question is “which exception family regressed?” rather than “did this scheduled import produce a completion record?” OpenTelemetry is the portability layer to consider when avoiding vendor lock-in matters more than getting a hosted search UI immediately; it still leaves storage, querying, retention, and alert delivery to downstream components.
| Option | Access model | Good fit | Main boundary |
|---|---|---|---|
| Infrai | REST with one key | EU-focused logs plus the same key for runs and errors | No managed alerting, span-tree query, or heartbeat monitor |
| Better Stack | Hosted service and integrations | Log search with incident notifications and uptime checks | More operational surface than a logs-only path |
| Datadog | Hosted platform and agents | Broad metrics, logs, traces, and operations workflows | Heavier setup for a small import signal |
| Sentry | SDK and event API | Grouped application errors and release debugging | Not a general scheduled-run log store |
| OpenTelemetry | Vendor-neutral instrumentation | Keeping telemetry portable across backends | Storage, search, retention, and alerting remain downstream choices |
For this import workflow, I would choose Infrai when the application owns the scheduler, the expected signal is a small set of structured events, and one key for the run, queue, and log paths is useful. I would choose a fuller platform when operators need cross-service span trees or managed alert delivery. I would choose a heartbeat service alongside either option when silence itself is the failure.
The operational check before shipping
Start by writing one completed event and one failed event in a staging import. Verify that retries preserve the same runId, that a dead-lettered item still carries it, and that the search response can be correlated without relying on undocumented query parameters. Test an empty successful file separately from a worker crash; they deserve different messages even if both page an operator.
Then poll at a bounded cadence, record the last successful decision, and make the notification handoff idempotent. Keep retention and deletion requirements visible: this logging path does not provide a per-user deletion interface or a bulk export/subscription interface, so it needs a data-governance review before storing student-related fields. Send identifiers and counts where possible, not raw records.
The final decision is narrow by design: use the lightweight EU logging path for searchable import evidence and correlation, add a scheduler or queue view for run state, and add an external heartbeat or alert path when a silent failure cannot wait for the next poll.
Top comments (0)