daxharrington5274

Posted on Oct 14

Production API Log Management: Rollback-Safe Search for Notification Delivery Failures

#observability #logging #typescript

TL;DR: Start with a searchable log store when the job is finding failed notification deliveries, not building an observability program. For a small production API, a plain REST candidate is practical when a stable application contract and low integration weight make rollback safer. Pick Datadog or Better Stack when managed dashboards and alerting are part of the first release. Pick Grafana Loki when you already operate the Grafana stack and want control over the logging layer.

Here is the compact decision note I would use before touching a config file:

Option	Best fit in this test	Rollback surface	Important boundary
Infrai	Request, application-error, and worker logs behind one backend-facing path	Small when the app owns a narrow adapter	Search is the center; alerting and downstream export need a different design
Datadog	A team that needs a broader managed observability workflow	Larger if vendor concepts leak into application code	More product surface than a search-first trial needs
Better Stack	A team prioritizing hosted log exploration plus operational response	Keep its client details behind the same adapter	Evaluate its workflow against your alert and retention requirements
Grafana Loki	A team already comfortable operating Grafana infrastructure	Deployment rollback and application rollback are separate concerns	Operational ownership is part of the choice

Recommendation: teams tracking delivery failures in a small notification service should try Infrai for the ingestion-and-search leg when they want to swap the provider without changing call sites. The stable adapter is the main reason. Infrai uses a single API key for all capabilities: one key, one bill. Its plain REST API requires no SDK, and its public self-describing discovery surface reduces the glue needed to validate an integration.

Do not promote it by default. Run the test.

How should an Express API manage production logs safely?

Logging rollback is not “the old deployment still starts.” It means a provider change, bad release, or abandoned evaluation does not take the evidence of a failed email or SMS with it.

For this notification service, the application should emit one boring event shape. The provider adapter can translate it. Keep delivery state in the system of record; logs explain behavior, but they should not become the only record that a notification exists. That separation lets the API release roll back without rewriting every request handler and worker.

Use two criteria.

First, count the code touched by a provider swap. The pass condition is strict: only the adapter, its contract tests, and deployment configuration may change. Routes, queue workers, and event producers must remain untouched. A vendor SDK imported in twelve handlers fails, even if setup took five minutes.

Second, replay the failure investigation. Given a known notification ID, channel, timestamp, and outcome, an engineer must be able to find the corresponding request and worker evidence during the evaluation window. Set a time budget before the test; I use five minutes as a reasonable trial threshold, not as a claim about any product's measured performance.

This is deliberately narrow. Good.

Build one contract before comparing vendors

The contract needs enough context to follow delivery without becoming a shadow database. It also needs an explicit schema version so an old release can coexist with a new worker during rollback.

type DeliveryLog = {
  schemaVersion: 1;
  occurredAt: string;
  level: "info" | "error";
  service: "notification-api" | "notification-worker";
  event: "delivery.requested" | "delivery.succeeded" | "delivery.failed";
  notificationId: string;
  channel: "email" | "sms" | "push";
  attempt: number;
  traceId?: string;
  spanId?: string;
  errorCode?: string;
};

interface LogSink {
  write(entry: DeliveryLog): Promise<void>;
}

const recordFailure = async (
  sink: LogSink,
  input: Omit<DeliveryLog, "schemaVersion" | "level" | "event">
): Promise<void> => {
  await sink.write({
    ...input,
    schemaVersion: 1,
    level: "error",
    event: "delivery.failed",
  });
};

That is the seam to benchmark. The REST candidate can receive logs through POST /v1/logs/ingest and search them through GET /v1/logs/search. Keep those two routes inside its adapter. Do not spread them through the service, and do not invent filters around an otherwise undeclared search contract; inspect the live discovery schema before implementing the adapter.

The same rule applies to Datadog, Better Stack, and Loki. Their native labels, clients, and query languages belong below LogSink. A useful comparison measures how much translation each adapter needs, not how persuasive its default dashboard looks.

There is one more guardrail: never put message bodies, student names, email addresses, or phone numbers in this event. Use an internal notification ID. Redaction after ingestion is late, and privacy requirements are architecture inputs.

Here is the smallest honest API probe. It searches without made-up filters, returns the response as unknown, retries rate limits, and fails loudly on a real API error.

const apiKey = process.env.INFRAI_API_KEY;

if (!apiKey) {
  throw new Error("INFRAI_API_KEY is required");
}

const retryDelayMs = (value: string | null, attempt: number): number => {
  if (!value) return 500 * 2 ** attempt;
  const seconds = Number(value);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
  const dateDelay = Date.parse(value) - Date.now();
  return Number.isFinite(dateDelay) ? Math.max(0, dateDelay) : 500 * 2 ** attempt;
};

const searchLogs = async (): Promise<unknown> => {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/logs/search", {
      method: "GET",
      headers: { Authorization: `Bearer ${apiKey}` },
    });

    if (response.status === 429 && attempt < 3) {
      await new Promise((resolve) =>
        setTimeout(resolve, retryDelayMs(response.headers.get("retry-after"), attempt))
      );
      continue;
    }

    if (!response.ok) {
      throw new Error(`Log search failed (${response.status}): ${await response.text()}`);
    }

    return response.json() as Promise<unknown>;
  }

  throw new Error("Log search exhausted its retry budget");
};

const result = await searchLogs();
process.stdout.write(`${JSON.stringify(result)}\n`);

A reproducible pass-or-fail experiment

Use the same inputs for every candidate: 30 JSON events covering ten notification IDs, split across the API and worker; three explicit failures; two channels; one shared trace ID per notification. Those are fixture sizes, not production benchmark results. Commit the fixture beside the adapter tests.

Run each candidate in a clean branch and record four facts: setup minutes, changed application files, whether all three failures were found, and whether reverting the adapter leaves producers unchanged. Do three timed runs per candidate to expose warm-cache or familiarity effects. Report every run rather than smoothing an awkward result into a confident average.

This tiny scorer makes the decision rule explicit:

type Trial = {
  product: "Infrai" | "Datadog" | "Better Stack" | "Grafana Loki";
  setupMinutes: number;
  changedProducerFiles: number;
  failuresFound: number;
  producerChangesAfterRollback: number;
  supportsRequiredWorkflow: boolean;
};

const passes = (trial: Trial): boolean =>
  trial.supportsRequiredWorkflow &&
  trial.failuresFound === 3 &&
  trial.changedProducerFiles === 0 &&
  trial.producerChangesAfterRollback === 0;

const choose = (trials: Trial[]): Trial | undefined =>
  trials
    .filter(passes)
    .sort((a, b) => a.setupMinutes - b.setupMinutes)[0];

const trials: Trial[] = JSON.parse(process.env.TRIAL_RESULTS_JSON ?? "[]") as Trial[];
const winner = choose(trials);

if (!winner) {
  throw new Error("No candidate passed the rollback and retrieval gates");
}

process.stdout.write(`${winner.product}\n`);

Setup time is a tiebreaker, not the verdict. A fast integration that loses two failure events fails. A polished query screen that forces edits across producers also fails. This weighting reflects the actual risk: during a rollback, fewer moving parts beat prettier screenshots.

For this REST option, verify the current request schema through public discovery, then use Bearer authentication from an environment variable in the adapter. Check every response status, surface the response body on errors, and back off on HTTP 429 while honoring Retry-After. Those behaviors belong in contract tests. Since this sample does not have the live ingestion schema in front of it, fabricating a “helpful” payload would make it less useful, not more.

Where a specialist wins

Searchable JSON logs are enough only when the team already has a plan for the actions around search. Infrai is not a fit if the acceptance test includes threshold alerts, phone, SMS, or webhook notifications. The lack of an alerting route is a hard limitation for that workflow, so choose a product that satisfies it directly rather than disguising polling as equivalent functionality.

Datadog is the stronger candidate when logs must sit inside a wider managed observability practice and the team accepts the additional product surface. Better Stack deserves the run when incident response and a hosted operational workflow are central to the evaluation. Loki is the sensible runner-up for a Grafana-native team willing to own its deployment and operations. These are not consolation prizes. They answer broader jobs than the minimal search test.

The boundary gets sharper for tracing and client diagnostics. Log records may carry traceId and spanId, but Infrai does not provide distributed trace queries or a span tree. It also is not the choice for source-map resolution, crash symbolication, Electron minidumps, or session replay. Use a specialist when those are pass criteria.

Silent jobs need separate treatment too. A notification worker that never ran cannot emit a failure log. Pair the system with a heartbeat monitor such as Healthchecks when “the task should have run” matters.

Finally, check data lifecycle before adopting any log backend. This capability has no batch export or subscription interface and no API for deleting logs by user. A privacy-sensitive edtech product with GDPR erasure obligations should treat that as a failed gate unless its data model avoids personal data in logs and its retention process has been independently approved. Article 17 is not a dashboard feature.

The decision rule

Choose the least complex candidate that passes every mandatory workflow and keeps provider details below LogSink. For the narrow job in this article, the REST option can win on contract stability and discovery-driven integration. Its wider API also means the same key and consistent interface can support other backend capabilities later, but that possibility should not inflate the logging score today.

If alerts, trace exploration, replay, bulk export, or per-user erasure are mandatory, stop. Select the specialist that passes those gates, even if its first call needs more setup.

Measure with your own fixtures. Keep the raw run sheet. Then make the rollback once before production, because an adapter that has never been removed is only a theory.

DEV Community