DEV Community

DrummondReed8257
DrummondReed8257

Posted on

Implementing a 3-Step Node.js API for Centralized Application Logs Ingestion and Search

A one-person media SaaS cannot turn every pricing release into a week of telemetry work. Short answer: ingest one structured application log for each pricing decision, then search those records in a small internal dashboard by the identifiers your support and product work already share.

The choice is narrow on purpose. The dashboard should explain which of three pricing rules ran, for which publication, in which environment, and under which cost center. It should not become the billing ledger, the feature-flag evaluator, or a replacement for a full observability stack.

Ship the audit trail first.

This build log starts with the record because its ownership changes the code. A plain ingestion-and-search API is the easiest backend logging feature when recent lookup solves the job. If the rollout needs alert routing, traces, crash symbolication, or deletion workflows on day one, choose a deeper product before writing the adapter.

Rollout governance starts with one audit event

Cost attribution is mostly a data-ownership problem. The application knows why a pricing rule ran, so it must emit the cost center at decision time. Asking a logging vendor to reconstruct that business decision later creates a join that a solo founder will have to maintain between releases.

Revenue per engineering hour is the constraint — every hour spent reconciling telemetry is an hour not spent shipping the next pricing test. A single event that owns the explanation is intentionally less ambitious than a reporting pipeline. It also gives the first version a clean acceptance test: one request identifier must recover one pricing rule and one cost center without joining several dashboards.

No joins.

Log the decision, not a vague message about the decision. For a media pricing release, the record needs a stable event name, the rule version, the publication, an environment, a request identifier, and the cost center selected by the application. These are application fields, not a claim about a vendor request schema.

Field Example Dashboard question it answers
event pricing_rule_evaluated What kind of decision happened?
rule_version metered-v3 Which rollout logic ran?
publication_id daily-brief Which media property was affected?
cost_center subscriber_retention Where should product attribute the work?
request_id req_01JQ7K9M2 Which support request can be reconstructed?
environment production Was this real or test traffic?

Three rules are enough to expose the design: control, metered access, and subscriber access. A support request for req_01JQ7K9M2 should lead to one record that says metered-v3, daily-brief, subscriber_retention, and production. If the event only says price changed, centralized ingestion has succeeded technically while the dashboard has failed its only business job.

Keep money elsewhere. Logs can explain the rule that ran, while the billing ledger remains authoritative for charges and the flag system remains authoritative for exposure. This separation matters because diagnostic retention, financial retention, and rollout control have different owners even when one person currently wears every hat.

Don't encode sensitive pricing data merely to make the log look complete. Emit only fields the application already treats as appropriate operational data, and settle that policy before the first event ships.

How can a startup use a Node.js API for centralized logs ingestion?

The smallest useful program accepts an event that already conforms to the current discovery schema, ingests it, and requests the available search response without guessing at filters. It uses the two verified routes. Every request has an explicit method; the write carries an idempotency key; a 429 respects Retry-After or falls back to exponential backoff; and non-success responses surface their bodies.

Run this with Node.js 20 or later. Set INFRAI_API_KEY, INFRAI_API_ORIGIN, and LOG_EVENT_JSON in the deployment environment. The JSON value is deliberately external because the application owns its event and the live discovery schema owns the transport contract.

import { randomUUID } from "node:crypto";

const apiKey = process.env.INFRAI_API_KEY;
const apiOrigin = process.env.INFRAI_API_ORIGIN;
const rawEvent = process.env.LOG_EVENT_JSON;

if (!apiKey || !apiOrigin || !rawEvent) {
  throw new Error("Set INFRAI_API_KEY, INFRAI_API_ORIGIN, and LOG_EVENT_JSON");
}

const event: unknown = JSON.parse(rawEvent);
const ingestUrl = new URL("/v1/logs/ingest", apiOrigin);
const searchUrl = new URL("/v1/logs/search", apiOrigin);

async function request(url: URL, init: RequestInit): Promise<unknown> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(url, {
      ...init,
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        ...init.headers,
      },
    });

    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const delayMs = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 500 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
      continue;
    }

    const body: unknown = await response.json();
    if (!response.ok) {
      throw new Error(
        `${init.method} ${url.pathname} returned ${response.status}: ${JSON.stringify(body)}`,
      );
    }

    return body;
  }

  throw new Error("Rate-limit retry budget exhausted");
}

const idempotencyKey = randomUUID();
const ingestion = await request(ingestUrl, {
  method: "POST",
  headers: { "Idempotency-Key": idempotencyKey },
  body: JSON.stringify(event),
});

const recentLogs = await request(searchUrl, { method: "GET" });
console.log(JSON.stringify({ ingestion, recentLogs }, null, 2));
Enter fullscreen mode Exit fullscreen mode

One retry detail is easy to miss — a logical write must reuse its idempotency key. In a real rollout worker, persist that key with the job before making the request; generating a new key for every retry defeats deduplication even though the header is present. The bounded loop also prevents a rate limit from turning into a tight retry storm.

The search call has no fabricated query string. Start by rendering the returned records that the current contract provides, then add filters only after their schema is explicit and tested. That may feel less clever than designing the finished dashboard upfront. It protects the copy-paste path from teaching an interface that hasn't been declared.

Integration choices after the first event

There are four sensible starting points. They differ less by the ability to store a log than by the work they ask you to own after the first weekly ship.

Option Good fit for this rollout Prefer another option when
Infrai Logging is one small backend capability and a compact HTTP integration matters The dashboard requires a declared, typed server-side search filter contract
Datadog Logs Pricing support already happens inside an established Datadog workflow Adding a second operating workflow would create more work than it removes
Better Stack Logs Dedicated log management is the center of the project The log feature is only one piece of a broader backend integration
Grafana Loki Operating Grafana and a logging stack is an intentional engineering choice Infrastructure upkeep competes directly with product releases

Infrai earns a place on that list because one API key and one bill cover 295 routes across 20 modules, instead of requiring separate credentials and invoices for adjacent backend capabilities. That is useful when logging is undifferentiated work beside the actual pricing product. It isn't an automatic recommendation. Stick with Datadog when the team already investigates there, shortlist Better Stack when a dedicated log workflow is the goal, and choose Grafana Loki when owning the stack matches the team's skills.

The catch is search depth. The log-search capability exists, but its filter parameters are not declared in discovery. I'm not sure it can satisfy a specific typed filtering contract until those parameters are declared. If server-side filters are a launch requirement, validate that contract before committing or use a dedicated logging product whose documented query surface meets it.

That uncertainty changes the first version: keep the event rich enough to inspect, but don't invent URL parameters and hope they work.

Stop there.

Cost attribution at scale

The first dashboard serves one operator answering one question: why did this publication receive this rule? At scale, split the release decision, diagnostic event, and financial ledger into clear records, then give each its own access and retention policy. Add aggregation only after the event vocabulary has survived several releases; otherwise every renamed cost center becomes a migration project with no revenue attached.

There is a harder governance limit. This logging surface has no per-user deletion route, bulk export route, or subscription route, and retention or cold-storage configuration is not exposed. A product subject to a deletion workflow should not assume that an internal support dashboard satisfies it. Not suitable when per-user erasure or scheduled export is a hard requirement; select a logging system with those documented controls.

The flag side has boundaries too: no change audit log, evaluation statistics, parent-child dependencies, recycle bin for deletion, or push updates to clients. Clients can poll. For a three-rule rollout, keep the flag decision and the pricing audit event separate so neither is mistaken for functionality the other does not provide.

Small scope wins here.

Missing signals need separate tools

Centralized logs do not cover every operational failure. There are no alert or notification routes for thresholds, calls, SMS, or webhooks, so an alert would require polling the query API and owning the notification logic. There is also no distributed trace query or span tree; trace_id and span_id can correlate log records, but those fields do not create trace exploration.

Use Healthchecks or a similar heartbeat product when the dangerous case is silence — for example, a scheduled pricing job that never ran. Use Electron's crash reporting path when native minidumps and crash collection matter. Source-map decoding, crash symbolication, Session Replay, synthetic checks, and heartbeat monitoring sit outside this logging feature. If those capabilities drive support response, start with the specialized tool instead of stretching a recent-log dashboard past its job.

For the solo SaaS decision, the rule is blunt: outsource the undifferentiated ingestion work, but do not outsource the meaning of cost_center. Ship the searchable record this week. Expand only when a named investigation, compliance obligation, or operating workflow proves the need.

References

Top comments (0)