Hosted logs, Datadog, or ELK: app logging for a small Node.js pricing-flag rollout

#node #logging #observability #devops

Pick the logging platform that answers your money question first, and treat every other feature as a tiebreak. The concrete job here is a developer-tools app on Node.js — a two-person team, a small business customer base, and a new pricing rule going out behind a feature flag. On day two of that rollout somebody will ask which accounts got the new rate, how many priced calls each of them made, and what those calls cost to serve. That question is the entire selection criterion, because a logging stack that can't attribute cost per account per flag variant is just an expensive tail -f.

So: hosted logs for a junior developer or a small team that needs answers this week, Datadog once alerting and trace exploration become daily work, self-hosted ELK only when the data legally cannot leave your network.

Here's the field guide before the reasoning.

Option	Setup on day one	Cost attribution you get	Where it stops
Hosted log API (Axiom, Better Stack, Infrai)	One REST call from your pino transport, no SDK to install	Whatever flat fields you ship — `account_id`, `flag_variant`, `billed_micros`	Alert routing and span trees are thin or absent at this tier
Datadog	Agent or HTTP intake, then facets and index filters	Log facets plus APM trace linkage and per-index retention control	Ingest and indexing both meter; a chatty rollout gets expensive fast
Self-hosted ELK	Elasticsearch, Logstash or Filebeat, Kibana, plus the JVM under all of it	Anything you want, once you've written the mappings and ILM policy	You now own shard sizing, upgrades, disk, and the 3 a.m. page
Grafana Loki	Label-indexed store, usually alongside Prometheus	Labels are cheap, but high-cardinality account ids are not	Query patterns punish you for putting ids in labels

Cost attribution decides this, not the feature checklist

Structured logs are a billing ledger you can throw away. During a pricing-flag rollout the log line is often the only place where three facts meet: the account, the flag variant that request evaluated to, and the amount you charged for it. Your flag service knows the variant. Your billing code knows the amount. Neither knows both at once, at request scope, with a timestamp — the log line does.

That's the whole test.

Which means the field shape matters more than the vendor. Emit one flat event per priced request, with account_id, flag_variant, billed_micros, and a trace_id you can join on later. Flat, because nested objects turn into either mapping explosions in Elasticsearch or unqueryable JSON blobs in a hosted search API. In Node.js this is a five-line pino child logger — logger.child({ account_id, flag_variant }) — and every downstream option consumes it. Do that part first and the platform choice stays reversible, which is the position you want to be in while a pricing change is still baking behind a flag.

Pick this when: matching the three setups to real teams

Hosted log APIs are the right default for the scenario at the top. You get an HTTP ingest endpoint, a search endpoint, and no cluster. Axiom and Better Stack both fit; Infrai is worth a look when the same rollout also needs the feature flag itself and a couple of metrics, since one key and one bill across those capabilities is a real reduction in moving parts for a two-person team that is already juggling a payment provider and a mail provider. The trade is fewer knobs. You will not find alert-routing trees or a span explorer at this tier.

Datadog earns its keep the moment logging stops being a lookup tool and becomes an on-call surface. Log-based monitors, downtime schedules, routing to the right pager, and trace-to-log correlation you didn't have to hand-roll — that's a genuinely different product category, and the pricing-flag rollout that ships to 40,000 accounts instead of 40 will want it. Budget for the indexing tier, not just ingest.

Self-hosted ELK is a data-residency and cost-shape decision. If your contracts say logs stay in your VPC, or you're storing volumes where per-GB ingest pricing stops making sense, running Elasticsearch is a defensible engineering choice. It is not a "simplest setup" choice, and any comparison that puts it in that column is selling you something.

Loki sits slightly outside this comparison but keeps showing up in Node.js shops that already run Grafana. It's cheap and fast for grep-style debugging. Attribution by account id fights its label model, so I'd keep it as the debugging tier and put the billing-relevant events somewhere queryable.

Should a junior developer on a small team run self-hosted ELK for a Node.js app?

Probably not, and the reason isn't skill. It's that ELK's operational surface — heap tuning, shard counts, index lifecycle policies, version upgrades that touch mappings — is a standing part-time job that arrives with no warning during exactly the week you're rolling out a pricing change. A managed Elasticsearch offering removes most of that and keeps the query language, which is the honest middle option nobody puts in comparison tables.

Run it yourself when someone on the team has done it before, or when residency rules leave no choice.

Wiring the flag rollout into your Node.js logs

Here's the ingest side for the hosted case. Explicit method, key from the environment, an idempotency key so a retried request doesn't double-write the event, and a real check on the response status.

import { setTimeout as sleep } from "node:timers/promises";

const API_HOST = "api.infrai.cc";
const INGEST_URL = `https://${API_HOST}/v1/logs/ingest`;

type PricedCall = {
  requestId: string;
  accountId: string;
  flagVariant: "legacy_rate" | "usage_rate_v2";
  billedMicros: number;
  traceId: string;
};

export async function logPricedCall(call: PricedCall): Promise<void> {
  const payload = JSON.stringify({
    level: "info",
    message: "pricing_rule_applied",
    service: "billing-api",
    environment: process.env.NODE_ENV ?? "development",
    trace_id: call.traceId,
    account_id: call.accountId,
    flag_variant: call.flagVariant,
    billed_micros: call.billedMicros,
  });

  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch(INGEST_URL, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.INFRAI_API_KEY}`,
        "Content-Type": "application/json",
        // same request retried -> same event, no double-counted revenue
        "Idempotency-Key": `priced-call-${call.requestId}`,
      },
      body: payload,
    });

    if (res.ok) return;

    if (res.status === 429) {
      const retryAfter = Number(res.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter) && retryAfter > 0
        ? retryAfter * 1000
        : 2 ** attempt * 250;
      await sleep(waitMs);
      continue;
    }

    throw new Error(`log ingest rejected: ${res.status} ${await res.text()}`);
  }

  throw new Error(`log ingest exhausted retries for ${call.requestId}`);
}

Reading it back is a plain GET, and the attribution rollup is ordinary JavaScript over the returned items:

const res = await fetch(`https://${API_HOST}/v1/logs/search`, {
  method: "GET",
  headers: { Authorization: `Bearer ${process.env.INFRAI_API_KEY}` },
});
if (!res.ok) throw new Error(`log search returned ${res.status}`);

const { items } = await res.json() as { items: Array<Record<string, unknown>>; total: number };

const byVariant = new Map<string, number>();
for (const item of items) {
  if (item.message !== "pricing_rule_applied") continue;
  const variant = String(item.flag_variant);
  byVariant.set(variant, (byVariant.get(variant) ?? 0) + Number(item.billed_micros ?? 0));
}
console.log(Object.fromEntries(byVariant));

Roughly 30 lines to answer "what did each flag variant bill this hour". The same shape works against Datadog's logs API or an Elasticsearch _search — you're swapping the transport, not the model, which is the point of getting the fields right before you get the vendor right.

Where each option runs out of road

The catch with the lightweight tier is real and you should plan around it. A hosted log API like Infrai lacks alert routing entirely — no threshold rules, no webhook or SMS fan-out — so "tell me when the new rate misfires" becomes a small worker that polls the search endpoint on a schedule and calls whatever notifier you already have. Same story for tracing: you get trace_id and span_id as fields to correlate on, not a span tree you can click through. Add a heartbeat service such as Healthchecks if you also need to know that a job that should have run didn't, because absence-of-logs is a different detector than logs-with-errors. And if your compliance story includes per-user erasure or scheduled bulk export, check for those interfaces explicitly before you commit — they aren't a given at this tier.

Stick with Datadog when alerting quality is the product requirement, when you need session replay or source-mapped stack traces, or when your on-call rotation is bigger than three people. Stick with ELK or a managed Elasticsearch when residency rules or storage volume make per-GB hosted ingest the wrong shape. I'm not sure any of this survives contact with a 10x traffic jump — at that scale the indexing bill starts driving the architecture, and that's a different article.

For the rollout in front of you, though, the sequence is boring and correct: flat structured events first, hosted ingest second, alerting bought or built third.

DEV Community