YancySterling6529

Posted on Oct 16

Node.js Feature Metrics Backend: Choose 3-Signal Reconstruction for SaaS Dashboard Charts

#observability #metrics #node

Use metrics for the charts, but preserve a small, structured event trail for reconstruction when a scheduled media import stops producing results. TL;DR: count three stage outcomes in a low-cardinality time series, then keep correlated start, finish, and failure events under an explicit retention policy. A logs-only dashboard looks quicker until every panel needs query parsing and a late job becomes indistinguishable from an empty source. A metrics-only design is cheap to read but cannot explain which run stalled.

The deciding constraint is incident reconstruction, not which backend can draw a line fastest. For a small Node.js team, the useful experiment is whether an on-call engineer can distinguish “scheduler did not fire,” “worker started but did not finish,” and “worker finished with zero accepted records” without inspecting personal data. That changes the telemetry shape.

Should a feature metrics dashboard choose an API or logs backend?

A counter named imports_completed_total answers one narrow question: did completion activity increase? It does not say whether no work was due, a queue stopped moving, an upstream feed was empty, or validation rejected every item. Those states all flatten into the same quiet chart.

Silence lies.

The focused design uses three signals per bounded dimension set: scheduled runs, started runs, and produced results. A fourth failure counter can help operations, but it should not be used as proof of liveness because a process can hang without emitting a failure. Keep labels to operational categories such as feed class, outcome, and deployment version. Do not put URLs, titles, tenant IDs, run IDs, or error text in metric labels; each new label value creates another time series and also makes erasure and access control harder.

For example, compare counts over the same scheduling window:

Observation	Likely boundary to inspect	Evidence needed next
No scheduled runs	scheduler or schedule configuration	deployment and schedule-change events
Scheduled, no starts	dispatch or worker availability	one correlated dispatch event
Starts, no finishes	worker, timeout, or dependency	run start plus terminal event or timeout
Finishes, zero results	source content or validation	aggregate rejection reasons
Results return after a gap	delayed execution	timestamps from the affected runs

This is a diagnostic map, not an automatic verdict. Clock skew, delayed ingestion, retries, and late-arriving events can alter the apparent order. The alert should open an investigation with a bounded window rather than claim a root cause.

The event contract is deliberately narrow

Start with a single scheduled job type and one alert rule. Emit counters for the chart and structured events for the individual run. Then deliberately exercise four states in a test environment: a normal result, an empty result, a rejected result, and a run that exceeds its deadline. The pass condition is practical: the chart shows the gap, while the event trail identifies the last completed boundary without requiring raw media content.

Here is a compact TypeScript boundary. The interfaces avoid coupling the application to a storage product, and the event payload excludes source material by construction.

type Stage = "scheduled" | "started" | "finished" | "failed";
type Outcome = "accepted" | "empty" | "rejected" | "error";

interface MetricSink {
  increment(name: string, labels: Record<string, string>, value?: number): void;
  observe(name: string, labels: Record<string, string>, value: number): void;
}

interface EventSink {
  write(event: ImportEvent): Promise<void>;
}

interface ImportEvent {
  occurredAt: string;
  runId: string;
  stage: Stage;
  feedClass: "partner" | "editorial" | "archive";
  outcome?: Outcome;
  acceptedCount?: number;
  rejectedCount?: number;
  durationMs?: number;
  deployment: string;
}

async function recordImportResult(
  metrics: MetricSink,
  events: EventSink,
  event: ImportEvent,
): Promise<void> {
  const labels = {
    feed_class: event.feedClass,
    stage: event.stage,
    outcome: event.outcome ?? "pending",
  };

  metrics.increment("media_import_runs_total", labels);

  if (event.acceptedCount !== undefined) {
    metrics.increment("media_import_results_total", {
      feed_class: event.feedClass,
      outcome: "accepted",
    }, event.acceptedCount);
  }

  if (event.durationMs !== undefined) {
    metrics.observe("media_import_duration_ms", {
      feed_class: event.feedClass,
      outcome: event.outcome ?? "error",
    }, event.durationMs);
  }

  await events.write(event);
}

The code records a runId only in the event stream. That is intentional. The dashboard needs stable aggregates; reconstruction needs correlation. Mixing those needs by attaching each run identifier to metrics turns a bounded KPI into an ever-growing set of series.

The simple approach I would reject is deriving every chart from free-form log messages. Message wording changes during ordinary refactors, parsing becomes part of the dashboard contract, and the distinction between a missing event and a delayed event remains implicit. Structured events improve that approach, but they still should not become the only chart store when the recurring read is a fixed time-series aggregation.

Two clocks define the incident

A stopped importer may never throw. Detection therefore needs an expected schedule plus a completion deadline. Evaluate the latest due run only after its allowed lateness, and compare stage counts within the same window. The exact interval cannot be universal: it must come from the import schedule and the tolerated delay for the newsroom or archive workflow.

Retries complicate the picture. Count attempts separately from logical runs, or one recovered run can appear as several scheduled jobs. Preserve a stable logical runId in events and an attempt number if the worker retries. Metrics can aggregate attempt outcomes without carrying either identifier.

Feature flags deserve the same discipline. Martin Fowler's Feature Toggles article separates flag categories by their purpose and expected longevity. A release flag that changes the importer should be recorded as deployment context, not copied into an unbounded label value. If a flag changes scheduling or validation, emit a configuration-change event so the timeline can explain why the graph moved. Remove short-lived flag dimensions after the rollout rather than leaving an accidental permanent axis.

The alert should describe the missing transition. “Scheduled runs observed, but no finished run before the deadline” is actionable. “Import count is low” is cheaper to write and harder to trust, especially for feeds whose natural volume can legitimately be zero.

Deletion belongs in the evidence design

EU data protection changes the design before storage selection. The GDPR defines personal data broadly and establishes data minimisation, storage limitation, integrity, and confidentiality principles. A media URL, contributor name, account identifier, or excerpt can create obligations that an aggregate dashboard never needed.

Collect operational categories instead: stage, coarse feed class, aggregate counts, duration, deployment, and a pseudonymous correlation identifier. Pseudonymisation can reduce exposure, but GDPR Article 4 still treats pseudonymised information as personal data when it can be attributed using additional information. Access, retention, and deletion controls still matter.

Metrics and events should have separate retention decisions because they serve different purposes. Aggregated history may be useful for capacity and reliability trends after detailed events have expired. Document who can query each store and test deletion behavior. Do not assume that moving fields into logs makes them exempt from governance.

There is also a cost reason to be strict, though cost is not the main decision. High-cardinality identifiers multiply metric series; verbose event payloads multiply ingestion and storage. A narrow schema reduces both without betting the system on a particular vendor.

Run the four-state drill before selecting storage

Measure the workflow, not a synthetic ingestion benchmark. Run the four failure states and record time to detection, time to identify the last completed stage, false alerts from legitimately empty feeds, chart query latency at the intended retention window, and the number of distinct metric series. Also inspect a sample event as a privacy reviewer would: can it identify a person, reveal unpublished media, or be joined back to an account?

Then test change. Rename an internal log message, deploy a new worker version, toggle a release path, and retry one logical run twice. The KPI chart should remain stable, while the event trail should retain enough context to order those changes. If either side fails, revise the schema before adding more dashboards.

Use the operational questions as selection criteria for any backend:

Can it calculate fixed-window rates and gaps without parsing prose?
Can it correlate a logical run across scheduler and worker boundaries?
Can access and retention differ between aggregates and detailed events?
Can the team export data in a documented format and test that export?
Does query performance remain acceptable at the measured series count and event volume?

The result is deliberately hybrid. Metrics carry the repeated, low-cardinality questions that charts ask all day. Structured events carry just enough evidence to rebuild a failed import's timeline. For a small team shipping Node.js features, that split keeps the first dashboard simple while preserving the detail needed when silence becomes an incident.

DEV Community