Backend Product Analytics: 4 Custom Metrics for Nightly Health Pipelines

Use backend custom metrics as the evidence layer for a product analytics dashboard, and use feature-flag statistics as release context. A flag can tell you which behavior was enabled. It cannot, by itself, prove that a nightly health-data pipeline accepted, transformed, rejected, or delivered a record.

That boundary matters more than the dashboard library. For a one-person SaaS shipping weekly, I would start with four low-cardinality outcome signals: runs, records accepted, records rejected, and run duration. Then I would attach a small release-context field such as pipeline_variant at the point where the backend actually evaluates the flag. This keeps the dashboard useful without turning every internal detail into a metric.

Short answer: measure business outcomes in the backend; treat flag data as an annotation that helps explain changes. Optimize for signal quality first. A dashboard nobody can trust costs more engineering time than it saves.

Should a product analytics dashboard use backend custom metrics or flag stats?

Feature flags and product outcomes describe different events. A flag evaluation records a configuration decision for some context. The nightly job produces operational outcomes later: a run starts, records pass or fail validation, transformations finish, and an export may complete. Those events can diverge because evaluation is not execution.

Consider a rollout that exposes a new normalization path to half of eligible tenants. An exposure count says the decision code selected the new path. It does not say how many records reached that path, how many were valid, or whether the run finished. Joining those concepts in one chart without preserving the distinction creates impressive-looking noise.

The clean model has two layers:

Layer	Question it answers	Useful dimensions	Keep out
Backend outcomes	What did the pipeline do?	outcome, stage, variant	patient ID, raw payload, error text
Release context	Under which behavior did it run?	flag key, variant	outcome claims inferred from exposure

The primary decision axis is signal quality versus noise. Every extra label creates more series and more ways to split a small sample. Keep a dimension only if it changes an action: pause a rollout, inspect a stage, retry a run, or fix a validation rule. Debug detail belongs in structured logs, linked by an opaque run identifier. This is also a data-governance boundary. Article 17 of the GDPR defines a right to erasure under specified conditions. A practical design should therefore avoid copying person-level identifiers into long-lived metric labels and should give structured-log records an explicit retention and deletion path. The legal applicability and retention period depend on the processing context; engineering cannot infer them from telemetry alone.

Noise wins otherwise.

The smallest working implementation

I would ship one typed recorder behind a generic interface. The pipeline owns the event names and allowed dimensions. The storage or dashboard adapter sits outside that contract, so changing the display layer does not rewrite domain code.

type PipelineVariant = "control" | "candidate";
type PipelineStage = "ingest" | "validate" | "transform" | "export";
type RecordOutcome = "accepted" | "rejected";

interface MetricSink {
  increment(
    name: string,
    value: number,
    labels: Record<string, string>,
  ): void;
  observe(
    name: string,
    value: number,
    labels: Record<string, string>,
  ): void;
}

interface PipelineMetrics {
  runStarted(variant: PipelineVariant): void;
  recordProcessed(
    stage: PipelineStage,
    outcome: RecordOutcome,
    variant: PipelineVariant,
  ): void;
  runFinished(durationMs: number, variant: PipelineVariant): void;
}

function createPipelineMetrics(sink: MetricSink): PipelineMetrics {
  return {
    runStarted(variant) {
      sink.increment("pipeline_runs_total", 1, { variant });
    },
    recordProcessed(stage, outcome, variant) {
      sink.increment("pipeline_records_total", 1, {
        stage,
        outcome,
        variant,
      });
    },
    runFinished(durationMs, variant) {
      sink.observe("pipeline_run_duration_ms", durationMs, { variant });
    },
  };
}

Four dashboard signals can now be derived without exposing a person: run count, accepted record count, rejected record count, and duration distribution. The code emits three metric names because accepted and rejected records are outcomes on the same counter. That shared counter prevents two independently evolving definitions of “processed.”

Keep the run identifier in logs instead of metric labels. A structured record can carry run_id, stage, outcome, a stable error code, and the evaluated variant. It should omit raw health data and direct identifiers. If an operator sees a rejection spike, the dashboard narrows the time window and variant; the logs supply bounded diagnostic evidence.

type PipelineLog = {
  timestamp: string;
  run_id: string;
  stage: PipelineStage;
  outcome: RecordOutcome;
  error_code?: "SCHEMA_INVALID" | "MAPPING_MISSING" | "EXPORT_REJECTED";
  pipeline_variant: PipelineVariant;
};

function pipelineLog(entry: PipelineLog): string {
  return JSON.stringify(entry);
}

This interface is deliberately boring. Good. The undifferentiated work should stay replaceable, while the accepted/rejected definitions remain owned by the application.

Make the dashboard earn its maintenance cost

A solo operator does not need a wall of charts. The first view should support one decision after the nightly schedule: did the run finish, and did the outcome mix move enough to investigate? Put completion and duration first. Put accepted and rejected counts next, split by stage only when diagnosing. Show the evaluated variant as a filter or annotation, not as the headline KPI.

Ratios need their numerator and denominator nearby. A rejection percentage without total records can make one rejected test record look important. A raw rejection count can look alarming when throughput grew. Display both values, then let the operator open the relevant structured logs.

Freshness deserves explicit treatment too. A blank chart, a delayed run, and a run with zero eligible records are three different states. Emit the run-start outcome, retain the scheduler's expected window in application configuration, and make “no completed run by the expected time” a separate condition. Do not translate absence into zero.

The revenue-per-hour test is blunt: if a panel cannot change the next action, remove it. Weekly shipping already creates enough release context. The dashboard should compress investigation time, not become another product that needs weekly care.

Failure modes worth testing before release

Test the instrumentation as a contract. A unit test should assert that every terminal record path emits exactly one accepted or rejected outcome. A run-level test should prove that completion is recorded only after the final stage succeeds. Test the candidate and control paths with the same fixtures, because a missing metric call in one branch creates a false improvement.

Retries are the sharp edge. If each attempt increments a business-outcome counter, a single record can appear several times. Decide what the dashboard represents. Attempt metrics may count every try for operational load, while product outcomes should count the final disposition once. Encode that distinction in separate names rather than expecting readers to remember it. Late data needs a rule as well. Attribute an outcome to the time it occurred, and expose the run's scheduled date as bounded log context if operators need to reconcile nightly batches. Rewriting historical aggregates can conceal what the system knew at the time. If correction is required, document the correction event. Deployment should fail open for telemetry transport and fail closed for invalid health records. In other words, a temporarily unavailable metric sink should not corrupt the pipeline result, while schema validation must still protect the domain boundary. Buffering, dropping, and retrying telemetry each have resource costs; set a finite queue and report dropped observations through an internal counter.

Do less here. It ships faster and is easier to audit.

What I would change at scale

At higher volume, I would preserve the same semantic contract and change the transport and aggregation behind it. Batch metric writes, bound queues, sample verbose success logs, and retain all error-class logs according to the approved policy. Cardinality budgets should be enforced in code review or automated tests, especially for tenant, run, and error fields.

I would also separate operational access from analytics access. The product dashboard needs aggregates. An incident investigation may need run-level logs. A deletion workflow needs an inventory of stores that can contain personal data, an authorized request path, and evidence that the applicable records were handled. Those jobs have different audiences and permissions, even when they originate from the same pipeline.

The trade-off remains simple. More dimensions make rare failures easier to isolate, but they increase noise, storage, and the chance of retaining sensitive context. Start with stage, outcome, and variant. Add another dimension only after a real decision cannot be made without it, then test its bounded values.

For a nightly health-data pipeline, backend outcomes are the durable product evidence. Feature-flag statistics explain rollout exposure. Keep both, join them through a small evaluated variant, and never ask one to impersonate the other.

Further reading

• GDPR Article 17, “Right to erasure”: https://gdpr-info.eu/art-17-gdpr/