LiamFoster1844

Posted on Oct 31

Node.js Feature Flags REST API Implementation for Nightly Pipeline Evidence (3 Trade-offs)

#node #featureflags #restapi #observability

Short answer: implement feature flags in Node.js behind a versioned REST API, log every evaluated decision with the nightly run context, and choose a narrow custom contract only when its audit and on-call burden fits the team.

A nightly data pipeline rarely fails at the line that first looks suspicious. The practical answer is to treat a feature-flag decision as an observable event, expose a small REST control surface, and retain enough context to replay the decision during incident reconstruction. Three decisions matter: who may change a flag, how a worker evaluates it, and which evidence survives after the run.

This is the least complex design I would put in front of a B2B SaaS platform team: a versioned flag document, deterministic evaluation from a request context, an audit record for every mutation, and structured logs emitted at the pipeline boundary. The API can be implemented in Node.js, but the contract is the important part.

The alert page is usually short: the 02:00 UTC import missed its completion SLO, one tenant has a partial snapshot, and the retry count is rising. The on-call needs to answer a narrower question before touching the retry button: did the new enrichment path change the decision for that tenant, or did the source data arrive late?

What should the on-call see first?

Start with the page, then work backward. A useful event contains the pipeline run ID, tenant ID, flag key, flag version, evaluated value, reason, and a correlation ID. It should also carry the schema version and the worker release, because a true decision can still be impossible to reproduce if the evaluator changed underneath it.

A log line such as enrichment_path=false is not enough. It tells us the result and hides the inputs that produced it. The following compact event shape can be serialized by a Node.js worker before it writes to its logging pipeline.

type FlagDecision struct {
    SchemaVersion string `json:"schema_version"`
    RunID string `json:"run_id"`
    TenantID string `json:"tenant_id"`
    FlagKey string `json:"flag_key"`
    FlagVersion int64 `json:"flag_version"`
    Enabled bool `json:"enabled"`
    Reason string `json:"reason"`
    CorrelationID string `json:"correlation_id"`
    WorkerRelease string `json:"worker_release"`
}

That event is useful only if it reaches storage before the worker exits. I prefer writing it where the choice affects work, not only inside the flag service. A control-plane audit trail says someone changed a document; a data-plane decision log says what the pipeline actually did. Those are different facts.

The signal should page on the outcome, not on the flag merely being enabled. An SLO can track successful completion of scheduled runs while a diagnostic query groups failed runs by flag_key, flag_version, and worker_release. This keeps a rollout from becoming an alert by itself and still makes a causal relationship visible when the error budget starts moving.

Do not put raw customer payloads into the decision event.

Tenant identifiers, run identifiers, and bounded reason codes are generally enough to join records. Payload data increases retention and access risk while making the query slower. This design has a clear limit: a small service with weak identity controls or no durable audit store is a poor control plane, even if its evaluator is correct. Keep the flag contract elsewhere until those operational prerequisites exist.

How does a REST flag contract preserve evidence?

The REST API needs fewer routes than most flag implementations grow into. A read endpoint returns the current document and its version. A mutation endpoint requires an expected version. The server rejects a stale write instead of silently replacing a change made by another operator.

Here is the essential optimistic-concurrency check in Go. It is deliberately boring; that is a feature for an incident-control path.

type Flag struct {
    Key string `json:"key"`
    Enabled bool `json:"enabled"`
    Version int64 `json:"version"`
}

func updateFlag(current Flag, requested bool, expectedVersion int64) (Flag, error) {
    if current.Version != expectedVersion {
        return Flag{}, fmt.Errorf("stale flag version: expected %d, current %d", expectedVersion, current.Version)
    }
    current.Enabled = requested
    current.Version++
    return current, nil
}

In an HTTP implementation, GET /flags/{key} can return the document with an ETag, while PUT /flags/{key} accepts If-Match and returns 412 Precondition Failed for a stale representation. The route names matter less than the invariant: every accepted mutation produces a new version and an audit record containing actor, timestamp, old value, new value, and reason.

Keep targeting rules bounded, validate their schema, and make the evaluation context explicit. For a nightly job, that context might include tenant ID, pipeline name, and release. It should not include an unreviewed slice of the source dataset.

The failure mode I watch for is an apparently successful rollback that changes the control document but cannot explain already-running workers. Decide whether workers poll, cache, or pin a version for a run, then record that behavior. A run that began with version 18 should remain queryable as version 18 after version 19 is published.

Where should instrumentation change?

Work backward from the missing evidence. If the page tells us only that the final table is incomplete, add signals at three points: flag evaluation, source acquisition, and output commit. Each point should share run_id and correlation_id; otherwise the operator has three logs and three separate searches.

The logs signal in OpenTelemetry is designed for records with severity, body, attributes, and resource context. That gives a standards-based shape for transporting the event, while the backend remains an implementation choice. The useful distinction is between the event fields needed for reconstruction and the transport used to collect them.

A worker can emit one decision event and one outcome event per tenant-run, rather than logging every row in a large import. That is a capacity decision. If a pipeline processes 40,000 tenant partitions and writes five events per partition, the upper bound is 200,000 records per run before retries; retention, indexing, and query concurrency must be planned against that bound. Sampling the decision event would make a rare tenant-specific change disappear, so sample noisy row-level diagnostics instead.

Keep the decision record boring.

The 200,000-event estimate is only the first capacity number. A retry policy can multiply it, a backfill can run beside the scheduled import, and a query that groups by tenant can cost more than an aggregate counter even when both describe the same incident. I would reserve write capacity for the normal run plus the largest planned backfill, then exercise the query path with realistic cardinality before attaching it to an incident procedure. The point is not to predict the exact storage bill; it is to prevent the observability path from becoming the next dependency that fails while the team is reconstructing the first failure. A small, stable event schema also makes retention rules easier to state: keep decisions for the period in which a nightly result can be corrected, and keep aggregate SLO history longer if the organization needs trend analysis.

The alert threshold also has a cost. A low threshold catches a small regression early but pages on normal late-arriving input. A high threshold protects the on-call from noise and delays detection. Set the alert around the service objective and keep a separate diagnostic query for flag cohorts; do not turn every flag change into a page. False positives spend the same scarce resource as a real incident: attention.

That distinction matters.

For a Node.js implementation, keep the evaluator pure and pass its result plus immutable context to the logger. Do not let middleware infer the tenant from mutable global state in a worker process. Test the evaluator with a table of contexts, then test the REST handler separately for authorization, version conflicts, and malformed rules. The trade-off is deliberate: this leaves targeting features narrower than a large control plane, but the narrower surface is easier to capacity-plan, review, and reconstruct under pressure.

Should we build the control plane or use an existing one?

This is a capacity and ownership decision, not a contest between dashboards. A small team may use a managed control plane to avoid owning availability, authentication integration, and audit retention. A team with strict data residency or unusual evaluation rules may build the narrow contract and accept the ongoing on-call work.

Decision area	Build a narrow service	Use an existing service
Evaluation model	Exact fit for tenant and run context	Constrained by its rule model
Audit evidence	Schema and retention are yours	Available only within its export boundary
On-call load	You own storage, upgrades, and recovery	You own integration and provider dependencies
Lock-in	Lower at the API boundary if the schema is portable	Higher when clients depend on proprietary semantics
Capacity planning	Query, write, and cache limits are explicit	Limits and export behavior need verification

For comparison, OpenTelemetry provides portable signal concepts and collection paths, Prometheus is primarily a metrics and time-series system, and Loki is oriented around log storage and querying. Those boundaries help assign responsibility: none of them, by that description alone, is a complete authorization, mutation, evaluation, and audit contract for feature flags. Treating a log backend as the flag control plane leaves the write policy and consistency model implicit.

Keep the application-facing contract independent from the chosen transport or storage backend. That lets the team replace a collector or query system without rewriting every worker, and makes a self-hosted versus managed decision reviewable in terms of SLOs, failure domains, staffing, and retention.

The false-positive bill arrives later

After the instrumentation change, the next page should answer three things quickly: which flag version the affected run used, which cohort saw the changed decision, and whether the output failure began before or after that decision. If it cannot, the system has telemetry but not incident evidence.

The safe stopping point is a versioned REST contract, deterministic evaluation, and structured decision records joined to the nightly run. Keep the rollout reversible, but make the record of each decision durable. The goal is not more flags. It is a reconstruction path that survives a rushed rollback and leaves the next operator with fewer guesses.

DEV Community