EchoF76

Posted on Nov 4

Node Toggle Service: Building Auditable Feature Flags for Incident Rollback

#observability #featureflags #developertools #operations

Short answer: build a feature flags service in Node around a small, immutable toggle decision event before you make it a log line. Store the flag key, actor, rollout inputs, decision, and a correlation identifier together. Keep current state separately so a rollback changes behavior without erasing the evidence that explains what happened.

That design is more useful than a larger toggle API for a developer-tools team. An incident responder can reconstruct which configuration was active, which request saw it, and which change made the result different. I build RAG and agent features, so I am wary of two traps: a notebook makes a toggle look like a boolean, while production makes it a decision with context. An extra event field has a token and storage cost; missing it during an incident costs more.

I initially treated the audit event as optional noise. Later I found the decision itself was the only reliable explanation of a mixed customer result.

Keep it boring.

The practical payoff appears during a partial rollback. Imagine a workspace that received the new ingestion path, retried twice, and then reported a malformed result after the operator disabled the flag. A current-state lookup says only that the flag is off. A revisioned evaluation record can show that the first request used revision 17, the retry used the same captured revision, and later traffic used revision 18. That timeline separates a rollback delay from a targeting error and gives the responder a bounded set of records to inspect. It also prevents a common debugging mistake: attributing every symptom in the incident window to the setting that is current at query time. The extra fields are not decoration; they are the join keys for the reconstruction.

How can a Node service build feature flags with safer toggles?

A flag service has two related records. The state record says what should happen now. The decision event says what happened for one evaluation. Mixing them makes rollback dangerous because replacing state can also overwrite the explanation of old behavior.

For a developer tool, the minimum decision event includes the flag key and revision, the tenant or workspace being evaluated, normalized targeting inputs, the resulting decision, the evaluation timestamp, and request or trace correlation. It should also identify the operator or automation that changed the state. Do not put the full customer payload in every event. A stable subject reference and the attributes that influenced targeting are enough.

A rollback creates a new state revision. It does not edit old decisions. If revision 17 enabled a new ingestion path and revision 18 disables it, an incident query can distinguish requests evaluated under each revision. That distinction matters when a retry arrives after an operator presses rollback.

OpenTelemetry describes logs as a structured signal with fields including timestamps, severity, body, attributes, and trace context. A toggle decision maps naturally to structured attributes, while the state-change event carries the actor and revision. The important part is the relationship between the records, not a particular log backend.

A short event might look like this:

{
  "event_type": "feature_flag.evaluated",
  "flag_key": "new_ingestion_path",
  "revision": 17,
  "subject_id": "ws_4821",
  "decision": "off",
  "reason": "percentage_bucket",
  "correlation_id": "req_91f2"
}

The identifiers are illustrative. The shape is the point: an investigator should join a request, an evaluation, and a configuration revision without guessing.

Why is the obvious in-memory service not enough?

The first implementation is often a dictionary and a conditional. That is a fine notebook experiment. It is a poor incident record because process restarts lose state, concurrent updates can race, and a boolean response gives no reason for the decision.

A useful boundary is a storage interface. The evaluator should not know if state lives in a file, database, or replicated cache. It needs a consistent snapshot and an append-only sink for evaluation events. The sink can be asynchronous, but its delivery policy must be explicit. If evidence is required for rollback analysis, silently dropping an event is not an acceptable default.

from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Protocol


@dataclass(frozen=True)
class FlagState:
    key: str
    revision: int
    enabled: bool
    rollout_percent: int


@dataclass(frozen=True)
class Evaluation:
    key: str
    revision: int
    subject_id: str
    decision: bool
    reason: str
    correlation_id: str
    evaluated_at: str


class FlagStore(Protocol):
    def read(self, key: str) -> FlagState: ...


def evaluate(store: FlagStore, key: str, subject_id: str,
             correlation_id: str, bucket: int) -> Evaluation:
    state = store.read(key)
    decision = state.enabled and bucket < state.rollout_percent
    reason = "disabled" if not state.enabled else "percentage_bucket"
    return Evaluation(
        key=key,
        revision=state.revision,
        subject_id=subject_id,
        decision=decision,
        reason=reason,
        correlation_id=correlation_id,
        evaluated_at=datetime.now(timezone.utc).isoformat(),
    )

This returns the event instead of hiding publication inside the conditional. The caller can publish with the request's correlation context, test failure policy, and expose a metric without coupling evaluation to a vendor SDK. In a real service, storage must provide a coherent read and revision update; a process-local object cannot guarantee that across workers.

The percentage bucket also needs a deterministic input. Hash a stable subject identifier with the flag key and map the result to a fixed range. A random number per request moves a customer between variants and makes replay impossible.

This is a deliberate trade-off, and the limitation is real: retaining more evidence increases storage and access-control work. The design is not suitable for a high-volume path if the team has no retention or redaction policy. In that case, narrow the event fields or choose a simpler operational signal, but do not call the result a reconstruction trail.

What should happen when evidence publication fails?

If the decision succeeds but event publication fails, should the request continue? The answer follows the purpose of the flag. For a cosmetic experiment, continuing with a health signal may be reasonable. For a migration gate or destructive workflow, missing the evaluation record can make rollback analysis incomplete.

Classify flags by failure policy before implementation: fail closed for safety-sensitive actions, tolerate delayed evidence for low-risk presentation changes, and never claim an event was persisted when it was only queued in memory. That classification belongs in configuration and tests, not in an undocumented catch block.

Keep operational metrics separate from decision events. An error counter tells alerting that the pipeline is unhealthy. The event stream answers which workspace received which decision. One cannot replace the other. Logs and traces also have different retention and access patterns, so define redaction rules before an incident forces the issue.

How do I test the rollback path before production?

An evaluation harness is more valuable here than a large unit-test count. Start with a table of flag states, subjects, and expected decisions, then add a request evaluated under revision 17 before a rollback to revision 18. The expected result comes from the captured revision, not from the state that happens to be current later.

def test_rollback_keeps_the_evaluated_revision(fake_store):
    fake_store.state = FlagState(
        key="new_ingestion_path",
        revision=17,
        enabled=True,
        rollout_percent=100,
    )

    event = evaluate(
        fake_store,
        "new_ingestion_path",
        "ws_4821",
        "req_91f2",
        42,
    )

    fake_store.state = FlagState(
        key="new_ingestion_path",
        revision=18,
        enabled=False,
        rollout_percent=0,
    )

    assert event.revision == 17
    assert event.decision is True

The test catches a serious mistake: looking up current state again while writing evidence. I also test duplicate delivery, malformed targeting input, unknown keys, clock skew, and a store revision conflict. These are mundane cases. They are exactly what makes an incident timeline unreliable.

For an agent or RAG workflow, add a replay check: feed recorded subject attributes and revision into a pure evaluator and compare the result with the stored decision. Before adopting the design, measure event volume, added latency at the chosen delivery guarantee, and the percentage of evaluations replayable from retained fields. Those measurements decide batching, retention, and synchronous versus asynchronous publication.

A small service boundary that survives change

The external API can remain modest: read a flag, evaluate a subject, and inspect revisions. The durable contract is the event schema and revision semantics.

Use idempotency for state changes. A disable request should carry an operator-generated change identifier, so retrying it cannot create competing revisions. Use optimistic concurrency so an old dashboard cannot overwrite a newer rollback. Return the written revision and include it in the audit event.

For reads, decide how stale a flag may be. A local cache reduces dependency pressure but can delay rollback. A short cache lifetime reduces that window but increases reads. A destructive path deserves a tighter freshness contract than a dashboard label.

Keep configuration and evaluation conceptually separate. The configuration plane authenticates operators, validates changes, and records revisions. The evaluation plane performs a predictable read and emits evidence at request volume. If you cannot reconstruct a customer incident from the retained revision, subject reference, decision, reason, and correlation context, the toggle service is not finished.

DEV Community