DEV Community

ArthurFinley2291
ArthurFinley2291

Posted on

Support Incident Reconstruction: Node.js Middleware Manages Feature Flags via Revisioned API

#node #featureflags #observability #backend

Short answer: Use request-local Node.js middleware to resolve one immutable feature-flag revision, append its decision context before the response, and make every API rollback an idempotent compare-and-swap.

Evidence first.

A feature-flag middleware for a customer-support backend should treat every evaluation as an auditable decision, with the decision context attached to the request evidence before a rollback is attempted. The practical rule is simple: the control API may change the next decision, but it must never rewrite the evidence for the decision already served.

This is an architecture decision record for an API-controlled flag in a support system where an agent needs to reconstruct why a customer saw a particular workflow. The example is deliberately small, because the dangerous part is not the HTTP handler. It is the boundary between mutable configuration and immutable incident evidence.

What must remain true during a rollback?

Three invariants drive the design. First, each request gets one resolved flag snapshot; downstream code does not fetch the flag again and observe a different value. Second, the audit record contains the flag key, value, configuration revision, evaluation time, and request correlation identifier. Third, a rollback creates a new revision and leaves the previous revision readable. Deleting or editing history makes incident reconstruction speculative.

The failure boundary is equally important. A control-plane timeout must not silently become a new business decision. For a support workflow, the middleware needs an explicit policy for an unavailable flag: use a declared fallback, mark the evaluation as degraded, and preserve that fact in the event. A fallback is an operational choice, not evidence that the flag was disabled.

Option Useful property Failure boundary Decision
Read the control API on every request Fresh configuration Request latency and split-brain evaluations Reject for the request path
Cache a value without a revision Low overhead Evidence cannot identify the served configuration Reject
Resolve one revision per request and emit context Stable evaluation and reconstruction Requires durable append and explicit expiry Adopt

This is the decision. Keep the request path boring and make the evidence precise.

How can Node.js middleware manage feature flags via an audit revision?

The control endpoint is allowed to publish a new revision, while the request middleware reads an immutable snapshot. In a Node.js service, the same shape can be implemented with an async middleware function and a request-local context. The following Go example shows the critical path without binding the article to a commercial SDK; the interfaces map directly to a Node.js adapter backed by an in-process cache or a standard configuration service.

package flags

import (
    "context"
    "time"
)

type Decision struct {
    Key       string
    Value     bool
    Revision  uint64
    Evaluated time.Time
    Degraded  bool
}

type Snapshot interface {
    Resolve(ctx context.Context, key string) (Decision, error)
}

type Evidence interface {
    Append(ctx context.Context, event map[string]any) error
}

func WithFlag(snapshot Snapshot, evidence Evidence, key string, next func(context.Context) error) func(context.Context) error {
    return func(ctx context.Context) error {
        decision, err := snapshot.Resolve(ctx, key)
        if err != nil {
            decision = Decision{
                Key:       key,
                Value:     false,
                Revision:  0,
                Evaluated: time.Now().UTC(),
                Degraded:  true,
            }
        }

        event := map[string]any{
            "type":        "feature_flag.evaluated",
            "flag_key":   decision.Key,
            "flag_value": decision.Value,
            "revision":   decision.Revision,
            "evaluated":  decision.Evaluated,
            "degraded":   decision.Degraded,
        }
        if err := evidence.Append(ctx, event); err != nil {
            return err
        }

        ctx = context.WithValue(ctx, decision.Key, decision)
        return next(ctx)
    }
}
Enter fullscreen mode Exit fullscreen mode

The important ordering is evidence before the business handler returns a customer-visible result. In production, the append operation should carry a correlation ID and an idempotency key derived from the request identity; the example omits transport plumbing to keep the invariant visible. A practical test matrix includes the request that starts just before a revision changes, the retry that arrives after a timeout, and the fallback that is selected while the control plane is unavailable. Those three cases expose different evidence failures, so collapsing them into one generic error metric loses information. If the evidence sink is unavailable, the service needs a deliberate fail-open or fail-closed policy. A customer-support system that promises reconstruction should usually fail closed for high-risk actions, while a low-risk display hint may continue with a degraded marker. That is a domain boundary, not a universal observability rule.

Logs are not automatically an audit trail. OpenTelemetry describes logs as a signal that records a timestamped event and its context, but retention, immutability, access control, and reconciliation remain system responsibilities. Put the flag decision into structured attributes, then route the resulting event to storage with the durability and retention properties required by the incident process.

What does the API need to publish, and what must it refuse?

The control API should accept a flag key, an intended value, an operator or automation identity, an expiry, and an expected current revision. The expected revision is a compare-and-swap guard: a rollback based on stale console state must fail rather than overwrite a newer decision. The response should return the newly created revision, never imply that historical rows were edited.

A minimal mutation is conceptually:

type Change struct {
    Key             string    `json:"key"`
    Value           bool      `json:"value"`
    ExpectedVersion uint64    `json:"expected_version"`
    Actor           string    `json:"actor"`
    ExpiresAt       time.Time `json:"expires_at"`
}

type Published struct {
    Key      string `json:"key"`
    Value    bool   `json:"value"`
    Revision uint64 `json:"revision"`
}
Enter fullscreen mode Exit fullscreen mode

The handler should reject an empty key, an expired change, an unknown revision, and a mutation whose actor cannot be authenticated. It should also reject an unbounded flag: every temporary incident switch needs an owner and an expiry, because forgotten flags become invisible branches in support investigations. These validations are cheap compared with explaining a contradictory customer timeline later.

Idempotency belongs here too. A retry after a network interruption must not publish two logically identical rollbacks. Store the idempotency key with the resulting revision, and return the original result for a repeated key. This is an exactly-once mindset applied to the effect, even though the transport and storage layers may each deliver a message more than once.

Which rejected design is still valid somewhere?

The rejected design is a direct control-plane read from the request handler followed by an unstructured log line. It is attractive during a first implementation: the API is authoritative, there is no cache invalidation problem, and a developer can see a message in a log viewer. The cost appears during a rollback. Two requests can cross a configuration change, one request can evaluate twice, and the text log may omit the revision that distinguishes the two states.

That design remains valid for a development-only tool where consistency and incident reconstruction are explicitly out of scope. It is also reasonable for a one-shot administrative command that is not serving customer traffic. It is not a sound default for a support workflow whose job is to explain what happened to one customer.

The alternative rejected earlier, a global cache with no request-local snapshot, has a narrower valid use case: coarse routing where a few seconds of propagation delay is acceptable and the route itself is not used as evidence. Once the value changes eligibility, pricing, access, or a support action, the revision must travel with the decision.

Operating the evidence after deployment

Tests should cover more than the happy path. A useful table-driven suite checks stale expected revisions, duplicate idempotency keys, expired changes, control-plane errors, and two evaluations within one request. A property test can assert that a published revision is never changed after append. Integration tests should then restart the service between publication and evaluation, because in-memory assumptions are where rollback stories often become fiction.

During deployment, expose counts for evaluated decisions, degraded evaluations, append failures, rejected stale writes, and expired flags. Keep the metric labels bounded: flag keys may be high-cardinality in a large system, so route detailed identity to structured events and use a controlled metric dimension. Traces can connect the request span to the evidence event; they should not be the only retained record when the support process requires reconstruction after trace retention expires.

The rollback runbook should name the revision being reverted, capture the operator identity, confirm propagation, and verify that a fresh support lookup can retrieve both the old and new decisions. Rollback safety is proven by the resulting timeline, not by the speed of the mutation endpoint.

A feature flag is configuration. A flag evaluation in a customer-support request is evidence. Treating those as separate records, joined by a stable revision and correlation identifier, gives the team a defensible reconstruction path while keeping the API small enough to reason about.

The trade-off is real: immutable evidence consumes retention and operational attention, while a request-local snapshot can serve a slightly stale revision. For a support investigation, that bounded staleness is preferable to a record that cannot identify what the customer actually received.

References

Top comments (0)