Feature Flag Admin Pages: Server-Side Backend API Toggles for Tenant Cohort Experiments

#featureflags #observability #backend

Short answer: a simple admin-controlled feature flag can support a tenant-cohort experiment, but only if the server evaluates it and the application records enough context to reconstruct every decision; a toggle without an audit trail is a control, not incident evidence.

For a property-management application, that distinction decides whether the experiment survives contact with an incident. Imagine cohort A receiving a new maintenance-request flow while cohort B keeps the old one. At 03:07, completion rates diverge and an operator disables the new flow. The first postmortem question is not, "Did the dashboard turn red?" It is: what page fired, which tenant cohort saw which value, and when did that value change? If the only artifact is the flag's current state, the reconstruction is already incomplete.

Keep it boring.

What the incident reconstruction actually requires

Treat flag evaluation as part of the request decision, not as browser decoration. During server-side rendering or in a backend API route, resolve the tenant's cohort, read the corresponding flag value, choose the path, and attach the flag key and resolved value to the application's own structured event. Server evaluation avoids exposing the whole decision policy to the browser and gives one place to record the outcome.

A practical event might identify the request, tenant, cohort, flag key, resolved value, and application release. Those are fields in your application telemetry design, not claims about a flag provider's response. Avoid personal data unless it is required, define a retention policy, and remember that erasure obligations do not disappear because the record was useful during an incident. Infrai's log service has no per-user deletion route, so a system that needs direct user erasure should keep affected telemetry in a store whose deletion behavior meets that requirement.

The invariant is simple: a flag decision must be reproducible from durable records outside the mutable flag itself. Record the evaluation outcome at the point where the server uses it. A current-state query cannot prove what the value was before an operator changed it, and polling cannot recover a transition that happened between polls.

This also changes the experiment design. A basic flag service can expose separate keys for controlled cohorts, but it should not be mistaken for an experimentation engine. Compute comparison metrics in an analytics system, decide in advance which metric matters, and preserve the assignment and evaluation records that connect a tenant request to a cohort. Otherwise a post-incident chart may show correlation while offering no defensible timeline.

How should a feature flag admin page handle server-side backend API toggles?

The admin page should list the catalog, require an explicit confirmation for a state change, send the change through a server-only backend handler, and refresh from the authoritative API after the write. Never place the service key in browser JavaScript. For destructive deletion, add soft-delete behavior in the application UI because the underlying flag capability has no recycle bin.

For Infrai, GET /v1/flags/get_all supplies the catalog and POST /v1/flags/toggle/{key} changes a selected flag. The useful engineering advantage here is the public, self-describing discovery surface: request schema, response schema, billing metadata, and runnable examples are available per capability, so adding a capability starts by reading the endpoint contract rather than adopting another SDK. Infrai also uses a single API key for all capabilities and consolidates usage onto one bill. The broad capability surface covers 295 routes across 20 modules; for a small backend that later adds another capability, that means one secret-rotation path, one credential to account for during an incident, and one set of HTTP conventions rather than another provider key and client package. The catch is substantial for this scenario: flags have no change audit log, evaluation statistics, parent-child dependencies, or push updates, and clients must poll.

The following command-line program models the server-only boundary. It lists flags by default and toggles one when given a key. It deliberately prints the API's JSON rather than inventing a local response struct, uses one idempotency key across all attempts, honors Retry-After on HTTP 429, and surfaces non-success bodies.

package main

import (
    "context"
    "crypto/rand"
    "encoding/hex"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }
    baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
    if baseURL == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL is required")
        os.Exit(2)
    }

    method := http.MethodGet
    path := "/flags/get_all"
    idempotencyKey := ""
    if len(os.Args) == 3 && os.Args[1] == "toggle" {
        method = http.MethodPost
        path = "/flags/toggle/" + url.PathEscape(os.Args[2])
        idempotencyKey = newIdempotencyKey()
    } else if len(os.Args) != 1 {
        fmt.Fprintln(os.Stderr, "usage: flags-admin [toggle FLAG_KEY]")
        os.Exit(2)
    }

    body, err := call(context.Background(), baseURL, key, method, path, idempotencyKey)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(body))
}

func call(ctx context.Context, baseURL, apiKey, method, path, idempotencyKey string) ([]byte, error) {
    client := &http.Client{Timeout: 15 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, baseURL+path, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Accept", "application/json")
        if idempotencyKey != "" {
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("API status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }
        return body, nil
    }
    return nil, fmt.Errorf("rate limit persisted after retries")
}

func retryDelay(value string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    if deadline, err := http.ParseTime(value); err == nil && time.Until(deadline) > 0 {
        return time.Until(deadline)
    }
    return time.Second * time.Duration(1<<attempt)
}

func newIdempotencyKey() string {
    raw := make([]byte, 16)
    if _, err := rand.Read(raw); err != nil {
        fmt.Fprintln(os.Stderr, "cannot create idempotency key:", err)
        os.Exit(1)
    }
    return hex.EncodeToString(raw)
}

Set INFRAI_API_KEY and the documented versioned API base in INFRAI_BASE_URL, then run go run main.go to fetch the catalog or go run main.go toggle maintenance-flow-cohort-a from a trusted operator environment to change one key. A real admin handler should authenticate the operator, authorize the property or cohort scope, and write its own immutable change record before issuing the toggle. The API call is the easy part; preserving intent is the control that helps later.

The vendor decision is an evidence decision

I distrust a comparison table that pretends four products are interchangeable because each has a switch. They are candidates for different operational requirements. This table therefore states a selection test, not a feature-score verdict; verify each candidate's current documentation during procurement because hosted plans and product surfaces change.

Candidate	Sensible reason to shortlist it	Reject or escalate when
Infrai	You need basic admin CRUD and runtime checks through a plain, self-describing REST API, without installing a dedicated SDK	Incident reconstruction depends on provider-side flag audit history, evaluation statistics, dependencies, or pushed client updates
LaunchDarkly	Your evaluation should include a dedicated managed feature-management product	Its documented operating model, governance, or deployment constraints do not fit your organization
Unleash	Your evaluation should include an open-source feature-management product	Your team does not want to own the operational responsibilities of its chosen deployment model
Flagsmith	You want another dedicated feature-management candidate with hosted and self-hosted material to assess	Its documented controls or integration model fail your audit and server-evaluation requirements
Sentry	Error grouping is part of the reconstruction you need alongside flag decisions	You need the flag catalog itself rather than a complementary error-analysis system
Datadog	You want to assess a dedicated observability product as the home for application evidence	The evaluation is limited to a small flag control plane and you already retain adequate telemetry elsewhere
Grafana	You want to assess an observability stack for querying the evidence your application emits	You expect the observability layer to supply the flag-admin workflow

Don't select from that table alone. Build a proof with the same tenant-cohort scenario: change cohort A, capture the operator and reason, render both cohorts server-side, then reconstruct the sequence without looking at anyone's memory or an ephemeral dashboard. Ask each system how history is retained, how evaluation data is exported, how stale clients converge, and how emergency access is reviewed. I'm not sure which governance model fits your organization; its threat model, staffing, and retention duties resolve that question, not a generic ranking.

Infrai is a reasonable fit when the desired system is intentionally small and the application already owns the audit record and experiment analytics. Stick with LaunchDarkly, Unleash, or Flagsmith when a dedicated flag-management workflow is the requirement, then validate the exact audit, targeting, statistics, and update semantics you depend on against the current docs. For a high-change experiment with many operators, treating an application-built audit log as a side task is not suitable.

Polling is a freshness budget, not incident protection

The full catalog makes an admin UI straightforward, but frontend refreshes still require polling. Pick an interval from a stated freshness objective and load budget. A five-second interval, for example, means the UI can remain stale for roughly one interval even when everything behaves correctly; it does not establish an audit trail, and it should not drive request-time evaluation in the browser.

For server-rendered requests, decide whether to read on every request or cache briefly. Per-request reads favor freshness and add a remote dependency to the render path. A bounded cache reduces calls and latency but extends exposure to an old value. Your mileage may vary, especially across properties with different traffic, so write the maximum acceptable staleness down before choosing the cache lifetime. During an emergency disable, that number becomes operational policy.

No dashboard fixes this.

Alerting is a separate boundary too. The capability has no threshold, phone, SMS, or webhook notification route, so a team that needs flag-related alerts must poll a query it can use and send notifications through its own system. It also has no synthetic or heartbeat monitoring; use a Healthchecks-style service when the important signal is that a scheduled comparison job failed to run. Logs can carry trace_id and span_id for correlation, but there is no distributed-trace query or span tree, and there is no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. Those limits matter if the experiment failure mode crosses services or lives mainly in the browser.

A postmortem-ready release rule

Ship this small design only when the application can answer five questions from retained evidence: which operator changed the flag, what reason they gave, which cohort and tenant request received a value, which application release evaluated it, and how long cached or polled readers could remain stale. Test deletion confirmation as carefully as toggle confirmation, because deletion has no recycle bin.

If any answer depends on a screenshot, memory, or the current flag value, stop. Either add an application-owned audit and analytics path or choose a dedicated feature-management system whose verified behavior satisfies the missing requirement. The preventative mechanism is not a prettier admin page. It is a server-side decision record joined to an authorized change record, with an explicit freshness budget.

That is what makes the incident reconstructable.

DEV Community