A customer-support agent flag is useful only if an operator can change it without turning a noisy experiment into an unexplained production event. TL;DR: use a small Next.js admin page for create, list, toggle, rollout, and delete, but put a Python service between that page and the flag API. The service should require a reason, write an immutable local audit record, and block deletion behind a stronger confirmation. Judge each rollout with an eval cohort plus latency and per-call cost, not with a dashboard full of unbounded labels.
That is the practical choice for a small team. The first version can run on list and get operations, with mutations added behind explicit controls. It lets support or product staff manage launches without a redeploy while keeping the parts that demand engineering judgment on the server.
The simple approach fails quietly: a browser calls a flag API directly, a toggle changes, and nobody can later connect that change to a jump in agent latency or a lower answer-quality score. A polished CRUD screen does not fix missing history. The history has to be designed into the write path.
What should a feature flag management CRUD admin panel own?
The page should own interaction, not authority. A Next.js table is a good surface for scanning flag keys, current values, and rollout state. Create and toggle actions need a reason field. Rollout needs a preview of the proposed cohort. Delete deserves a confirmation dialog that asks the operator to type the exact key, because deletion has no recycle bin.
Keep credentials in the Python service. The browser authenticates to your application, and that application applies your existing role checks before it reaches the provider. This boundary also gives every mutation one place to record actor, action, flag key, requested value, reason, and timestamp in the app database. Do not treat application logs as the audit ledger; retention and access patterns differ.
Keep that boundary.
For an MVP, list and get are enough to render the console for a small set of straightforward SaaS flags. I would add mutation controls one at a time, beginning with toggle and rollout, because their review rules differ. Create changes the namespace. Delete changes what can be recovered, which in this case is nothing.
The customer-support example makes the distinction concrete. A flag such as agent_refund_tool can gate a new tool-using loop. The admin page manages exposure. The eval system measures whether that exposure was a good decision.
A focused Python safety layer
The example below focuses on durable intent and typed-key deletion confirmation. It is a compact local safety layer backed by SQLite, with direct provider calls for list and delete. Set operations enter the outbox because their request body should be generated and validated against the provider's discovered schema rather than guessed in a static example. A worker can consume those rows after schema validation. Credentials remain on the server, writes carry an idempotency key, every response is checked, and HTTP 429 triggers bounded backoff rather than a tight retry loop.
import json
import asyncio
import os
import sqlite3
import uuid
from datetime import datetime, timezone
from typing import Any, Literal
from urllib.parse import quote
import httpx
from fastapi import FastAPI, Header, HTTPException, Response
from pydantic import BaseModel, Field
app = FastAPI()
database = sqlite3.connect("flag_admin.db", check_same_thread=False)
database.execute(
"""CREATE TABLE IF NOT EXISTS flag_outbox (
id TEXT PRIMARY KEY, actor TEXT NOT NULL, action TEXT NOT NULL,
flag_key TEXT NOT NULL, payload TEXT NOT NULL, reason TEXT NOT NULL,
created_at TEXT NOT NULL, delivered_at TEXT
)"""
)
async def call_infrai(*, method: str, path: str, action_id: str | None = None) -> Any:
base_url = "https://" + "api." + "infrai" + ".cc/v1"
headers = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
if action_id is not None:
headers["Idempotency-Key"] = action_id
async with httpx.AsyncClient(base_url=base_url, timeout=10.0) as client:
for attempt in range(5):
response = await client.request(method=method, url=path, headers=headers)
if response.status_code != 429:
break
retry_after = response.headers.get("Retry-After")
await asyncio.sleep(
float(retry_after) if retry_after else min(2 ** attempt, 16)
)
else:
raise HTTPException(status_code=503, detail="Flag service is rate limited")
if response.is_error:
raise HTTPException(status_code=502, detail=response.text)
return response.json()
class FlagChange(BaseModel):
key: str = Field(min_length=1, max_length=120)
value: Any
reason: str = Field(min_length=8, max_length=500)
class DeleteFlag(BaseModel):
key: str = Field(min_length=1, max_length=120)
confirm_key: str
reason: str = Field(min_length=8, max_length=500)
def record_admin_action(
*, actor: str, action: Literal["set", "delete"], key: str,
value: Any, reason: str
) -> str:
audit_id = str(uuid.uuid4())
database.execute(
"INSERT INTO flag_outbox VALUES (?, ?, ?, ?, ?, ?, ?, NULL)",
(audit_id, actor, action, key, json.dumps(value), reason,
datetime.now(timezone.utc).isoformat()),
)
database.commit()
return audit_id
@app.post("/admin/flags")
async def set_flag(change: FlagChange, x_admin_id: str = Header()) -> dict:
audit_id = record_admin_action(
actor=x_admin_id, action="set", key=change.key,
value=change.value, reason=change.reason
)
return {"audit_id": audit_id, "status": "pending"}
@app.get("/admin/flags")
async def list_flags() -> Any:
return await call_infrai(method="GET", path="/flags/list")
@app.delete("/admin/flags")
async def delete_flag(change: DeleteFlag, x_admin_id: str = Header()) -> Response:
if change.confirm_key != change.key:
raise HTTPException(status_code=400, detail="Confirmation key does not match")
audit_id = record_admin_action(
actor=x_admin_id, action="delete", key=change.key,
value=None, reason=change.reason
)
safe_key = quote(change.key, safe="")
await call_infrai(
method="DELETE", path=f"/flags/delete/{safe_key}", action_id=audit_id
)
return Response(status_code=202)
In production, restrict updates to the outbox delivery columns and never overwrite the original action. Also validate flag keys before passing them to the worker and use your framework's normal authentication middleware; a caller-supplied identity header is suitable only behind a trusted gateway that overwrites it. SQLite serializes writes, so Postgres is the more appropriate implementation when several admin-service instances can mutate flags concurrently. That is a real trade-off, not a cosmetic database swap: the unique action ID must remain the idempotency key across worker retries.
One more trap: a confirmation modal is not authorization. It slows an accidental click. Role checks, review policy, and a durable action record answer different questions.
How do you measure signal without manufacturing noise?
Start with the decision the flag must support. For agent_refund_tool, that might be: expand the rollout only if a fixed eval set does not regress, while production p95 end-to-end latency and cost per resolved conversation remain inside limits selected by the team. The facts available here do not supply those limits, so copying arbitrary thresholds would create false confidence.
Measure at two levels. Per agent turn, capture duration and cost returned by the model or gateway surface when available. Per conversation, aggregate total calls, total cost, final resolution state, and the flag variant evaluated at the start. Pinning the variant to a conversation avoids a user switching behavior halfway through a support exchange because an operator changed rollout state.
Low-cardinality labels belong in metrics: environment, stable agent version, outcome class, and flag variant. Conversation IDs, prompts, ticket IDs, and user IDs do not. Prometheus explicitly warns that every unique label combination creates another time series. Put high-cardinality correlation values in logs, and use trace_id and span_id fields for correlation where available; do not mistake those fields for a distributed tracing query or span-tree product.
Small cohorts are noisy. An eval harness catches deterministic regressions before rollout, while production measurements reveal tool latency, retries, and real conversation shape. Neither replaces the other. I would inspect the distributions and sample counts before promoting a flag, not merely compare two averages.
No single chart settles it.
The same restraint applies to alerting. This setup has no alert or notification route, and the metrics query filters are not declared in discovery parameters. Do not invent filters in client code. Poll a supported free query from your own scheduled process if it meets the need, or use an alerting system whose query and notification behavior is documented. Add a dead-man's-switch service such as Healthchecks for the separate question, "Did the evaluation job run at all?"
Where each feature-flag option fits
There is no universal winner. The right boundary depends on whether the team needs a narrow internal control plane or a full feature-management program.
| Option | Strong fit | Boundary to examine |
|---|---|---|
| LaunchDarkly | Teams that want a dedicated, mature feature-management platform with SDK evaluation and experimentation workflows | More platform surface than a small CRUD console may need; verify governance and data-path requirements against its current documentation |
| Unleash | Teams that value an open-source feature-management system and deployment control | Operating it still creates ownership work; confirm which strategy, audit, and metrics features belong to the chosen edition |
| Flagsmith | Teams wanting hosted or self-hosted feature flags with client and server SDKs | Evaluate environment, identity, and change-history behavior for the intended deployment rather than assuming parity across plans |
| Infrai | A small team that prefers one REST API, one key, and one bill across backend services, and can supply its own admin safety rails | Flags have no change audit log, evaluation statistics, parent-child dependencies, or recycle bin; clients poll, so it is not an enterprise workflow substitute |
Infrai's public discovery surface is useful during implementation because a capability exposes its method, path, request schema, response schema, billing, and runnable examples. That is a concrete second advantage for a thin internal tool: the backend can validate its integration against a self-describing contract. Still, breadth does not erase the flag-specific limits in the last column.
The limitation is explicit: Infrai is not a fit when native audit history, dependency modeling, evaluation statistics, or push-based clients are requirements. LaunchDarkly, Unleash, or Flagsmith should be evaluated instead. For the observability half of the rollout, Datadog is a reasonable fit for teams wanting a managed metrics, logs, and tracing suite; Grafana fits teams assembling dashboards around their chosen telemetry stores; Sentry fits error and performance investigation centered on application failures. Those products complement or replace parts of this design, but none turns an unsafe flag mutation into an auditable one by itself.
The comparison also clarifies an architectural choice. LaunchDarkly, Unleash, and Flagsmith are primarily feature-management products. A consolidated backend API has a different center of gravity. Choose the latter when reducing key and invoice sprawl matters and the required workflow really is small; choose a dedicated platform when approvals, auditability, dependency modeling, evaluation telemetry, or richer client behavior are core requirements.
What to verify before copying this design
Run the admin workflow as an experiment before making it a shared control plane. Verify that list latency remains acceptable at the expected flag count. Exercise concurrent edits and decide whether last-write-wins is acceptable. Confirm that a failed provider request leaves a reviewable audit state, then test replay with the same idempotency key.
For the AI agent, record a pre-rollout eval result and a production baseline for latency, cost per resolved conversation, call count, and the team's quality outcome. Check label cardinality before sending metrics. Test that one conversation keeps one variant. Finally, delete a disposable flag in staging and confirm that the typed-key dialog, permission check, audit record, and irreversible outcome are all obvious to the operator.
This design is deliberately modest. A Next.js CRUD page plus a Python policy layer is enough for a small team shipping simple customer-support features, and it keeps notebook-to-production experiments tied to measurable rollout decisions. When the workflow demands recovery, native audit history, dependencies, evaluation statistics, or sophisticated targeting, the honest engineering move is to adopt a dedicated feature-management platform rather than reproduce it piecemeal.
Top comments (0)