DEV Community

Cover image for Shipping an AI Agent Behind a Feature Flag Without Flying Blind
Gabriel Anhaia
Gabriel Anhaia

Posted on

Shipping an AI Agent Behind a Feature Flag Without Flying Blind

#ai #typescript #agents #node

Flagging a normal feature is simple: on for 1% of users, watch the error rate,
ramp. Errors are binary, so the signal is clean.

An agent breaks that. Almost nothing throws. A wrong answer, a tool called
with plausible-but-wrong arguments, an agent that gave up early — all of it
returns 200 OK. Ramping on error rate means ramping on a number that will
stay flat while the feature is quietly bad.

So the flag needs to control more than a boolean, and the ramp needs signals
that a healthy-looking 200 cannot hide behind.

Make the flag a configuration, not a switch 

export type AgentFlag = {
  enabled: boolean;
  mode: "off" | "shadow" | "suggest" | "auto";
  maxCostUsdPerRun: number;
  allowedTools: readonly string[];
  fallback: "legacy" | "error";
};

export async function flagFor(user: User): Promise<AgentFlag> {
  return flags.evaluate<AgentFlag>("agent-v1", {
    userId: user.id,
    plan: user.plan,
    locale: user.locale,
  });
}
Enter fullscreen mode Exit fullscreen mode

mode is the part that earns its keep, because it gives you a ladder rather
than a cliff:

  • shadow — the agent runs, output is stored and never shown. Real traffic, zero user risk, real cost.
  • suggest — output is shown as a proposal a human accepts or edits.
  • auto — the agent acts.

Most of the learning happens in shadow and suggest. Going from off to
auto because "it worked in staging" skips both.

Shadow mode is where the surprises are 

const flag = await flagFor(user);
const legacy = await legacyPath(input);

if (flag.mode === "shadow") {
  void runAgent(input, ctx)
    .then((out) => shadowStore.record({
      runId: ctx.runId,
      legacy: legacy.result,
      agent: out.result,
      agreed: compare(legacy.result, out.result),
      costUsd: out.costUsd,
      latencyMs: out.latencyMs,
    }))
    .catch((e) => logger.warn("shadow failed", { err: String(e) }));

  return legacy;                    // user never sees the agent
}
Enter fullscreen mode Exit fullscreen mode

Two details. void and a .catch — the shadow run must never affect the
response, including by throwing. And storing both outputs, because the
useful metric is the disagreement rate, not the agent's output in isolation.

Where they disagree is your review queue. Twenty disagreements read by hand
tells you more than any aggregate.

Budget for it: shadow mode costs full price and returns nothing to users. Cap
the sample rather than shadowing everything.

Shadow, suggest and auto as rungs on a ladder, with the shadow path<br> discarded before the<br> response.

Ramp on quality signals, not on the error rate

Four numbers worth gating on, none of which is 5xx:

metrics.increment("agent.outcome", 1, { outcome, mode: flag.mode });
metrics.histogram("agent.cost_usd", costUsd, { mode: flag.mode });
metrics.increment("agent.tool_error", 1, { tool, kind });
metrics.increment("agent.user_action", 1, { action });   // accepted | edited | discarded
Enter fullscreen mode Exit fullscreen mode

Outcome distribution — the share of runs ending in complete versus
gave_up, budget_exceeded, max_turns. A rising max_turns share is the
earliest sign of degradation I know of.

Cost p95, not the mean. The mean hides a tail of runs looping through
twenty turns.

Tool error rate by tool — one tool at 30% failure is a broken integration,
and it will look like "the AI is bad".

User action — in suggest mode you get this free. The share of proposals
edited before acceptance is the closest thing to a quality metric you will get
without human graders.

Automate the rollback, because nobody watches a dashboard at 3am 

export async function guardrail() {
  const w = await metrics.window("15m", { mode: "auto" });

  const trips = [
    w.rate("agent.outcome", { outcome: "gave_up" }) > 0.15,
    w.p95("agent.cost_usd") > 0.50,
    w.rate("agent.user_action", { action: "discarded" }) > 0.30,
    w.rate("agent.tool_error") > 0.10,
  ];

  if (trips.some(Boolean)) {
    await flags.update("agent-v1", { mode: "suggest" });   // one rung down
    await pager.notify("agent-v1 demoted to suggest", { trips });
  }
}
Enter fullscreen mode Exit fullscreen mode

Demoting one rung rather than switching off entirely. suggest keeps the
feature alive with a human in front of it, which is usually the right response
to "quality dropped", and it keeps producing the data you need to diagnose it.

Volume-gate the check so ten requests at 4am cannot trip it.

Give the flag a fallback and test that path 

try {
  return await withTimeout(runAgent(input, ctx), 30_000);
} catch (err) {
  metrics.increment("agent.fallback", 1, { reason: kindOf(err) });
  if (flag.fallback === "legacy") return legacyPath(input);
  throw err;
}
Enter fullscreen mode Exit fullscreen mode

Keeping the legacy path alive during rollout is what makes the flag a real
control. The failure I have seen most often is deleting it at 50% because
"it's working", which turns the last half of the ramp into a one-way door.

Test the fallback in CI. A path only exercised during an incident is a path
that breaks during an incident.

The ramp that works

  1. Internal only, auto. Not a sample — your own team, hitting it daily.
  2. 1% shadow. Read 20 disagreements by hand. This is the step people skip and the one that finds the systematic problems.
  3. 5% suggest. Watch edit rate. Above ~40% and the output is not good enough for auto.
  4. 5% auto, guardrails armed. Hold for a week, enough to see a weekday pattern and at least one deploy.
  5. 25% → 50% → 100%, holding at least a day at each and watching cost p95 as closely as quality.

The pace is set by how long it takes a problem to become visible. For a
feature whose failures are silent, that is days, not hours.

A staged ramp with quality gates at each rung and an automatic demotion<br> path.

Pin the model in the flag 

export type AgentFlag = {
  // ...
  model: string;
  promptVersion: string;
};
Enter fullscreen mode Exit fullscreen mode

Rolling out a feature and a model change together makes attribution
impossible. Putting both in the flag payload means every stored run records
which combination produced it, and a regression can be traced to one variable
rather than argued about.

If this was useful

AI That Ships covers rollout for AI
features — shadow and suggest modes, the metrics that actually move before
users complain, automated guardrails, and fallback paths that still work when
you need them.

AI That Ships — Taking AI Features to Production

The full series is at
xgabriel.com/ai-in-typescript.

Top comments (0)