The Feature Flag That Needed a Restart

I flipped a feature flag in appsettings.json on a running service, curled the endpoint, and got the old behavior back. Flipped it again. Saved harder. Still off. So I did what apparently was tradition on that team: restarted the app, watched the flag come on, and moved on with my day. For months my mental model was "config reload is flaky". The reload was working the entire time. My code was reading the config through an interface that only ever looks once.

One section, four readers

To pin down exactly who notices a config edit, I built the smallest ASP.NET Core app that could answer the question. One Features section, one options class, four endpoints reading it four different ways — three interfaces plus the trap version I'll get to.

var builder = WebApplication.CreateBuilder(args);

builder.Services.Configure<FeatureOptions>(builder.Configuration.GetSection("Features"));
builder.Services.AddSingleton<ExportService>();

var app = builder.Build();

app.MapGet("/options",  (IOptions<FeatureOptions> o)         => Show("IOptions", o.Value));
app.MapGet("/snapshot", (IOptionsSnapshot<FeatureOptions> o) => Show("IOptionsSnapshot", o.Value));
app.MapGet("/monitor",  (IOptionsMonitor<FeatureOptions> o)  => Show("IOptionsMonitor", o.CurrentValue));
app.MapGet("/frozen",   (ExportService s)                    => Show("frozen singleton", s.Frozen));

app.Services.GetRequiredService<IOptionsMonitor<FeatureOptions>>()
    .OnChange(o => Console.WriteLine($"[reload] Features changed: ExportEnabled={o.ExportEnabled}"));

app.Run();

Start it, hit all four endpoints, and everyone agrees:

IOptions           ExportEnabled=False MaxPageSize=50
IOptionsSnapshot   ExportEnabled=False MaxPageSize=50
IOptionsMonitor    ExportEnabled=False MaxPageSize=50
frozen singleton   ExportEnabled=False MaxPageSize=50

Then, with the app still running, I edited appsettings.json — ExportEnabled to true, MaxPageSize to 200 — and curled again:

IOptions           ExportEnabled=False MaxPageSize=50
IOptionsSnapshot   ExportEnabled=True MaxPageSize=200
IOptionsMonitor    ExportEnabled=True MaxPageSize=200
frozen singleton   ExportEnabled=False MaxPageSize=50

Same file, same section, same running process. Two readers saw the change, two never will. That split is the entire topic.

Why IOptions never looks twice

IOptions<T> is a singleton that binds the section on first use and caches the result for the lifetime of the process. No change tokens, no invalidation, nothing. It's not broken; it's just answering a different question — "what was the config when this process warmed up?" My flag bug was exactly this. The reload pipeline delivered the new value and IOptions had no reason to care.

IOptionsSnapshot<T> is scoped, so it re-binds once per request. IOptionsMonitor<T> is a singleton that subscribes to configuration change tokens, keeps CurrentValue up to date, and gives you OnChange. Both picked up my edit without a single line of plumbing, because WebApplication.CreateBuilder already registers appsettings.json with reloadOnChange: true. The machinery you'd think you need to build is on by default; the only decision left is which interface you inject, and that's the one nobody thinks about.

One caveat from the trenches: reload depends on file-change notifications actually arriving. In my container a plain in-place edit triggered OnChange exactly once, which is the polite case — editors that save via rename-and-replace can make it fire twice, and on some mounted or network file systems events never arrive and you need DOTNET_USE_POLLING_FILE_WATCHER=1. Kubernetes ConfigMap symlink swaps are their own adventure. Test the notification path in your real environment before you bet a prod flag flip on it.

The version that actually got me

Here's the part I find sneaky. You can lose the reload while holding the correct interface:

public sealed class ExportService(IOptionsMonitor<FeatureOptions> monitor)
{
    // Right interface, wrong moment: CurrentValue read once, kept forever.
    public FeatureOptions Frozen { get; } = monitor.CurrentValue;
}

That's the /frozen endpoint above, stale forever, with IOptionsMonitor sitting right there in the constructor. Reading CurrentValue at construction time turns a live monitor back into IOptions with extra steps. The fix is boring: keep the monitor in the field and read CurrentValue where you use it, or subscribe with OnChange if you need to react. Code review barely catches this one because the injection looks textbook.

What snapshot's freshness costs

Snapshot re-binds every request, and binding is reflection over your options type. I wanted a number for that, so the sample has a /bench endpoint: 100,000 iterations each of creating an empty scope, creating a scope and resolving a snapshot, and reading monitor.CurrentValue. This is a container, best of three runs, and I care about ratios, not absolutes:

100,000 iterations:
  empty scope               19.0 ms
  scope + snapshot bind    715.7 ms
  monitor.CurrentValue       2.1 ms

So a scope-plus-bind lands around 7 µs while a CurrentValue read is around 20 ns — roughly 300× apart, and the empty-scope row shows it's the binding, not the scope, doing the damage. Honest reading: 7 µs per request for one options type will never show up on your dashboard. Multiply by a dozen options classes injected across every request and it's still small. Pick by lifetime semantics, not this benchmark.

My actual rule, stated as the opinion it is: IOptionsMonitor in singletons and pretty much everywhere else too, IOptionsSnapshot only when a request must see one consistent value from the first middleware to the last log line mid-reload, and plain IOptions when the value is genuinely fixed at boot. And if your config only ever changes at deploy time — env vars, immutable images, the works — skip the whole question. Env var changes don't hot-reload anyway, and IOptions everywhere is the cheapest and most honest description of how your system behaves.

Full runnable sample, live-edit demo included: https://github.com/ssukhpinder/dev-to-code-samples/tree/main/021-options-reload-lifetimes

What's your house rule — snapshot everywhere, monitor everywhere, or restart-and-pretend? And has per-request consistency ever actually saved you mid-reload? Tell me in the comments, because I've never caught it in the act.

— still flipping flags nobody asked me to flip

Comments

[Ivan Rossouw]

IOptionsMonitor solves freshness, but not consistency for an entire operation. A long-running background job can read CurrentValue, reload midway, and combine two configuration versions. I capture it once at the operation boundary and log a configuration version; the next operation receives the update while the one in flight stays coherent. The distributed case adds another trap: one process reloading doesn’t prove every replica has converged. Exposing the effective configuration version per instance makes mixed-fleet behavior observable. At that point, would you keep file reload or move the flags to a release-control system with staged rollout and audit?

[Vadym Arnaut]

The /frozen case has a twin one layer further down, where the value never arrives at all. We set statement_timeout through libpq connect_args and Supavisor in transaction-pooling mode silently drops startup options, so prod ran on the 2min cluster default for months. SHOW statement_timeout was the only thing that told the truth. Same rule as your four endpoints: read the effective value at runtime, never trust that setting it took. Wrote it up here.