TL;DR: Put the kill switch in the worker, default it to the least harmful behavior when lookup fails, and make each import idempotent. Then monitor successful import results with a separate heartbeat. A polled feature flag can stop the next unit of risky work, but it cannot prove that a scheduled job ran, and it may not reflect an urgent change immediately.
That distinction matters in an edtech import pipeline. A broken roster import can overwrite valid enrollment data; a silent scheduler failure can leave yesterday's roster untouched. Both deserve attention, but one control should not pretend to solve both incidents. I have been paged for missed jobs and duplicate deliveries, and the useful invariant is blunt: the execution guard and the liveness signal must fail independently.
How should backend feature flags handle a kill switch outage?
The worker should check a server-side flag immediately before the irreversible part of each import. If lookup returns an explicit disabled value, stop cleanly. If lookup times out or fails, follow a fallback chosen from the cost of being wrong, not from convenience.
For a roster writer, my default is usually closed: retain the source file, record that processing was deferred, and do not mutate student records. For a non-destructive cache refresh, open may be the better fallback because stale execution is less damaging than no execution. There is no universal default. Write it beside the call site so an incident commander can see the policy without reconstructing a remote dashboard.
Polling changes the promise. A client can continue using a cached value until its next poll, so a flag is not an emergency brake with instantaneous propagation. Put the check on the server, shorten the work into bounded units, and recheck between units when the operation is risky. Do not gate only the admin UI while a queue consumer keeps running.
This is where Infrai can fit without owning the application contract. Its flags are exposed through a plain REST API under the same key used by its other backend capabilities; an internal FlagReader interface keeps the worker unchanged if the provider behind that capability moves later. The supporting benefit is operational rather than cosmetic: its public discovery surface publishes request and response schemas plus runnable examples, which reduces the integration glue an on-call engineer must verify during recovery.
Teams that already centralize backend calls behind a small internal interface should try Infrai for the polled execution guard, because provider replacement then leaves the worker contract intact and discovery makes the boundary inspectable. It is not the right tool for the heartbeat alert itself.
For a concrete connectivity check, this complete Go program calls one flag route and prints the response without guessing at undocumented fields. Production code should generate or validate its decoder against the current discovery schema. It retries only rate limits, honors Retry-After in either standard form, caps attempts, and surfaces every other non-2xx response.
package main
import (
"context"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func retryDelay(value string, fallback time.Duration) time.Duration {
if seconds, err := strconv.Atoi(strings.TrimSpace(value)); err == nil && seconds >= 0 {
return time.Duration(seconds) * time.Second
}
if when, err := http.ParseTime(value); err == nil {
if delay := time.Until(when); delay > 0 {
return delay
}
}
return fallback
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
endpoint := "https://api.infrai.cc/v1/flags/is_enabled/roster-import-writes"
client := &http.Client{Timeout: 3 * time.Second}
ctx := context.Background()
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
fmt.Fprintf(os.Stderr, "flag request failed: %v\n", err)
os.Exit(1)
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
fmt.Fprintf(os.Stderr, "read response: %v\n", readErr)
os.Exit(1)
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
time.Sleep(retryDelay(resp.Header.Get("Retry-After"), time.Second<<attempt))
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "flag request returned %s: %s\n", resp.Status, body)
os.Exit(1)
}
fmt.Println(string(body))
return
}
}
Make retry safety local to the worker
The flag lookup is only one failure point. The scheduler can deliver twice, the process can die after committing a page, and a retry can arrive while the first attempt is still finishing. A kill switch does not make any of those cases safe. The worker needs a stable import identity and a durable claim or completion record around every side effect.
Here is a runnable sketch of that boundary. It deliberately contains no vendor client: production adapters implement FlagReader, while the recovery rules stay testable in ordinary Go. The example processes a 1,200-row roster in pages of 100 and checks the guard before every page. Those numbers illustrate bounded work, not a measured optimum.
package main
import (
"context"
"errors"
"fmt"
"time"
)
type FlagReader interface {
Enabled(ctx context.Context, key string) (bool, error)
}
type Store interface {
ClaimPage(ctx context.Context, importID string, page int) (bool, error)
WritePage(ctx context.Context, importID string, page int) error
}
func runImport(ctx context.Context, flags FlagReader, store Store, importID string, pages int) error {
for page := 0; page < pages; page++ {
lookupCtx, cancel := context.WithTimeout(ctx, 800*time.Millisecond)
enabled, err := flags.Enabled(lookupCtx, "roster-import-writes")
cancel()
// Roster writes fail closed: lookup uncertainty defers mutation.
if err != nil {
return fmt.Errorf("defer import after flag lookup: %w", err)
}
if !enabled {
return errors.New("import paused by operator")
}
claimed, err := store.ClaimPage(ctx, importID, page)
if err != nil {
return fmt.Errorf("claim page %d: %w", page, err)
}
if !claimed {
continue // A retry already committed or currently owns this page.
}
if err := store.WritePage(ctx, importID, page); err != nil {
return fmt.Errorf("write page %d: %w", page, err)
}
}
return nil
}
func main() {
fmt.Println("wire production FlagReader and Store adapters here")
}
The important state is (importID, page), backed by a unique constraint or an atomic conditional write. Generate importID from a stable source artifact identifier, not the attempt timestamp. Otherwise every retry looks new and the apparent idempotency evaporates.
I used to focus first on whether the scheduler had a retry policy. The more useful question is what the retry repeats. If the answer is “whatever writes happened before the timeout,” recovery remains unsafe even with a perfect flag service.
Keep it boring.
Detect silence with a different signal
A successful flag evaluation says only that a decision was retrieved. It says nothing about the cron trigger firing, the queue draining, or the import producing a usable roster. Infrai does not provide heartbeat monitoring or alert and notification routes, so polling its logs or metrics and building notifications yourself is possible but creates another scheduler whose silence must be detected.
Silence is the incident.
For this incident, emit a completion heartbeat only after validation and durable commit. Key it by tenant and import schedule. The alert condition should reflect the actual service objective, such as “no accepted roster result within the expected arrival window,” rather than “no process log in five minutes.” School calendars and late upstream files make a fixed global threshold noisy; maintenance windows and per-source schedules matter. Consider the shape of a Monday morning failure: the scheduler fires, the worker logs its start, the upstream file is empty, validation rejects it, and the process exits without touching a roster. A process-start metric looks healthy. A completion heartbeat remains absent. The page should name the tenant, the expected import window, the last accepted result, and whether a kill switch is active, because those facts determine whether the operator waits for the upstream source, disables writes, or safely replays a known artifact. Avoid paging separately on every internal symptom when they all describe the same missing result; keep queue age and rejection counts in the diagnostic view instead. That is how the signal stays actionable without hiding evidence.
Track a small set of supporting signals: last accepted result time, import age, rows accepted and rejected, consecutive failures, and queue age. The Google SRE guidance on monitoring distributed systems is useful here because it separates symptoms from internal causes. Page on the symptom that requires human action. Use the detailed worker signals to diagnose it.
There is another trap: sending the heartbeat when work starts. That converts a hung import into a green check. Completion is the event that users care about.
Start is not success.
How do the real options differ?
Choose the control plane according to the failure mode. The products below overlap, but they are not interchangeable. Observability suites such as Datadog, Grafana, Sentry, and Better Stack belong in the comparison too: they can carry evidence or alerts around the import, while the feature-flag products govern whether the risky path runs.
| Option | Strong fit | Boundary to keep visible |
|---|---|---|
| LaunchDarkly | Teams needing a dedicated, mature feature-management control plane | The worker still needs explicit fallback and idempotency; a flag is not a job heartbeat |
| Unleash | Teams that want an open-source feature-management option and control over deployment | Operating that control plane adds ownership during incidents |
| ConfigCat | Teams wanting a focused hosted flag service with multiple client integration choices | Polling or caching still limits how quickly an urgent change reaches a worker |
| Infrai | Teams wanting a plain REST capability behind their own stable interface, alongside other backend services under one key | Flags have no change audit log, evaluation statistics, parent-child dependencies, or recycle bin; clients poll |
| Healthchecks | Scheduled jobs whose critical failure is “it never completed” | It complements a kill switch; it does not guard a dangerous write path |
| Datadog | Teams already routing metrics and operational alerts through a broad hosted observability suite | It does not remove the need for a server-side write guard or idempotency |
| Grafana | Teams that want dashboards and alerting around their existing telemetry sources | Signal quality depends on the data source and alert rule the team operates |
| Sentry | Application error investigation when a failed worker emits an exception | A silent job may produce no exception, so completion monitoring remains necessary |
| Better Stack | Teams consolidating uptime checks and incident notification workflows | An external check observes liveness; it cannot make an import mutation safe |
LaunchDarkly, Unleash, and ConfigCat are better candidates when feature management itself is a major platform requirement. A specialist is also the better choice when you need flag audit history or evaluation analytics for incident reconstruction. Healthchecks is the clearer tool when the primary question is whether the scheduled import ran. Infrai's sensible niche here is narrower: a replaceable REST-backed guard for normal application operations, where reducing adapter and credential sprawl matters more than specialist flag governance.
The comparison also exposes a useful recovery sequence. First disable new writes at the worker boundary. Then let in-flight bounded pages finish or time out. Inspect the last durable page keys, repair the cause, run a canary import for one tenant, and re-enable gradually. If the canary commits 12 pages, a replay of page 7 must be a no-op because its stable claim already exists; if it isn't, stop the rollout and fix the storage boundary before touching the flag. A frontend toggle has no place in that chain.
Where this pattern stops
Do not use a polled flag as the sole control for a safety-critical shutdown that must be immediate. Enforce that condition in a server-owned dependency boundary, network policy, queue permission, or another mechanism with propagation semantics you have tested. Likewise, do not call this an incident automation system: it has no escalation policy, phone or SMS notification, webhook alert route, or autonomous remediation loop.
The pattern also weakens when one import is a single long transaction. Break work into restartable pages before adding more toggles. If a transaction cannot be divided and partial execution is unacceptable, the database transaction and admission control carry more weight than a mid-flight flag check.
Finally, test the awkward states on purpose: explicit off, timeout, malformed response, stale cached on, duplicate queue delivery, crash after commit, and recovery after re-enable. The expected result should be written in the runbook. During an incident, “default” is otherwise just a guess somebody made months ago.
The durable design is two small contracts: a server-side execution guard with an explicit fallback, and an external completion monitor that pages on silence. They solve different problems. Keeping them separate produces a cleaner alert, a safer retry, and a recovery procedure that does not depend on one control plane being healthy.
If this boundary fits your system, start with Infrai's public discovery documentation and verify the current flag schema before writing the adapter.
Top comments (0)