TL;DR: Put the kill decision in a small server-side gate, cache the last valid flag snapshot, and make the fallback disable the new import path. For a property-management importer, rollback safety matters more than keeping an experimental parser alive during a control-plane outage. The flag may choose a path; a separate freshness check must detect when scheduled imports stop producing results.
| Choice | Control dependency | Rollback behavior | Best fit |
|---|---|---|---|
| Cached remote flag plus local emergency override | Remote evaluation during normal operation | Reuses the last valid snapshot; local override can force the old path | Default for staged parser rollouts |
| Local configuration only | Deployment system | Requires a config change and usually a process restart or redeploy | Small fleets with infrequent changes |
| Remote flag with no cache | Remote service on every evaluation | Ambiguous when the service is unreachable | Avoid for an import kill switch |
Recommendation: use the first option, but treat the cached value and the import-health signal as different state. The flag answers, "Which parser may run?" The health check answers, "Did the scheduled job produce a credible result recently?" Combining those questions creates a switch that can look healthy while the pipeline is silent.
How should a Node.js backend feature flag kill switch fail?
A scheduled property import can return without throwing and still produce zero usable records. The source file may be empty, a schema change may cause every row to be rejected, or a cursor may stop advancing. An HTTP success counter misses all three.
Safe outage handling starts by separating those failure modes. The Node.js backend owns parser selection and fallback defaults; the scheduler owns cadence; domain validation decides whether the import made progress. Don't let a healthy feature flag request stand in for any of them.
The useful signal is progress. Record the scheduled run ID, source watermark, accepted count, rejected count, completion time, and parser variant. Then alert on the absence of an expected result, not merely on exceptions. Google's four golden signals are a good boundary for the surrounding service: latency, traffic, errors, and saturation. The import adds a domain signal that those four cannot infer: freshness of accepted property data.
For a job expected every 15 minutes, maxSilenceMs is an explicit operating-policy input. It is not a universal constant. Set it from the real schedule, normal completion variance, and alert response time, then test it with delayed and missing runs.
A clean status code proves very little here.
Rollback must work while dependencies fail
The first decision criterion is control independence. A kill switch that needs the failing parser, its datastore, and a live flag network call has the same blast radius as the code it is meant to stop. Keep evaluation in process. Cache a validated snapshot, and give operators a narrowly scoped local override that can only select a known implementation.
Fail closed for the new path. If no valid snapshot has ever loaded, choose the established parser. If a previously valid snapshot becomes stale, fall back to the established parser and emit a state-change event. Never guess from a malformed payload.
This is also where config bloat sneaks in. One rollout flag, one emergency override, and one freshness policy are enough. A flag per building, importer, error class, and retry mode turns incident response into archaeology. Keep targeting data separate from recovery semantics.
The second criterion is reversibility under concurrency. Read the flag once at the start of a run and pin that decision to the run ID. If evaluation changes halfway through a 40,000-row file, mixing parser outputs makes replay and comparison harder. The next run can use the new decision. The current run stays internally consistent.
A small TypeScript control boundary
The interface below keeps vendor code outside the importer. All outcomes are typed, every decision carries a reason, and the local override wins. The numeric values are example policy choices, not measured recommendations.
type ParserVariant = "stable" | "candidate";
type FlagSnapshot = {
candidateEnabled: boolean;
fetchedAtMs: number;
};
type Decision = {
variant: ParserVariant;
reason: "local-kill" | "fresh-snapshot" | "stale-snapshot" | "no-snapshot";
};
interface SnapshotStore {
read(): Promise<FlagSnapshot | null>;
}
interface DecisionSink {
emit(event: { runId: string; decision: Decision; snapshotAgeMs?: number }): void;
}
async function chooseParser(input: {
runId: string;
nowMs: number;
localKill: boolean;
maxSnapshotAgeMs: number;
snapshots: SnapshotStore;
decisions: DecisionSink;
}): Promise<Decision> {
const snapshot = await input.snapshots.read();
const ageMs = snapshot ? input.nowMs - snapshot.fetchedAtMs : undefined;
let decision: Decision;
if (input.localKill) {
decision = { variant: "stable", reason: "local-kill" };
} else if (!snapshot) {
decision = { variant: "stable", reason: "no-snapshot" };
} else if (ageMs! > input.maxSnapshotAgeMs) {
decision = { variant: "stable", reason: "stale-snapshot" };
} else {
decision = {
variant: snapshot.candidateEnabled ? "candidate" : "stable",
reason: "fresh-snapshot"
};
}
input.decisions.emit({ runId: input.runId, decision, snapshotAgeMs: ageMs });
return decision;
}
Evaluate chooseParser once per run. Persist the returned decision beside the output manifest. That gives a replay job enough context to select the same parser without asking a mutable control plane what the flag says now.
The progress detector stays separate:
type ImportResult = {
runId: string;
completedAtMs: number;
accepted: number;
rejected: number;
sourceWatermark: string;
};
function hasStalled(input: {
nowMs: number;
maxSilenceMs: number;
latest: ImportResult | null;
}): boolean {
if (!input.latest) return true;
return input.nowMs - input.latest.completedAtMs > input.maxSilenceMs;
}
Zero accepted rows should not automatically mean failure; a genuinely empty source can be valid. Compare the result with source expectations and watermark movement. That rule belongs in domain validation, where it can be tested, rather than inside the generic flag client.
Measure the switch before trusting it
A rollback path is production code. Benchmark its hot path and drill its cold path. The meaningful measurements are flag-evaluation latency, cache-read failure rate, time from override to new-run selection, and the count of runs pinned to each parser. Report decision reasons as bounded labels. Do not attach property IDs, filenames, or run IDs to metric labels; retain those in logs or manifests instead.
For storage, an analytical database can support high-volume event investigation and aggregations; ClickHouse documents itself as a column-oriented SQL database for analytics. That makes it one possible evidence store, not part of the safety boundary. The importer must still choose a parser if the analytics store is unavailable.
Test the ugly transitions directly:
- Start with no cached snapshot and verify that the stable parser is selected.
- Load a valid candidate snapshot, then make snapshot refresh fail. Verify behavior on both sides of the configured age limit.
- Flip the local kill during a run. Verify that the active run remains pinned and the next run selects the stable parser.
- Complete a run with zero accepted rows and an unchanged watermark. Verify that domain validation marks no progress.
- Replay a stored manifest and verify that it uses the recorded parser decision.
I would reject a design whose kill-switch test needs a live SaaS account. The boundary above is tiny on purpose: an in-memory fake can exercise every branch, while an adapter contract test can cover whichever flag system the deployment uses. Time-to-first-test matters more than a dashboard screenshot.
When local configuration is the better runner-up
Local configuration wins when the fleet is small, changes are rare, and the deployment mechanism can propagate an emergency value inside the rollback objective. It has fewer moving parts. It is also easy to audit through the same change process as application configuration.
Its cost is operational speed and granularity. A restart may interrupt active imports unless the process drains work, and a broad config rollout can affect every property at once. Measure propagation in the actual deployment system; do not infer it from a happy-path test on one instance.
The remote-plus-cache design earns its extra machinery when staged exposure and quick reversal are real requirements. Its limitation is extra state: cached values can be fresh, stale, absent, or malformed. It is not suitable when the team cannot test those transitions or operate the local override. In that case, choose local configuration and accept the slower rollback. This trade-off is cleaner than pretending more machinery is always safer.
Choose the smallest control path that can meet the measured rollback objective while its dependencies are unavailable. Pin each import run to one decision, fail back to the established parser on missing or invalid state, and alert on missing domain progress independently of flag health.
That is the whole contract. Everything else is adapter code.
Top comments (0)