Short answer: treat a missing feature flag key as an incident-reconstruction problem, not as permission to recreate the key. Keep the last evaluated value, the configured fallback, the lookup outcome, and the configuration revision in one structured decision record. For a healthtech notification service, default to the behavior whose delivery risk is already understood, then verify whether the key was deleted, renamed, or queried in the wrong environment before changing control-plane state.
The decision rule is blunt: do not repair a 404 by immediately creating an object with the same name. A recreated key may look identical to an operator while representing a new lifecycle. It also destroys the cleanest clue in the timeline: the old key stopped existing before a client attempted evaluation. Preserve that boundary first.
How should feature flags handle a missing key after delete and recreate?
A notification worker usually has two relevant histories. The configuration system records what definitions existed; the worker records what it actually decided for a delivery. Those histories move on different clocks. A lookup can fail after a delete while one process still holds an earlier value and another has already fallen back. If the key is recreated during triage, responders now have three states to distinguish: the former definition, the interval in which it was absent, and the new definition.
Same spelling is weak evidence.
For incident reconstruction, log an immutable evaluation event at the point where the notification path branches. It should answer: which tenant-safe subject was evaluated, which logical key was requested, which environment and revision were visible, whether the value came from a current evaluation, cache, or fallback, and which notification outcome followed. Do not put patient data, message bodies, email addresses, phone numbers, or raw user identifiers into that record. A stable pseudonymous correlation value is enough to join the evaluation to the delivery attempt under the service's retention and access rules.
The useful distinction is between transport status and application decision. A 404 or not-found result explains the lookup outcome. It does not explain whether the service sent, suppressed, queued, or retried a notification. Record both, connected by a correlation ID.
This method has a limitation: decision events increase log volume, and longer retention increases both storage and privacy exposure. The trade-off is deliberate. Keep a compact event for the incident window, restrict access, and expire it under the service's data policy rather than retaining full notification payloads. You can't reconstruct detail that was never recorded, but collecting everything creates a different operational risk.
| Evidence | Question it answers | Keep cardinality bounded by |
|---|---|---|
| Evaluation outcome | Did the lookup resolve, use cache, or use fallback? | A small outcome enum |
| Configuration revision | What state did the worker observe? | Revision identifier |
| Decision value | Which branch ran? | Boolean or compact variant |
| Delivery outcome | What happened to the notification? | A small status enum |
| Correlation value | Which events belong to one attempt? | Logs or traces, not metric labels |
Capture the decision before attempting repair
Put the instrumentation around a generic evaluator rather than around one provider's client. This keeps the operational contract stable if the backing implementation changes. The wrapper below makes the fallback explicit and emits one record for every branch. Its evaluator interface deliberately returns metadata with the value; without a revision and source, a successful lookup is much less useful during a postmortem.
package flags
import (
"context"
"errors"
)
var ErrKeyNotFound = errors.New("feature flag key not found")
type Evaluation struct {
Value bool
Revision string
Source string // current, cache, or fallback
}
type Evaluator interface {
Bool(ctx context.Context, key string, subject string) (Evaluation, error)
}
type Recorder interface {
Record(ctx context.Context, event DecisionEvent)
}
type DecisionEvent struct {
CorrelationID string
Key string
SubjectHash string
Revision string
Source string
Outcome string
Value bool
}
func EvaluateNotification(
ctx context.Context,
evaluator Evaluator,
recorder Recorder,
correlationID string,
key string,
subjectHash string,
fallback bool,
) (bool, error) {
result, err := evaluator.Bool(ctx, key, subjectHash)
if err == nil {
recorder.Record(ctx, DecisionEvent{
CorrelationID: correlationID, Key: key, SubjectHash: subjectHash,
Revision: result.Revision, Source: result.Source,
Outcome: "resolved", Value: result.Value,
})
return result.Value, nil
}
if errors.Is(err, ErrKeyNotFound) {
recorder.Record(ctx, DecisionEvent{
CorrelationID: correlationID, Key: key, SubjectHash: subjectHash,
Source: "fallback", Outcome: "key_not_found", Value: fallback,
})
return fallback, nil
}
recorder.Record(ctx, DecisionEvent{
CorrelationID: correlationID, Key: key, SubjectHash: subjectHash,
Outcome: "evaluation_error", Value: fallback,
})
return fallback, err
}
The important choice is not that false is always safe. It is not. Suppressing a marketing reminder and suppressing a medication-related notification carry different consequences. Select the fallback per notification class during design review, store it in version-controlled service configuration, and test it as part of the application contract. The evaluator receives that value; it does not invent one after an error.
Also separate a missing key from a transient evaluation error. The sample permits the caller to continue with the declared fallback for a missing key, but returns other errors so the caller can apply its normal queue policy. That prevents an unavailable dependency from masquerading as an intentional configuration deletion.
Reconstruct the timeline, then choose an action
Start with the affected delivery attempt, not the configuration dashboard. Find its correlation value and arrange the worker's evaluation record, queue transition, and delivery result by their recorded timestamps. Then compare the observed revision with the configuration change history. Clock skew can make wall-clock ordering ambiguous, so revision identifiers and queue attempt numbers carry more weight than two timestamps separated by a tiny interval.
Next, establish scope. Is key_not_found limited to one environment, one deployed version, or every worker? A single version points toward a stale key name in that release. One environment points toward configuration drift or an environment-selection error. A clean change boundary across workers is consistent with deletion propagating through the fleet. These are diagnostic hypotheses, not automatic verdicts; confirm them against deployment and configuration records.
Only then pick a recovery:
- If callers still reference a deliberately retired key, roll back or patch the caller according to the release plan. Recreating the definition preserves the stale dependency.
- If the deletion was unintended and the prior definition is available from an audited revision, restore through the normal reviewed change path. Give the restoration its own revision and incident annotation.
- If the worker queried the wrong environment or key, correct that mapping and deploy it. Do not create a duplicate definition to make the bad query succeed.
- If evidence is incomplete, hold the declared fallback and preserve queue state while responders resolve the uncertainty. A speculative configuration edit adds another variable.
This is where the runbook earns its keep. It turns "make the error disappear" into "explain the decision, contain its effect, and restore a known state."
Verify recovery without erasing the failure boundary
Test the missing-key path before production. The useful test deletes the evaluator from the equation entirely: inject ErrKeyNotFound, assert the exact fallback, and assert the recorded source and outcome. A second test should return an unrelated error and verify that it remains visible to the caller.
package flags
import (
"context"
"testing"
)
type missingEvaluator struct{}
func (missingEvaluator) Bool(context.Context, string, string) (Evaluation, error) {
return Evaluation{}, ErrKeyNotFound
}
type memoryRecorder struct {
events []DecisionEvent
}
func (r *memoryRecorder) Record(_ context.Context, event DecisionEvent) {
r.events = append(r.events, event)
}
func TestMissingKeyUsesDeclaredFallback(t *testing.T) {
recorder := &memoryRecorder{}
value, err := EvaluateNotification(
context.Background(), missingEvaluator{}, recorder,
"attempt-42", "clinical-reminder-enabled", "subject-hash", true,
)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if !value {
t.Fatal("expected the declared fallback")
}
if len(recorder.events) != 1 || recorder.events[0].Outcome != "key_not_found" {
t.Fatal("expected one missing-key decision event")
}
}
After a repair, verify both planes. The control plane should show the reviewed revision you intended, while fresh worker events should show that same revision and resolved rather than key_not_found. Then check the downstream delivery outcomes for the affected notification class. A green configuration screen alone proves too little.
Use metrics for rates, logs or traces for individual reconstruction. Count missing-key outcomes by service, environment, and logical key only if that label set is controlled; keep correlation values and subject hashes out of metric labels. Alert on a sustained change from the service's established baseline, and link the alert to a query that exposes representative decision records. The exact threshold must come from normal traffic and the notification class's risk, not a universal number.
Roll back by revision, not by memory
A rollback plan needs two independent levers: the application release that contains the key reference and the configuration revision that defines its behavior. Record both in the incident timeline. If recovery changes either one, predefine the condition that sends it back, such as renewed missing-key outcomes, an unexpected shift in send versus suppress decisions, or a delivery-failure increase for the affected class.
Keep the absence interval visible after recovery. Do not rewrite old events to associate them with the recreated definition, and do not collapse old and new revisions into one dashboard series. The gap is evidence. It explains why two notification attempts near the same deployment could take different branches.
Once stable, add a deployment check that enumerates the application's required logical keys against the target environment through the same generic contract used in testing. The check should report mismatch and stop the rollout before traffic shifts; it should not create missing definitions. Pair that with a retirement sequence: remove call sites, confirm no evaluations for the key across the agreed observation window, then delete the definition. The duration of that window is an operational choice tied to traffic cycles and rollback policy.
The postmortem action is therefore concrete: make every notification decision reconstructable across application version, configuration revision, evaluation source, and delivery result. Defaults contain impact. Evidence tells you what to fix.
Top comments (0)