DEV Community

FinnianFox8297
FinnianFox8297

Posted on

Media Pricing Feature Flags: Idempotency for Retries and Duplicate Writes

#observability #featureflags #sre

Ship a pricing feature flag only after idempotency makes retries converge on one result instead of creating duplicate writes, and expose that operation identity in traces, logs, and cost records. The deciding constraint is attribution: if a media purchase can be charged twice while the telemetry shows two unrelated requests, nobody can prove whether the new rule, the retry policy, or a downstream timeout caused the extra spend.

TL;DR: evaluate the flag once, create an immutable rollout operation, and retry that operation rather than reevaluating the flag or inventing a new request. Persist the idempotency result beside the business write. Record the flag decision, rule version, cohort, operation ID, and cost dimensions on the same trace. Then verify both business uniqueness and attributed cost before increasing exposure.

How can feature flag retries cause duplicate writes?

The dangerous case is an ambiguous outcome. A scheduler sends a request to enable a pricing rule for a campaign cohort. The backend commits the change, but the connection closes before the response reaches the caller. From the scheduler's point of view, the request failed. From the database's point of view, it worked.

A retry is reasonable. A fresh write is not.

HTTP defines PUT as idempotent, but a verb does not make application side effects safe by itself. A toggle endpoint implemented as "append a rollout event, update the active rule, and enqueue repricing" can repeat all three effects after a timeout. POST can also be made retry-safe when the application supplies a stable operation key and enforces it transactionally. The useful question is therefore not which verb appears in the route. It is whether two deliveries of the same intent converge on one durable result.

Flag evaluation creates another trap. Suppose the first attempt evaluates a 10% cohort and commits. Before the retry, exposure moves to 25%. Reevaluating on the second attempt can attach a different decision to the same logical action. Now an incident query cannot distinguish a retry from a new rollout step, and cost attribution inherits that ambiguity.

Treat these as separate identities:

  • rollout_id: the planned change, such as pricing-v4-publisher-a-wave-03.
  • operation_id: one requested mutation within that rollout.
  • attempt: a delivery count used for diagnosis, never for uniqueness.
  • rule_version: the immutable pricing logic selected by the flag decision.

The operation ID stays fixed across transport retries. The attempt number changes. Mixing those two fields is a common way to make a dashboard look healthy while duplicate work accumulates underneath it.

Make the mutation replay the original result

The backend needs one transaction boundary around the idempotency claim and the business mutation. A separate cache check leaves a race: two workers can both observe a miss and both write. A database uniqueness constraint on the operation identity turns that race into a deterministic branch.

This design has a real limitation: it adds a durable write and retained result for every mutation. It is a poor fit for read-only flag evaluations or telemetry events where occasional duplication is explicitly acceptable and no external effect occurs. In those paths, a stable event identifier plus downstream deduplication may be the better trade-off. For pricing changes and charges, however, the extra coordination belongs on the correctness path because cleanup after a duplicate external effect is harder than preventing it.

This Go sketch keeps the interface generic and makes the required behavior explicit. The repository implementation should use a transaction and a unique key on (scope, operation_id). Stored results need a retention period at least as long as the maximum retry and redelivery window; choosing the actual duration is an operational decision, not a universal constant.

package rollout

import (
    "context"
    "errors"
)

var ErrKeyReused = errors.New("operation key reused with different input")

type Command struct {
    OperationID string
    RolloutID   string
    Cohort      string
    RuleVersion string
    Enabled     bool
}

type Result struct {
    Revision int64
    Applied  bool
}

type Store interface {
    // Apply atomically records the operation and changes the rollout state.
    // An exact replay returns the original result; conflicting input returns ErrKeyReused.
    Apply(ctx context.Context, cmd Command) (Result, error)
}

type Service struct {
    store Store
}

func (s Service) SetPricingRule(ctx context.Context, cmd Command) (Result, error) {
    if cmd.OperationID == "" || cmd.RolloutID == "" || cmd.RuleVersion == "" {
        return Result{}, errors.New("missing stable rollout identity")
    }
    return s.store.Apply(ctx, cmd)
}
Enter fullscreen mode Exit fullscreen mode

An exact replay should return the original status and revision, not execute the mutation again. Reusing a key with different input is a conflict and should be visible as such. Do not quietly accept it: that would make one identifier mean two intents.

The scheduler also owns part of the contract. It must generate the operation ID before the first network attempt, persist it with the scheduled job, and send the same command until it receives a terminal result. Retries should use bounded exponential backoff with jitter and a deadline. A retry budget prevents an unavailable dependency from turning one rollout into an unbounded queue of attempts.

Avoid encoding correctness in process memory. Restarts happen precisely when incident pressure is high, and queued work may move to another worker. Durable identity is the handoff mechanism.

Instrument attribution at the decision boundary

Cost attribution needs to follow the business decision, not the HTTP status alone. Start a trace span around the logical operation and propagate W3C Trace Context to downstream services. On that span, record low-cardinality fields suitable for grouping: environment, cohort, pricing rule version, workload class, and outcome. Keep operation IDs available for exemplars or targeted lookup rather than promoting them to unbounded metric labels.

The OpenTelemetry semantic-conventions guidance is relevant here: attributes give spans queryable context, while high-cardinality values can make metric dimensions expensive and difficult to operate. Logs can carry the precise operation ID; metrics should answer aggregate questions.

A compact event model is enough:

Signal Required context Question it answers
Trace span rollout, cohort, rule version, outcome Where did this operation wait or fail?
Structured log operation ID, attempt, stored revision Was this delivery a replay or a new mutation?
Counter rule version, cohort, result class Are retries or conflicts rising during exposure?
Cost record rollout, rule version, media account, charge class Which decision produced this spend?

Do not label a replay as another successful application. Report it as a successful replay with applied=false; otherwise the deployment graph overcounts changes. Likewise, a timeout at the caller is not evidence that the mutation failed. Call it unknown until the caller queries by operation ID or retries the same operation and receives the stored result.

Analytical storage can help join rollout dimensions to cost events, but ingestion deduplication is not the correctness boundary. Merge timing, retention, and query behavior differ across storage designs. The source-of-truth transaction must prevent duplicate business effects before events reach an analytical system.

Verify the rollout before widening exposure

Begin with a cohort whose cost can be isolated. Before changing exposure, capture a baseline for pricing-rule applications, replay counts, conflicts, queue age, downstream errors, and spend per accepted media action. Compare like with like: the same account class, placement class, and observation window. A total-cost graph without those dimensions can hide one expensive cohort inside normal traffic.

Then exercise the ambiguous path deliberately in a non-production environment. Commit the mutation and interrupt the response. Deliver the identical command again. The expected evidence is one business revision, one downstream effect, and a replay result tied to the same operation ID. Next, reuse that ID with a changed rule version. The request must be rejected as a conflict.

Three checks matter during each production step:

  1. Business uniqueness: count distinct operation IDs and compare them with committed rollout revisions and downstream pricing jobs.
  2. Decision consistency: confirm every replay retains the original cohort and rule version.
  3. Cost reconciliation: join accepted media actions to the rollout and rule version, then account separately for unattributed records.

Zero unattributed records is the target. An unexplained gap is a stop condition, even when latency and error-rate graphs look normal. Cost is a correctness signal for this rollout, not a decorative finance dashboard.

Hold each exposure step long enough to cover the scheduler's retry window and the downstream processing delay. There is no honest universal number for that hold. Derive it from queue configuration and measured latency, document it in the runbook, and make the promotion check executable.

Roll back without creating a second write storm

Rollback should create a new operation that disables the new rule version for the affected cohort. Never mutate or delete the original idempotency record. Operators need the history to explain which media actions used which rule, and late redeliveries must still resolve to their original result.

Pause promotion first. Drain or quarantine jobs whose intent is no longer valid, while allowing exact retries of already committed operations to resolve normally. If the pricing calculation has already produced external charges, flag rollback cannot erase them; route those records through the established reconciliation process and preserve the rollout dimensions.

The rollback gate is short: the previous rule is active, queue age is falling, no new applications of the disabled version appear, and every post-rollback cost record is attributable. Keep watching beyond the immediate HTTP recovery. Scheduled deliveries often surface later than request dashboards suggest.

The durable lesson is plain. A feature flag controls selection, while an idempotency record controls effects. Observability connects the two so a retry can be proven harmless and every charge can be traced to one decision.

References

Top comments (0)