DEV Community: kanaria007

Make A/B Tests Operable: Goal Surface Deterministic Logs Safe Automation (dry-run / shadow / canary / rollout)

kanaria007 — Thu, 16 Apr 2026 17:00:00 +0000

A/B tests and staged rollouts break in predictable ways:

You optimize a single KPI (CTR/CVR/revenue), Goodhart kicks in, and you quietly destroy SLOs, safety, fairness, or policy compliance.
You can’t replay what happened, so postmortems become “guessing + meetings.”
The automation boundary is vague, and you only notice the damage after 100% rollout.

This post compresses the fix into one minimal pattern:

Evaluation (Goal Surface: multi-objective + constraints)
× Evidence (deterministic logs: replayable)
× Execution (safe automation: dry-run / shadow / canary / rollout)

0) Fix one failure story first

A common real incident:

You ship a new logic (recommendation/search/billing/risk/scoring/routing) via A/B.
A single KPI (e.g., CTR) improves, so you expand the rollout.
Meanwhile error_rate / p95_latency / cost is getting worse.
Logs are thin, so you can’t answer “which input / which branch caused the regression.”
You notice only at 100%, and now you can’t even roll back with confidence.

This is not three independent mistakes.
Evaluation, logging, and automation boundaries are entangled.

1) Evaluation is not a scalar: Goal Surface (multi-objective + constraints)

1.1 What is a Goal Surface?

A Goal Surface is a contract that prevents “winning” by a single number.

Primary (what you want to improve): CVR, churn, search success rate…
Guardrails (SLO gates): error_rate, p95_latency, crash_rate…
Constraints (must not violate): policy violations = 0, PII leaks = 0, forbidden features = 0, (optionally) imbalance / disparate impact monitoring…
Budgets (allowed degradation): p95 +2% max, error_rate +0.1% max…

“Win” means: Primary improves while Guardrails + Constraints hold.

1.2 Minimal Goal Surface format (JSON policy)

Keep evaluation rules outside code and pin them with policy_version.

{
  "policy_id": "exp-eval-policy",
  "policy_version": "2026-02-18",
  "primary": [
    { "metric": "conversion_rate", "direction": "up", "min_lift": 0.002 }
  ],
  "guardrails": [
    { "metric": "error_rate_5m", "op": "<=", "threshold": 0.01, "budget_delta": 0.001 },
    { "metric": "p95_latency_ms_5m", "op": "<=", "threshold": 400, "budget_delta": 20 }
  ],
  "constraints": [
    { "kind": "must_be_zero", "metric": "pii_leak_incidents" },
    { "kind": "must_be_false", "metric": "prohibited_feature_present" }
  ],
  "missing_data_policy": {
    "required_metrics": ["conversion_rate","error_rate_5m","p95_latency_ms_5m","pii_leak_incidents","prohibited_feature_present"],
    "on_missing": "DEGRADE"
  }
}

Key move: on_missing = DEGRADE.
If you can’t observe the required evidence, you stop safely instead of “rolling forward on vibes.”

2) Make failures replayable: a minimal deterministic log spec

A/B tests become painful not because they fail, but because you can’t replay what happened.

So logs are not “for humans.” Logs are for replay.

2.1 Minimal fields you should not skip

run_id: join key for the entire rollout/experiment execution (stable across stages)
event_id: unique per event (often run_id + stage)
experiment_id / variant_id
policy_id / policy_version (the contract)
input_digest: fingerprint of the evaluation input (so you can prove “same input” later)
state_transition: from/to (stage changes)
decision: ACCEPT / REJECT / DEGRADE + reason codes
evidence_refs: dashboard IDs, alert policy IDs, tickets, approvals
metrics_snapshot: the gate evidence (at least required metrics)

2.2 NDJSON append-only example

{"ts":"2026-02-18T10:00:00+09:00","run_id":"exp-42:B:2026-02-18","event_id":"exp-42:B:2026-02-18:CANARY_10","experiment_id":"exp-42","variant_id":"B","policy_id":"exp-eval-policy","policy_version":"2026-02-18","stage":"CANARY_10","state_transition":{"from":"SHADOW","to":"CANARY_10"},"input_digest":"sha256:...","metrics_snapshot":{"conversion_rate":0.132,"error_rate_5m":0.008,"p95_latency_ms_5m":360,"pii_leak_incidents":0,"prohibited_feature_present":false},"decision":{"verdict":"ACCEPT","reason_codes":[],"missing":[]},"evidence_refs":{"dashboard_id":"dash-foo","alert_policy_id":"alert-slo-foo","change_id":"CHG-123"}}
{"ts":"2026-02-18T10:15:00+09:00","run_id":"exp-42:B:2026-02-18","event_id":"exp-42:B:2026-02-18:CANARY_25","experiment_id":"exp-42","variant_id":"B","policy_id":"exp-eval-policy","policy_version":"2026-02-18","stage":"CANARY_25","state_transition":{"from":"CANARY_10","to":"CANARY_25"},"input_digest":"sha256:...","metrics_snapshot":{"conversion_rate":0.131,"error_rate_5m":0.013,"p95_latency_ms_5m":420,"pii_leak_incidents":0,"prohibited_feature_present":false},"decision":{"verdict":"REJECT","reason_codes":["guardrail_violation:error_rate_5m"],"missing":[]},"evidence_refs":{"dashboard_id":"dash-foo","alert_policy_id":"alert-slo-foo","change_id":"CHG-123"}}

With logs like this, you don’t “discuss.” You query.

3) Safe automation boundary: dry-run / shadow / canary / rollout

Once you have Goal Surface + deterministic logs, execution becomes a stage model.

3.1 Recommended stages

dry-run: show plan only (do not execute)
shadow: run in the background, collect comparisons (no user impact)
canary: small percent (10% → 25% → 50% → 100%)
rollout: staged expansion (auto-stop / auto-rollback on gate failures)

flowchart LR
  DR[Dry-run] --> SH[Shadow]
  SH --> C10[Canary 10%]
  C10 --> C25[Canary 25%]
  C25 --> C50[Canary 50%]
  C50 --> R[Rollout 100%]
  C10 -.gate fail.-> RB[Rollback/Stop]
  C25 -.gate fail.-> RB
  C50 -.gate fail.-> RB
  R -.gate fail.-> RB

3.2 Stop conditions must be deterministic

Observation is missing/stale → DEGRADE (hold)
Guardrail violation → REJECT (stop/rollback)
Constraint violation → REJECT (immediate stop)

This removes “judgment by mood.”

3.3 Turn DEGRADE into an operational metric (SLO)

DEGRADE is not failure. It’s a safe branch.
But if DEGRADE keeps increasing, operations get stuck.

So you manage DEGRADE with SLOs—not feelings.

3.3.1 Reason code taxonomy (aggregate-able granularity)

Avoid free-text. Prefer categories that map to “one action to fix.”

A) observation_missing:* (telemetry missing / delayed)

observation_missing:required_metric
observation_missing:delayed_pipeline
observation_missing:broken_dashboard_ref

B) evidence_missing:* (audit grounds missing)

evidence_missing:change_id
evidence_missing:approval
evidence_missing:ticket_link

C) precondition_unknown:* (preconditions not satisfied)

precondition_unknown:assignment_not_stable
precondition_unknown:sample_size_not_reached
precondition_unknown:variant_mapping_ambiguous

3.3.2 Two SLOs that actually work

DEGRADE rate SLO
Example: DEGRADE / (ACCEPT + REJECT + DEGRADE) <= 1% (rolling 7d)
Time-to-resume SLO
Example: P95 time_to_resume <= 30m for observation issues
Example: P95 time_to_resume <= 24h for approval waits

3.3.3 Make DEGRADE re-enterable (resume-friendly)

A DEGRADE event should include:

reason_codes + missing
resume_token (opaque is fine)
requested_actions (typed “next steps”)

Example:

{
  "ts": "2026-02-18T10:30:00+09:00",
  "run_id": "exp-42:B:2026-02-18",
  "event_id": "exp-42:B:2026-02-18:CANARY_25",
  "stage": "CANARY_25",
  "decision": {
    "verdict": "DEGRADE",
    "reason_codes": ["observation_missing:required_metric"],
    "missing": ["p95_latency_ms_5m"]
  },
  "resume": {
    "resume_token": "opaque:rsn_7f3a9c...",
    "requested_actions": [
      {"name": "collect_metric", "params": {"metric": "p95_latency_ms_5m"}},
      {"name": "rerun_gate_check", "params": {"stage": "CANARY_25"}}
    ]
  }
}

Now DEGRADE becomes “a queue,” not “a meeting.”

4) Minimal implementation (stdlib-only): gate evaluation + append logs + replay

You can run this pattern without external libraries.
The purpose is to show the operational skeleton.

(Code below uses PEP 604 union types like float | None, so it requires **Python 3.10+.)

4.1 Stable input fingerprint: `input_digest` (sha256)

# digest.py
from __future__ import annotations

import hashlib
import json
from typing import Any, Dict

def canonical_json(obj: Any) -> str:
    return json.dumps(obj, ensure_ascii=False, sort_keys=True, separators=(",", ":"))

def sha256_hex(s: str) -> str:
    return hashlib.sha256(s.encode("utf-8")).hexdigest()

def input_digest(inp: Dict[str, Any]) -> str:
    return "sha256:" + sha256_hex(canonical_json(inp))

4.2 Gate evaluation: Goal Surface → Verdict

Important: the returned ACCEPT/REJECT/DEGRADE is an operational gate, not “statistical winner.”

ACCEPT = guardrails/constraints are satisfied; safe to proceed to next stage
REJECT = stop/rollback
DEGRADE = hold (missing observation/evidence/preconditions)

# gate_eval.py
from __future__ import annotations

from dataclasses import dataclass
from datetime import datetime
from typing import Any, Dict, List, Tuple

Verdict = str  # "ACCEPT" | "REJECT" | "DEGRADE"

@dataclass(frozen=True)
class Decision:
    verdict: Verdict
    reason_codes: Tuple[str, ...]
    missing: Tuple[str, ...]

def _missing_required(policy: Dict[str, Any], metrics: Dict[str, Any]) -> List[str]:
    req = policy.get("missing_data_policy", {}).get("required_metrics", [])
    out: List[str] = []
    for k in req:
        if k not in metrics:
            out.append(k)
    return out

def evaluate(policy: Dict[str, Any], metrics: Dict[str, Any]) -> Decision:
    mdp = policy.get("missing_data_policy", {})

    # Wrapper-friendly normalization:
    # - {"metrics": {...}, "_meta": {...}} OR a flat dict.
    snapshot = metrics if isinstance(metrics, dict) else {}
    if isinstance(snapshot.get("metrics"), dict):
        data = snapshot["metrics"]
        meta = snapshot.get("_meta") if isinstance(snapshot.get("_meta"), dict) else {}
        # Pull a few optional fields up if present (minimal).
        for k in ("freshness_seconds", "watermark_event_time", "sources"):
            if k in snapshot and k not in meta:
                meta[k] = snapshot[k]
    else:
        data = snapshot
        meta = snapshot.get("_meta") if isinstance(snapshot.get("_meta"), dict) else {}

    # 0) Missing required metrics → DEGRADE/REJECT per policy
    missing = _missing_required(policy, data)
    if missing:
        on_missing = mdp.get("on_missing", "DEGRADE")
        rc = ("observation_missing:required_metric",)
        if on_missing == "REJECT":
            return Decision("REJECT", rc, tuple(missing))
        return Decision("DEGRADE", rc, tuple(missing))

    # 0.5) Freshness gate (optional)
    max_fresh = mdp.get("max_freshness_seconds")
    if max_fresh is not None:
        fs = meta.get("freshness_seconds")
        if fs is None:
            on_stale = mdp.get("on_stale", "DEGRADE")
            rc = ("observation_missing:freshness_seconds",)
            if on_stale == "REJECT":
                return Decision("REJECT", rc, ("freshness_seconds",))
            return Decision("DEGRADE", rc, ("freshness_seconds",))
        try:
            fs_v = float(fs)
            thr_v = float(max_fresh)
        except (TypeError, ValueError):
            return Decision("DEGRADE", ("observation_invalid:freshness_seconds",), ("freshness_seconds",))
        if fs_v > thr_v:
            on_stale = mdp.get("on_stale", "DEGRADE")
            rc = ("observation_stale:over_freshness_budget",)
            if on_stale == "REJECT":
                return Decision("REJECT", rc, ("freshness_seconds",))
            return Decision("DEGRADE", rc, ("freshness_seconds",))

    # 0.6) Watermark skew gate (optional): sources[*].watermark_event_time
    max_skew = mdp.get("max_watermark_skew_seconds")
    if max_skew is not None:
        def _parse_iso(ts: Any) -> float | None:
            if isinstance(ts, str):
                try:
                    return datetime.fromisoformat(ts).timestamp()
                except ValueError:
                    return None
            return None

        wms: List[float] = []
        sources = meta.get("sources")
        if isinstance(sources, list):
            for s in sources:
                if isinstance(s, dict):
                    t = _parse_iso(s.get("watermark_event_time"))
                    if t is not None:
                        wms.append(t)

        if len(wms) < 2:
            return Decision("DEGRADE", ("observation_missing:watermark_event_time",), ("watermark_event_time",))

        skew = float(max(wms) - min(wms))
        if skew > float(max_skew):
            return Decision("DEGRADE", ("observation_stale:watermark_skew",), ("watermark_event_time",))

    # 1) Constraints (must not violate)
    reasons: List[str] = []
    for c in policy.get("constraints", []):
        kind = c.get("kind")
        metric = c.get("metric")
        if not metric:
            return Decision("REJECT", ("invalid_constraint:missing_metric",), ())
        if metric not in data:
            return Decision("DEGRADE", ("observation_missing:constraint_metric",), (metric,))
        v = data[metric]

        if kind == "must_be_zero":
            try:
                if float(v) != 0.0:
                    reasons.append(f"constraint_violation:{metric}")
            except (TypeError, ValueError):
                return Decision("DEGRADE", ("observation_invalid:constraint_value",), (metric,))
        elif kind == "must_be_false":
            if not isinstance(v, bool):
                return Decision("DEGRADE", ("observation_invalid:constraint_value",), (metric,))
            if v is True:
                reasons.append(f"constraint_violation:{metric}")
        else:
            return Decision("REJECT", (f"unknown_constraint_kind:{kind}",), ())

    if reasons:
        return Decision("REJECT", tuple(reasons), ())

    # 2) Guardrails (SLO gates)
    for g in policy.get("guardrails", []):
        m = g["metric"]
        op = g["op"]
        thr = g["threshold"]

        if m not in data:
            return Decision("DEGRADE", ("observation_missing:guardrail_metric",), (m,))

        try:
            v = float(data[m])
            t = float(thr)
        except (TypeError, ValueError):
            return Decision("DEGRADE", ("observation_invalid:guardrail_value",), (m,))

        ok = True
        if op == "<=":
            ok = v <= t
        elif op == "<":
            ok = v < t
        elif op == ">=":
            ok = v >= t
        elif op == ">":
            ok = v > t
        else:
            return Decision("REJECT", (f"unknown_guardrail_op:{op}",), ())

        if not ok:
            return Decision("REJECT", (f"guardrail_violation:{m}",), ())

    # 3) Primary “winner” belongs to a separate stats layer.
    return Decision("ACCEPT", (), ())

4.3 Append-only NDJSON logging

# append_log.py
from __future__ import annotations

import json
from datetime import datetime, timezone, timedelta
from pathlib import Path
from typing import Any, Dict

JST = timezone(timedelta(hours=9))

def now_iso() -> str:
    return datetime.now(JST).isoformat()

def append_ndjson(path: Path, event: Dict[str, Any]) -> None:
    line = json.dumps(event, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("a", encoding="utf-8") as f:
        f.write(line + "\n")

4.4 Stage driver (minimal)

# rollout_driver.py
from __future__ import annotations

import json
from pathlib import Path
from typing import Any, Dict

from digest import input_digest
from gate_eval import evaluate
from append_log import append_ndjson, now_iso

def load_json(path: Path) -> Dict[str, Any]:
    with path.open("r", encoding="utf-8") as f:
        return json.load(f)

def main() -> int:
    policy = load_json(Path("policy.json"))
    log_path = Path("logs/decision.ndjson")

    experiment_id = "exp-42"
    variant_id = "B"

    stages = ["DRY_RUN", "SHADOW", "CANARY_10", "CANARY_25", "CANARY_50", "ROLLOUT_100"]

    prev_stage = None
    for stage in stages:
        # In production: fetch metrics snapshot from Prometheus/Cloud Monitoring/warehouse.
        metrics = load_json(Path(f"snapshots/{stage}.json"))

        inp = {
            "experiment_id": experiment_id,
            "variant_id": variant_id,
            "stage": stage,
            "policy_id": policy["policy_id"],
            "policy_version": policy["policy_version"],
            "metrics_snapshot": metrics,
        }
        d = evaluate(policy, metrics)

        run_id = f"{experiment_id}:{variant_id}:{policy['policy_version']}"
        event_id = f"{run_id}:{stage}"

        event = {
            "ts": now_iso(),
            "run_id": run_id,
            "event_id": event_id,
            "experiment_id": experiment_id,
            "variant_id": variant_id,
            "policy_id": policy["policy_id"],
            "policy_version": policy["policy_version"],
            "stage": stage,
            "state_transition": {"from": prev_stage, "to": stage},
            "input_digest": input_digest(inp),
            "metrics_snapshot": metrics,
            "decision": {"verdict": d.verdict, "reason_codes": list(d.reason_codes), "missing": list(d.missing)},
            "evidence_refs": {"dashboard_id": "dash-foo", "alert_policy_id": "alert-slo-foo", "change_id": "CHG-123"},
        }
        append_ndjson(log_path, event)

        if d.verdict == "REJECT":
            print(f"[STOP] stage={stage} reasons={d.reason_codes}")
            return 1

        if d.verdict == "DEGRADE":
            print(f"[HOLD] stage={stage} missing={d.missing}")
            return 2

        prev_stage = stage
        print(f"[OK] stage={stage}")

    return 0

if __name__ == "__main__":
    raise SystemExit(main())

4.5 Replay (“what happened?” becomes queryable)

# replay.py
from __future__ import annotations

import json
from pathlib import Path
from typing import Any, Dict, Iterable, List

def read_ndjson(path: Path) -> Iterable[Dict[str, Any]]:
    with path.open("r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                yield json.loads(line)

def main() -> int:
    path = Path("logs/decision.ndjson")
    events = list(read_ndjson(path))

    rejects = [e for e in events if e.get("decision", {}).get("verdict") == "REJECT"]
    for e in rejects:
        print(json.dumps({
            "ts": e.get("ts"),
            "run_id": e.get("run_id"),
            "event_id": e.get("event_id"),
            "stage": e.get("stage"),
            "reason_codes": e.get("decision", {}).get("reason_codes", []),
            "policy_version": e.get("policy_version"),
        }, ensure_ascii=False))
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Now postmortems are replayable.

4.6 The missing link from PoC to production (the real constraints)

The minimal code above hides the hardest parts. This section makes them explicit.

4.6.1 Treat `metrics_snapshot` as evidence (with freshness)

The hard part is aligning multiple sources (Prometheus/logs/warehouse) to the “same window.” In practice, perfect alignment is unrealistic.

So treat the snapshot as evidence with meta:

collected_at, window
watermark_event_time (event-time “how far data is complete”)
per-source watermarks
freshness_seconds (how late is this snapshot?)

Then gate freshness in policy:

{
  "missing_data_policy": {
    "required_metrics": ["conversion_rate","error_rate_5m","p95_latency_ms_5m"],
    "on_missing": "DEGRADE",
    "max_freshness_seconds": 120,
    "max_watermark_skew_seconds": 60,
    "on_stale": "DEGRADE"
  }
}

Freshness DEGRADE reason codes become SLO targets:

observation_stale:over_freshness_budget
observation_stale:watermark_skew
observation_missing:required_metric

4.6.2 Don’t mix “promotion (winner)” with “safety gate”

The gate evaluator above answers: “Is it safe to proceed?”
It does not answer: “Did B win statistically?”

Separate layers:

Safety gate: Guardrails + Constraints (canary progression)
Promotion gate: statistics / sample size / sequential testing (100% rollout)

When promotion isn’t ready, DEGRADE (precondition unknown):

precondition_unknown:sample_size_not_reached
precondition_unknown:assignment_not_stable
precondition_unknown:significance_not_ready

Policy can make this explicit:

{
  "promotion_gate": {
    "required_for_stages": ["ROLLOUT_100"],
    "min_sample_size": 100000,
    "require_ci_low_gt": 0.0,
    "on_not_ready": "DEGRADE"
  }
}

4.6.3 Rollback atomicity: automate only reversible changes

Auto-rollback is safest for:

feature flags / routing (stateless switches)
backward-compatible config

Auto-rollback is dangerous for:

schema changes
irreversible migrations
external side effects (emails, billing)

So classify changes:

reversible → allow auto rollout
stateful_or_irreversible → dry-run/shadow only; require human approval after DEGRADE
unsafe_for_experiment → don’t A/B it

4.6.4 Align by watermark contract, not “perfect 5-min windows”

Don’t chase perfect synchronization across sources. Use an explicit watermark contract:

watermark_event_time says “complete up to here”
allow only bounded skew; otherwise DEGRADE

This makes “data pipeline delay” a designed state, not a surprise.

4.6.5 Implement the driver as an orchestrator (state machine)

Real rollouts run for hours/days. Don’t use a long-running loop.

Use:

scheduler (cron/workflow engine), or
event-driven triggers (snapshot arrival, approval granted, alert fired)

Split the “driver” into two layers. This small separation makes the implementation stable.

4.6.5.1 Split into two layers: Orchestrator vs Evaluator

In production, the driver is much more stable if you separate:

(A) Orchestrator (state + re-entry management)
Persist run_id state and manage “when this run can be evaluated again.”

Typical fields:

current stage
resume_token (pointer needed to resume)
last snapshot_id / snapshot_digest
last decision (ACCEPT/REJECT/DEGRADE + reasons)
next evaluation condition (e.g., fresh snapshot arrival / approval granted / time elapsed)

(B) Evaluator (the deterministic judge)
This is the pure-ish part: evaluate(policy, metrics_snapshot) -> verdict.
Keep it as close to a pure function as possible.

This separation removes the unrealistic assumptions of a “single long-running process”:
crashes, restarts, duplicated execution, and partial progress become normal—and safe.

4.6.5.2 DEGRADE is not “stop”—it’s “stop in a re-enterable way”

To make DEGRADE operable, you must be able to re-evaluate the same stage under the same rules after missing grounds are resolved.

So on DEGRADE, always persist (and log):

reason_codes / missing
resume_token (opaque is fine; a DB pointer is often enough)
requested_actions (machine-readable next steps)

Then the Orchestrator can trigger re-evaluation when:

a fresh snapshot arrives
an approval is granted
a deadline passes (scheduled re-check)

DEGRADE becomes a queue with re-entry—not a meeting.

4.6.6 Make Promotion Gate mandatory for 100% rollout (not only safety gates)

As discussed in 4.6.2, gate_eval answers:

“Is it safe to proceed?”

It does not answer:

“Did the variant actually win (business decision)?”

The side effect of this separation is real:

If you reach ROLLOUT_100 with “safe to proceed” only,
you can end up fully rolling out a variant that is slightly worse (statistical noise, but real business loss).

The minimal practical fix is: requirements differ by stage.

4.6.6.1 Canary uses Safety Gate; 100% requires Promotion Gate

CANARY (10/25/50)
- Required: Safety Gate (Guardrails / Constraints)
- Statistics: optional (log as signal)
ROLLOUT_100
- Required: Safety Gate (of course)
- Required: Promotion Gate (winning condition)

Treat Promotion Gate as a separate layer and include “promotion evidence” inside the snapshot, for example:

sample_size
effect_estimate
ci_low / ci_high (confidence interval)
p_value (if needed)
sequential_test_state (for sequential tests)

If Promotion Gate is not satisfied, fail safe with DEGRADE:

precondition_unknown:sample_size_not_reached
precondition_unknown:significance_not_ready
precondition_unknown:assignment_not_stable

4.6.6.2 Make “100% promotion conditions” explicit in policy

Operationally, the clearest approach is to express promotion conditions directly in policy:

{
  "promotion_gate": {
    "required_for_stages": ["ROLLOUT_100"],
    "min_sample_size": 100000,
    "require_ci_low_gt": 0.0,
    "on_not_ready": "DEGRADE"
  }
}

Now “Safety Gate passed, but we are not winning, so we do not go to 100%” becomes a deterministic decision—not vibes.

Tip:

Canary stages stop on safety.
100% promotion stops on winning conditions (Promotion Gate).

This two-layer stop mechanism makes the whole rollout dramatically easier to operate.

5) Closing: three pieces, one operable system

Goal Surface prevents single-metric Goodhart.
Deterministic logs make incidents replayable.
Safe automation makes staged rollout enforceable with REJECT/DEGRADE gates.

Ultimately, turning experiments into operations is not about “smartness.”
It’s about determinism—contracts, evidence, and safe boundaries.

Billing That Won’t Break: Deterministic Design to Kill Double Charges (API Contract DB Boundary Execution Patterns)

kanaria007 — Mon, 13 Apr 2026 17:00:00 +0000

Billing isn’t protected by “correct code” alone.

What protects it is determinism under execution wobble:

retries
timeouts
duplicate requests (user double-clicks)
async delays
worker crashes mid-flight

If you can’t enforce “the same thing can’t be done twice”, you will eventually ship double charges.

In previous posts in this Determinism Series, we used verifiers to “lock” probabilistic outputs. Billing needs the same mindset—but with databases and external payment providers instead of LLMs.

This article compresses the core design into three layers:

L1) API contract: freeze compatibility as a contract; stop breaking changes in CI
L2) DB boundary: make double execution physically impossible with constraints
L3) Execution patterns: combine idempotency / outbox / retry so wobble converges to one result

1) The classic accident: how double charges actually happen

A typical timeline:

Client sends POST /charges
Server writes to DB → calls the external payment API
The payment API succeeds, but the server response times out
Client retries (or the user clicks again)
Server treats it as “new” → double charge

The key is this split:

Network failure and payment success can be separated.

So “success vs failure” becomes ambiguous by default.

To kill this class of accident, you must pin identity.

2) The conclusion: determinism is built in 3 layers

L3 (execution): converge to the same outcome, even with retries

If a retry arrives, return the same charge_id
If async processing wobbles, capture at most once
If a worker dies mid-flight, re-run continues safely

L2 (database): make double execution impossible

unique constraints on business identity and external IDs
one-way state transitions
append-only audit ledger

L1 (API): stop contract drift in CI

schema as a “contract”
fail the PR if someone:
- adds required fields without a safe migration
- changes meanings/types
- removes endpoints

3) L3: Five deterministic patterns that kill most double-charge bugs

Pattern 1) Idempotency-Key (pin request identity)

Client includes Idempotency-Key.
Server guarantees:

same key → returns the same charge_id
state may advance, but identity is stable

Important nuance:

You must pin business identity, not “request text.”

For billing, “same charge” must be well-defined.

Pattern 2) Unique constraint + Upsert (DB refuses “twice”)

Make the identity column(s) UNIQUE and use:

INSERT ... ON CONFLICT ...

So the second request is not treated as new.

Pattern 3) Outbox (move external effects outside the DB transaction)

Trying to do “DB commit + external payment call” as one atomic thing always breaks.

Instead:

write charges + outbox in one DB transaction and commit
a worker processes the outbox (at-least-once)
external calls must be idempotent
store the provider’s returned ID and use it for reconciliation

Provider idempotency differs (headers vs body vs constraints).
The only thing you can deterministically control is:

the idempotency key you send, not the provider’s payment_id.

Why “exactly-once” is a myth (and why Outbox + Reconciliation is not optional)

When you process billing, you’re always spanning two different systems of truth:

your database transaction log, and
the payment provider’s ledger.

There is no general mechanism that lets you atomically commit both.

Even if your DB transaction commits, the network can fail before you observe the provider response.
Even if the provider captures the payment, your process can crash before persisting the result.
So the system naturally falls into this unavoidable ambiguity:

“Did the external effect happen, or did we just not see it?”

This is why “exactly-once execution” is not something you implement by effort.
Instead, you design for convergence:

Idempotency prevents “retry = duplicate effect.”
Outbox makes “planned external effect” durable and replayable.
State machine + ledger makes progress and corrections explicit.
Reconciliation resolves ambiguity when reality and internal state diverge.

In billing, the realistic goal is not “perfect atomicity.”
It’s:

At-least-once delivery + exactly-once effect (via idempotency) + auditable convergence (via reconciliation).

Pattern 4) State machine (track how far you got)

Minimal example:

PENDING → CAPTURED → (maybe REFUNDED, etc.)
transitions are one-way
rollback becomes correction events, not “rewind the past”

The truth table: internal state vs provider state (what you actually do)

A small table like this prevents “we thought it was fine” incidents.
Treat provider state as the authority for external effects, and use reconciliation to converge.

Internal (`charges.status`)	Provider state	Meaning	Deterministic action
`PENDING`	`UNKNOWN` (timeout / no response)	You don’t know if money moved	Reconcile by querying provider using your idempotency key / metadata; stay `PENDING` until confirmed
`PENDING`	`CAPTURED`	External effect happened but you didn’t record it	Set `external_payment_id`, append ledger “CAPTURED (reconciled)”, transition to `CAPTURED`
`PENDING`	`FAILED` / `NOT_FOUND`	External effect did not happen	Append ledger “FAILED”, transition to `FAILED` (or retry if policy allows)
`CAPTURED`	`CAPTURED`	Normal stable state	No-op
`CAPTURED`	`FAILED` / `NOT_FOUND`	Rare but serious inconsistency	Open a case; treat as RML-3; investigate provider disputes
`FAILED`	`CAPTURED`	The “double-charge seed”	Treat as incident; correct forward (refund + explanation) and fix the retry/idempotency path

Notes:

The table only works if you can reliably lookup provider state by your own stable key (idempotency key / metadata).
Any row that means “money moved but we didn’t record it” is why reconciliation must exist as a first-class feature.

Pattern 5) Reconciliation (assume drift will happen)

External provider state and internal state will drift eventually.

So build a reconciler as a feature, not a “later patch”:

compare provider transactions vs internal ledger
detect differences
route to auto-correction or human approval

4) L2: A minimal PostgreSQL boundary (DDL)

Design goals:

append-only ledger
idempotency table
charge state machine
outbox queue

Also: never store money as float.
Store minor units as integers (JPY=yen, USD=cents). Currency defines meaning.

-- 1) Charge state machine
create table charges (
  charge_id text primary key,
  customer_id text not null,
  order_id text not null,
  amount bigint not null check (amount > 0),
  currency text not null default 'JPY',
  status text not null check (status in ('PENDING','CAPTURED','FAILED')),
  external_payment_id text,
  created_at timestamptz not null default now(),
  updated_at timestamptz not null default now(),
  constraint uq_charge_business_key unique (customer_id, order_id)
);

-- External payment id must be unique once assigned
create unique index uq_charges_external_payment_id
  on charges(external_payment_id)
  where external_payment_id is not null;

-- 2) Idempotency (pins retries)
create table idempotency_keys (
  endpoint text not null,
  idem_key text not null,
  request_hash text not null,

  -- Pin the knot: which charge this key maps to + snapshot of first response
  charge_id text not null references charges(charge_id),
  response_status int not null,
  response_body_json text not null,

  created_at timestamptz not null default now(),
  primary key (endpoint, idem_key)
);

-- 3) Outbox (planned external effects)
create table outbox (
  outbox_id bigserial primary key,
  event_type text not null,
  aggregate_id text not null,
  payload_json text not null,
  status text not null default 'PENDING'
    check (status in ('PENDING','PROCESSING','DONE','DEAD')),

  created_at timestamptz not null default now(),
  available_at timestamptz not null default now(),
  locked_at timestamptz,
  lock_token text,
  attempts int not null default 0,
  last_error text
);

create index idx_outbox_ready
  on outbox (available_at, outbox_id)
  where status = 'PENDING';

create index idx_outbox_stuck
  on outbox (locked_at, outbox_id)
  where status = 'PROCESSING';

-- Cleanup helpers
create index idx_outbox_done_created_at
  on outbox (created_at, outbox_id)
  where status = 'DONE';

create index idx_outbox_dead_created_at
  on outbox (created_at, outbox_id)
  where status = 'DEAD';

create index idx_idempotency_created_at
  on idempotency_keys (created_at);

-- 4) Append-only audit ledger
create table charge_ledger (
  ledger_id bigserial primary key,
  charge_id text not null references charges(charge_id),
  event text not null,
  details_json text not null,
  created_at timestamptz not null default now()
);

Minimal survivability tips

keep truth in charge_ledger (append-only); keep charges as “current state + references”
request_hash is mandatory: same Idempotency-Key with different body must fail

5) L1: Freeze the API contract in CI (minimal)

Make OpenAPI/JSON Schema a real contract. Stop breaking changes in PRs.

5.1 Minimal OpenAPI excerpt

openapi: 3.0.3
info:
  title: Billing API
  version: 1.0.0
paths:
  /charges:
    post:
      summary: Create a charge (idempotent)
      parameters:
        - in: header
          name: Idempotency-Key
          required: true
          schema: { type: string }
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/CreateChargeRequest"
      responses:
        "200":
          description: Existing charge (idempotent replay)
          content:
            application/json:
              schema: { $ref: "#/components/schemas/ChargeResponse" }
        "201":
          description: Created
          content:
            application/json:
              schema: { $ref: "#/components/schemas/ChargeResponse" }
components:
  schemas:
    CreateChargeRequest:
      type: object
      x-idempotency-hash-fields: [customer_id, order_id, amount, currency]
      required: [customer_id, order_id, amount]
      properties:
        customer_id: { type: string }
        order_id:    { type: string }
        amount:  { type: integer, format: int64, minimum: 1 }
        currency:
          type: string
          description: ISO 4217 currency code (minor-unit integer in `amount`)
          default: JPY
          pattern: "^[A-Z]{3}$"
    ChargeResponse:
      type: object
      required: [charge_id, status, amount, currency]
      properties:
        charge_id: { type: string }
        status:    { type: string }
        amount: { type: integer, format: int64 }
        currency: { type: string, pattern: "^[A-Z]{3}$" }
        external_payment_id: { type: string, nullable: true }

Note the key idea:

x-idempotency-hash-fields declares the hash surface of “same charge”.

If schema changes, CI should detect “you changed the meaning” or “you forgot to update hash surface”.

5.2 Minimal breaking-change checker (stdlib-only, JSON OpenAPI)

To stay dependency-free, convert OpenAPI to JSON in CI (e.g., openapi_prev.json and openapi.json) and run a Python script to detect:

removed paths/methods
added required fields
type changes
invalid or drifting x-idempotency-hash-fields

# scripts/check_openapi_breaking.py
from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any, Dict, Tuple


def load(path: Path) -> Dict[str, Any]:
    with path.open("r", encoding="utf-8") as f:
        return json.load(f)


def get_schemas(doc: Dict[str, Any]) -> Dict[str, Any]:
    return doc.get("components", {}).get("schemas", {})


def validate_idempotency_hash_surface(schema_name: str, schema: Dict[str, Any]) -> Tuple[str, ...]:
    """
    Validate `x-idempotency-hash-fields` on a request schema (the “hash surface” of identity).

    Contract (minimal):
    - `x-idempotency-hash-fields` must exist (list[str])
    - Every listed field must exist under `properties`
    - As a conservative rule, `required` fields should be included in the hash surface
      (otherwise “same Idempotency-Key, different meaning” can slip through).
    """  
    issues = []

    fields = schema.get("x-idempotency-hash-fields")
    if fields is None:
        issues.append(f"schema '{schema_name}': missing x-idempotency-hash-fields")
        return tuple(issues)

    if not isinstance(fields, list) or not fields or not all(isinstance(x, str) for x in fields):
        issues.append(f"schema '{schema_name}': invalid x-idempotency-hash-fields (expected non-empty list[str])")
        return tuple(issues)

    if len(set(fields)) != len(fields):
        issues.append(f"schema '{schema_name}': x-idempotency-hash-fields contains duplicates: {fields}")

    props = schema.get("properties", {}) or {}
    for f in fields:
        if f not in props:
            issues.append(f"schema '{schema_name}': hash field '{f}' not found in properties")

    req = set(schema.get("required", []) or [])
    missing = req - set(fields)
    if missing:
        issues.append(f"schema '{schema_name}': required fields missing from hash surface: {sorted(missing)}")

    return tuple(issues)


def _resolve_schema_ref(doc: Dict[str, Any], ref: str) -> Dict[str, Any] | None:
    # Minimal implementation: only supports "#/components/schemas/<Name>".  
    prefix = "#/components/schemas/"
    if not isinstance(ref, str) or not ref.startswith(prefix):
        return None
    name = ref[len(prefix):]
    schemas = get_schemas(doc)
    s = schemas.get(name)
    return s if isinstance(s, dict) else None


def _operation_requires_idempotency_key(op: Dict[str, Any]) -> bool:
    params = op.get("parameters", []) or []
    if not isinstance(params, list):
        return False

    for p in params:
        if not isinstance(p, dict):
            continue
        if p.get("in") != "header":
            continue
        if p.get("name") != "Idempotency-Key":
            continue
        if p.get("required") is True:
            return True
    return False


def _extract_request_schema(doc: Dict[str, Any], op: Dict[str, Any]) -> tuple[str, Dict[str, Any]] | None:
    """
    Extract requestBody schema from an operation (minimal implementation).

    - Prefer application/json (fallback to the first media type)
    - Resolve $ref only for "#/components/schemas/<Name>"
    - Inline schema is also supported (schema_name becomes "<inline>")
    """  
    rb = op.get("requestBody")
    if not isinstance(rb, dict):
        return None

    content = rb.get("content")
    if not isinstance(content, dict) or not content:
        return None

    # Prefer application/json; otherwise use the first content type.  
    media = content.get("application/json")
    if not isinstance(media, dict):
        media = next((v for v in content.values() if isinstance(v, dict)), None)
    if not isinstance(media, dict):
        return None

    schema = media.get("schema")
    if not isinstance(schema, dict):
        return None

    ref = schema.get("$ref")
    if isinstance(ref, str):
        resolved = _resolve_schema_ref(doc, ref)
        if resolved is None:
            return None
        schema_name = ref.split("/")[-1]
        return (schema_name, resolved)

    # inline schema (minimal support)
    return ("<inline>", schema)


def validate_idempotent_operations(doc: Dict[str, Any]) -> Tuple[str, ...]:
    """
    Operation-driven validation:
    - An operation that requires Idempotency-Key should have a requestBody
    - That requestBody schema must declare x-idempotency-hash-fields
    - If declared, it must also pass validate_idempotency_hash_surface
    """
    issues: list[str] = []

    paths = doc.get("paths", {}) or {}
    if not isinstance(paths, dict):
        return tuple(issues)

    for path, item in paths.items():
        if not isinstance(item, dict):
            continue
        for method, op in item.items():
            if method.lower() not in ("get", "post", "put", "patch", "delete"):
                continue
            if not isinstance(op, dict):
                continue

            if not _operation_requires_idempotency_key(op):
                continue

            extracted = _extract_request_schema(doc, op)
            if extracted is None:
                issues.append(f"operation '{path} {method}': Idempotency-Key is required but requestBody schema cannot be extracted/resolved")
                continue
            schema_name, s = extracted

            if "x-idempotency-hash-fields" not in s:
                issues.append(f"schema '{schema_name}': missing x-idempotency-hash-fields (required by operation '{path} {method}')")
                continue

            issues.extend(validate_idempotency_hash_surface(schema_name, s))

    return tuple(issues)

def breaking_changes(prev: Dict[str, Any], cur: Dict[str, Any]) -> Tuple[str, ...]:
    issues = []

    prev_paths = prev.get("paths", {}) or {}
    cur_paths = cur.get("paths", {}) or {}

    # 1) removed paths/methods
    for p, methods in prev_paths.items():
        if p not in cur_paths:
            issues.append(f"removed path: {p}")
            continue
        if not isinstance(methods, dict):
            continue
        for m in methods.keys():
            if m not in cur_paths[p]:
                issues.append(f"removed method: {p} {m}")

    # 2) added required fields in schema (likely breaking)
    prev_s = get_schemas(prev) or {}
    cur_s = get_schemas(cur) or {}

    for name, ps in prev_s.items():
        cs = cur_s.get(name)
        if cs is None:
            issues.append(f"removed schema: {name}")
            continue
        if not isinstance(ps, dict) or not isinstance(cs, dict):
            continue

        prev_req = set(ps.get("required", []) or [])
        cur_req = set(cs.get("required", []) or [])
        added_req = cur_req - prev_req
        if added_req:
            issues.append(f"schema '{name}': added required fields: {sorted(added_req)}")

        # 3) property type changes (minimal check)
        pp = ps.get("properties", {}) or {}
        cp = cs.get("properties", {}) or {}
        if isinstance(pp, dict) and isinstance(cp, dict):
            for prop, pdef in pp.items():
                cdef = cp.get(prop)
                if cdef is None:
                    issues.append(f"schema '{name}': removed property '{prop}'")
                    continue
                if not isinstance(pdef, dict) or not isinstance(cdef, dict):
                    continue
                pt = (pdef.get("type"), pdef.get("format"))
                ct = (cdef.get("type"), cdef.get("format"))
                if pt != ct:
                    issues.append(f"schema '{name}.{prop}': type changed {pt} -> {ct}")

    # 4) Identity hash surface contract check (schema-driven)
    # Validate every schema that declares x-idempotency-hash-fields (no CI changes needed for new schemas)
    for sname, schema in cur_s.items():
        if isinstance(schema, dict) and ("x-idempotency-hash-fields" in schema):
            issues.extend(validate_idempotency_hash_surface(sname, schema))

    # 5) Operations that require Idempotency-Key must have x-idempotency-hash-fields
    # in their requestBody schema, and the surface must be consistent (operation-driven)
    issues.extend(validate_idempotent_operations(cur))

    return tuple(issues)


def main() -> int:
    prev_path = Path(sys.argv[1])
    cur_path = Path(sys.argv[2])

    prev = load(prev_path)
    cur = load(cur_path)

    issues = breaking_changes(prev, cur)
    if issues:
        print("BREAKING CHANGES DETECTED:")
        for i in issues:
            print(f"- {i}")
        return 1

    print("OK: no breaking changes detected")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Minimal GitHub Actions example (same vibe as the previous post).

name: contract
on: [pull_request]
jobs:
  openapi:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python scripts/check_openapi_breaking.py openapi_prev.json openapi.json

Dedicated OpenAPI diff tools are great if you need rigor, but in many teams it’s enough to start by blocking only the truly destructive changes with a minimal checker like this.

6) Minimal implementation sketch (Kotlin + JDBC + PostgreSQL)

This is the “deterministic kernel”:

If idempotency record exists:

same hash → replay same charge_id
different hash → 409 Conflict
1. Insert charge by business key (UNIQUE)
2. Write ledger + outbox in the same transaction
3. Insert idempotency record with response snapshot
4. Return 201 for first creator; 200 for replays

6.1 request_hash (prevent “same key, different body”)

import java.security.MessageDigest

fun sha256Hex(s: String): String {
  val md = MessageDigest.getInstance("SHA-256")
  val bytes = md.digest(s.toByteArray(Charsets.UTF_8))
  return bytes.joinToString("") { b -> "%02x".format(b.toInt() and 0xff) }
}

data class CreateChargeRequest(
  val customerId: String,
  val orderId: String,
  val amount: Long,
  val currency: String = "JPY",
)

fun requestHash(req: CreateChargeRequest): String {
  // Stable ordering + length-prefixing to avoid delimiter collisions.
  fun part(k: String, v: String): String = "$k=${v.length}:$v"
  val canonical = listOf(
    part("customerId", req.customerId),
    part("orderId", req.orderId),
    part("amount", req.amount.toString()),
    part("currency", req.currency),
  ).joinToString("|")
  return sha256Hex(canonical)
}

6.2 Key JDBC constraint: avoid “aborted transaction” traps

In PostgreSQL, once a statement fails inside a transaction (e.g., UNIQUE violation), the transaction becomes aborted.
So prefer:

INSERT ... ON CONFLICT DO NOTHING over
catching unique violations and continuing

Sample Code

import java.sql.Connection
import java.util.UUID

class Conflict(msg: String) : RuntimeException(msg)

fun createCharge(
  conn: Connection,
  idemKey: String,
  endpoint: String,
  req: CreateChargeRequest,
): Pair {
  val reqHash = requestHash(req)

  // Important: restore JDBC autoCommit so the caller sees no side effects.  
  val prevAutoCommit = conn.autoCommit
  conn.autoCommit = false
  try {
    // 1) If idemKey already exists, return the same result (= retry-safe)
    val existing = selectIdempotency(conn, endpoint, idemKey)
    if (existing != null) {
      if (existing.requestHash != reqHash) {
        throw Conflict("Idempotency-Key reused with different request body")
      }

      // Idempotent replay converges to "return the same charge" (expect 200)
      val row = selectChargeById(conn, existing.chargeId)
        ?: throw IllegalStateException("charge not found for idempotency record: ${existing.chargeId}")

      val resp = ChargeResponse(
        chargeId = row.chargeId,
        status = row.status,
        amount = row.amount,
        currency = row.currency,
        externalPaymentId = row.externalPaymentId,
      )
      conn.commit()
      return 200 to resp
    }

    // 2) Create a charge (business-key UNIQUE prevents double creation)
    // Recommended: converge with ON CONFLICT instead of catching exceptions
    val newChargeId = "ch_" + UUID.randomUUID().toString()
    val insertedCharge = insertCharge(conn, newChargeId, req)

    val charge = if (insertedCharge) {
      // 3) Ledger (audit: append-only)
      appendLedger(conn, newChargeId, "CHARGE_CREATED", """{"idem_key":"${jsonEscape(idemKey)}"}""")

      // 4) Outbox (queue external payment as a "planned effect")
      insertOutbox(
        conn = conn,
        eventType = "charge.capture.requested",
        aggregateId = newChargeId,
        payloadJson = """{"charge_id":"${jsonEscape(newChargeId)}"}"""
      )

      ChargeRow(
        chargeId = newChargeId,
        status = "PENDING",
        amount = req.amount,
        currency = req.currency,
        externalPaymentId = null,
      )
    } else {
      // (customer_id, order_id) already exists → return the existing charge as the "identity-converged" result
      val row = selectChargeByBusinessKey(conn, req.customerId, req.orderId)
        ?: throw IllegalStateException("charge exists by business key, but cannot be loaded")

      // If the existing charge has different amount/currency, treat it as a deterministic conflict.  
      if (row.amount != req.amount || row.currency != req.currency) {
        throw Conflict("Charge already exists for (customer_id, order_id) with different amount/currency")
      }
      row
    }

    // 5) Pin response (idempotency_keys)
    val resp = ChargeResponse(
      chargeId = charge.chargeId,
      status = charge.status,
      amount = charge.amount,
      currency = charge.currency,
      externalPaymentId = charge.externalPaymentId,
    )

    // Only the "first committer" of the idem key returns 201 (others converge to 200)
    val firstStatus = if (insertedCharge) 201 else 200
    val insertedIdem = insertIdempotency(
      conn = conn,
      idemKey = idemKey,
      endpoint = endpoint,
      requestHash = reqHash,
      chargeId = charge.chargeId,
      responseStatus = firstStatus,
      response = resp,
    )

    val fixed = selectIdempotency(conn, endpoint, idemKey) ?: error("idempotency insert failed")
    if (fixed.requestHash != reqHash) {
      throw Conflict("Idempotency-Key reused with different request body")
    }

      // Return the latest state (status may advance asynchronously).  
    val row = selectChargeById(conn, fixed.chargeId)
      ?: throw IllegalStateException("charge not found for idempotency record: ${fixed.chargeId}")

    val resp2 = ChargeResponse(
      chargeId = row.chargeId,
      status = row.status,
      amount = row.amount,
      currency = row.currency,
      externalPaymentId = row.externalPaymentId,
    )

    conn.commit()
    val httpStatus = if (insertedCharge &amp;&amp; insertedIdem) 201 else 200
    return httpStatus to resp2
  } catch (e: Exception) {
    runCatching { conn.rollback() }
    throw e
  } finally {
    conn.autoCommit = prevAutoCommit
  }
}

data class ChargeRow(
  val chargeId: String,
  val status: String,
  val amount: Long,
  val currency: String,
  val externalPaymentId: String?,
)

fun selectChargeByBusinessKey(conn: Connection, customerId: String, orderId: String): ChargeRow? {
  conn.prepareStatement(
    """
    select charge_id, status, amount, currency, external_payment_id
    from charges
    where customer_id = ? and order_id = ?
    """.trimIndent()
  ).use { ps -&gt;
    ps.setString(1, customerId)
    ps.setString(2, orderId)
    ps.executeQuery().use { rs -&gt;
      if (!rs.next()) return null
      return ChargeRow(
        chargeId = rs.getString(1),
        status = rs.getString(2),
        amount = rs.getLong(3),
        currency = rs.getString(4),
        externalPaymentId = rs.getString(5),
      )
    }
  }
}

fun selectChargeById(conn: Connection, chargeId: String): ChargeRow? {
  conn.prepareStatement(
    """
    select charge_id, status, amount, currency, external_payment_id
    from charges
    where charge_id = ?
    """.trimIndent()
  ).use { ps -&gt;
    ps.setString(1, chargeId)
    ps.executeQuery().use { rs -&gt;
      if (!rs.next()) return null
      return ChargeRow(
        chargeId = rs.getString(1),
        status = rs.getString(2),
        amount = rs.getLong(3),
        currency = rs.getString(4),
        externalPaymentId = rs.getString(5),
      )
    }
  }
}

data class IdempotencyRow(
  val requestHash: String,
  val chargeId: String,
  val firstResponseStatus: Int,
)

fun selectIdempotency(conn: Connection, endpoint: String, idemKey: String): IdempotencyRow? {
  conn.prepareStatement(
    """
    select request_hash, charge_id, response_status
    from idempotency_keys
    where endpoint = ? and idem_key = ?
    """.trimIndent()
  ).use { ps -&gt;
    ps.setString(1, endpoint)
    ps.setString(2, idemKey)
    ps.executeQuery().use { rs -&gt;
      if (!rs.next()) return null
      return IdempotencyRow(
        requestHash = rs.getString(1),
        chargeId = rs.getString(2),
        firstResponseStatus = rs.getInt(3),
      )
    }
  }
}

fun insertCharge(conn: Connection, chargeId: String, req: CreateChargeRequest): Boolean {
  // uq_charge_business_key prevents double creation for the same (customer_id, order_id).  
  // Recommended: converge with ON CONFLICT instead of branching on unique-violation exceptions.  
  return conn.prepareStatement(
    """
    insert into charges (charge_id, customer_id, order_id, amount, currency, status)
    values (?, ?, ?, ?, ?, 'PENDING')
    on conflict (customer_id, order_id) do nothing
    """.trimIndent()
  ).use { ps -&gt;
    ps.setString(1, chargeId)
    ps.setString(2, req.customerId)
    ps.setString(3, req.orderId)
    ps.setLong(4, req.amount)
    ps.setString(5, req.currency)
    ps.executeUpdate() == 1
  }
}

fun appendLedger(conn: Connection, chargeId: String, event: String, detailsJson: String) {
  conn.prepareStatement(
    "insert into charge_ledger (charge_id, event, details_json) values (?, ?, ?)"
  ).use { ps -&gt;
    ps.setString(1, chargeId)
    ps.setString(2, event)
    ps.setString(3, detailsJson)
    ps.executeUpdate()
  }
}

fun insertOutbox(conn: Connection, eventType: String, aggregateId: String, payloadJson: String) {
  conn.prepareStatement(
    """
    insert into outbox (event_type, aggregate_id, payload_json, status)
    values (?, ?, ?, 'PENDING')
    """.trimIndent()
  ).use { ps -&gt;
    ps.setString(1, eventType)
    ps.setString(2, aggregateId)
    ps.setString(3, payloadJson)
    ps.executeUpdate()
  }
}

fun jsonEscape(s: String): String = buildString {
  for (ch in s) {
    when (ch) {
      '\\' -&gt; append("\\\\")
      '"'  -&gt; append("\\\"")
      '\b' -&gt; append("\\b")
      '\u000C' -&gt; append("\\f")
      '\n' -&gt; append("\\n")
      '\r' -&gt; append("\\r")
      '\t' -&gt; append("\\t")
      else -&gt; if (ch.code &lt; 0x20) {
        append(String.format("\\u%04x", ch.code))
      } else {
        append(ch)
      }
    }
  }
}

fun toResponseJson(resp: ChargeResponse): String {
  // Minimal stable format (in production, prefer a JSON library or jsonb).  
  val ext = resp.externalPaymentId?.let { "\"${jsonEscape(it)}\"" } ?: "null"
  return """{"charge_id":"${jsonEscape(resp.chargeId)}","status":"${jsonEscape(resp.status)}","amount":${resp.amount},"currency":"${jsonEscape(resp.currency)}","external_payment_id":$ext}"""
}

fun insertIdempotency(
  conn: Connection,
  idemKey: String,
  endpoint: String,
  requestHash: String,
  chargeId: String,
  responseStatus: Int,
  response: ChargeResponse,
): Boolean {
  return conn.prepareStatement(
    """
    insert into idempotency_keys (endpoint, idem_key, request_hash, charge_id, response_status, response_body_json)
    values (?, ?, ?, ?, ?, ?)
    on conflict (endpoint, idem_key) do nothing
    """.trimIndent()
  ).use { ps -&gt;
    ps.setString(1, endpoint)
    ps.setString(2, idemKey)
    ps.setString(3, requestHash)
    ps.setString(4, chargeId)
    ps.setInt(5, responseStatus)
    ps.setString(6, toResponseJson(response))
    ps.executeUpdate() == 1
  }
}

7) Outbox worker (FOR UPDATE SKIP LOCKED)

Outbox is at-least-once. You must assume:

worker crashes mid-flight
locks get stuck
retries happen

Key rules:

lock and commit quickly
call external payment outside DB transaction
apply results in a short transaction
idempotency is mandatory at the provider boundary
implement lease-based requeue for stuck PROCESSING rows

Sample Code

import java.sql.Connection
import java.util.UUID
import javax.sql.DataSource

data class OutboxRow(
  val outboxId: Long,
  val eventType: String,
  val aggregateId: String,
  val payloadJson: String,
  val attempts: Int,
)

fun lockNextOutbox(conn: Connection, lockToken: String, leaseSeconds: Int): OutboxRow? {
  // Pick PENDING (or reclaim stuck PROCESSING) and mark it as PROCESSING.  
  conn.prepareStatement(
    """
    with picked as (
      select outbox_id, event_type, aggregate_id, payload_json
      from outbox
      where
        (status = 'PENDING' and available_at &lt;= now())
        or (status = 'PROCESSING' and locked_at &lt; now() - (? * interval '1 second'))
      order by outbox_id
      for update skip locked
      limit 1
    )
    update outbox o
      set status='PROCESSING',
          locked_at=now(),
          lock_token=?,
          attempts=attempts+1
    from picked
    where o.outbox_id = picked.outbox_id
    returning o.outbox_id, o.event_type, o.aggregate_id, o.payload_json, o.attempts
    """.trimIndent()
  ).use { ps -&gt;
    ps.setInt(1, leaseSeconds)
    ps.setString(2, lockToken)
    ps.executeQuery().use { rs -&gt;
      if (!rs.next()) return null
      return OutboxRow(
        outboxId = rs.getLong(1),
        eventType = rs.getString(2),
        aggregateId = rs.getString(3),
        payloadJson = rs.getString(4),
        attempts = rs.getInt(5),
      )
    }
  }
}

fun markOutboxDone(conn: Connection, outboxId: Long, lockToken: String): Int {
  conn.prepareStatement(
    "update outbox set status='DONE', lock_token=null, locked_at=null where outbox_id=? and lock_token=?"
  ).use { ps -&gt;
    ps.setLong(1, outboxId)
    ps.setString(2, lockToken)
    return ps.executeUpdate()
  }
}

fun markOutboxRetry(conn: Connection, outboxId: Long, lockToken: String, delaySeconds: Int, err: String): Int {
  conn.prepareStatement(
    """
    update outbox
      set status='PENDING',
          available_at=now() + (? * interval '1 second'),
          last_error=?,
          lock_token=null,
          locked_at=null
    where outbox_id=? and lock_token=?
    """.trimIndent()
  ).use { ps -&gt;
    ps.setInt(1, delaySeconds.coerceIn(1, 24 * 60 * 60))
    ps.setString(2, err.take(5000))
    ps.setLong(3, outboxId)
    ps.setString(4, lockToken)
    return ps.executeUpdate()
  }
}

fun markOutboxDead(conn: Connection, outboxId: Long, lockToken: String, err: String): Int {
  conn.prepareStatement(
    """
    update outbox
      set status='DEAD',
          last_error=?,
          lock_token=null,
          locked_at=null
    where outbox_id=? and lock_token=?
    """.trimIndent()
  ).use { ps -&gt;
    ps.setString(1, err.take(5000))
    ps.setLong(2, outboxId)
    ps.setString(3, lockToken)
    return ps.executeUpdate()
  }
}

// Provider returns its own payment_id; you do not deterministically generate it.
// The deterministic anchor you control is the idempotency key (Idempotency-Key / Request-Id equivalent).  
data class CaptureResult(
  val providerPaymentId: String,       // payment ID returned by the provider
  val providerIdempotencyKey: String,  // idempotency key we send (header/param/etc.)
)

// sha256Hex() is the shared utility defined in section 6.2

fun providerIdempotencyKeyFor(chargeId: String): String {
  // Providers often impose length/charset constraints (e.g., max 255 chars).
  // If raw chargeId is risky, shorten via sha256.  
  return "cap_" + sha256Hex(chargeId).take(32)
}

fun capturePaymentIdempotent(chargeId: String): CaptureResult {
  val idem = providerIdempotencyKeyFor(chargeId)

  // In production:
  // - Stripe: set `Idempotency-Key` header to `idem`
  // - PayPal/etc: set the equivalent request-id/idempotency key in header/body
  // Then persist the provider’s payment_id/transaction_id (this becomes a reconciliation anchor).  
  val providerPaymentId = "pay_" + UUID.randomUUID().toString() // example: in production this is assigned by the provider

  return CaptureResult(
    providerPaymentId = providerPaymentId,
    providerIdempotencyKey = idem,
  )
}

/**
 * Recommended: accept DataSource and do NOT hold a DB connection during external HTTP calls.
 *
 * Flow:
 * 1) lock TX (short): claim outbox row as PROCESSING with lock_token
 * 2) external call (outside TX; do not hold a DB connection)
 * 3) apply TX (short): finalize charges/ledger/outbox
 */
fun processOutboxOnce(ds: DataSource): Boolean {
  fun  withTx(conn: Connection, block: () -&gt; T): T {
    val prev = conn.autoCommit
    conn.autoCommit = false
    try {
      val r = block()
      conn.commit()
      return r
    } catch (e: Exception) {
      runCatching { conn.rollback() }
      throw e
    } finally {
      conn.autoCommit = prev
    }
  }

  val lockToken = UUID.randomUUID().toString()

  val leaseSeconds = 5 * 60 // tune to external API timeout/p99 + buffer
  val maxAttempts = 25

  fun retryDelaySeconds(attempts: Int): Int {
    // 30s * 2^(attempts-1), capped at 15 min (minimum 30s)
    val base = 30
    val capped = attempts.coerceIn(1, 10) // 2^9 = 512 is sufficient
    val exp = 1 shl (capped - 1)
    return (base * exp).coerceIn(base, 15 * 60)
  }

  // 1) lock TX (keep it short)
  val row: OutboxRow = ds.connection.use { conn -&gt;
    withTx(conn) { lockNextOutbox(conn, lockToken, leaseSeconds) }
  } ?: return false

  // 2) Unknown event types go to DEAD (short TX)
  if (row.eventType != "charge.capture.requested") {
    ds.connection.use { conn -&gt;
      withTx(conn) {
        markOutboxDead(conn, row.outboxId, lockToken, "unknown event_type=${row.eventType}")
      }
    }
    return true
  }

  val chargeId = row.aggregateId

  // 3) External call (outside TX; no DB connection held)
  val cap: CaptureResult = try {
    capturePaymentIdempotent(chargeId)
  } catch (e: Exception) {
    ds.connection.use { conn -&gt;
      withTx(conn) {
        if (row.attempts &gt;= maxAttempts) {
          markOutboxDead(
            conn,
            row.outboxId,
            lockToken,
            "max_attempts_exceeded(payment): attempts=${row.attempts} err=${e.javaClass.simpleName}:${e.message}"
          )
        } else {
          val delay = retryDelaySeconds(row.attempts)
          markOutboxRetry(
            conn,
            row.outboxId,
            lockToken,
            delay,
            "payment_error(attempts=${row.attempts},delay=${delay}s)=${e.javaClass.simpleName}:${e.message}"
          )
        }
      }
    }
    return true
  }

  // 4) Apply TX (short)
  try {
    ds.connection.use { conn -&gt;
      withTx(conn) {
        val updated = conn.prepareStatement(
          """
          update charges
            set status='CAPTURED',
                external_payment_id=?,
                updated_at=now()
          where charge_id=?
            and status='PENDING'
          """.trimIndent()
        ).use { ps -&gt;
          ps.setString(1, cap.providerPaymentId)
          ps.setString(2, chargeId)
          ps.executeUpdate()
        }

        // Only append to ledger when a state transition actually occurs (prevents ledger bloat on retries)
        if (updated == 1) {
          appendLedger(
            conn,
            chargeId,
            "CAPTURED",
            """{"external_payment_id":"${jsonEscape(cap.providerPaymentId)}","provider_idempotency_key":"${jsonEscape(cap.providerIdempotencyKey)}"}"""
          )
        }

        // Even if updated==0 (already CAPTURED, etc.), mark outbox as DONE (converges idempotently)
        markOutboxDone(conn, row.outboxId, lockToken)
      }
    }
    return true
  } catch (e: Exception) {
    ds.connection.use { conn -&gt;
      runCatching {
        withTx(conn) {
          if (row.attempts &gt;= maxAttempts) {
            markOutboxDead(
              conn,
              row.outboxId,
              lockToken,
              "max_attempts_exceeded(db): attempts=${row.attempts} err=${e.javaClass.simpleName}:${e.message}"
            )
          } else {
            val delay = retryDelaySeconds(row.attempts)
            markOutboxRetry(
              conn,
              row.outboxId,
              lockToken,
              delay,
              "db_error(attempts=${row.attempts},delay=${delay}s)=${e.javaClass.simpleName}:${e.message}"
            )
          }
        }
      }
    }
    return true
  }
}

8) Ten golden cases (grow determinism like a verifier)

If you read the earlier post “Grow the Verifier, Not the Prompt,” apply the same discipline here:

Freeze the behaviors that would otherwise create money incidents.

Suggested cases:

ACCEPT (3): new (201), replay (200), normalization (if any)
DEGRADE (4): external uncertain (still PENDING), missing payment method, order status unknown, observability missing
REJECT (3): reused Idempotency-Key w/ different body (409), invalid amount, forbidden ops

The key is: don’t “test everything.” Freeze how you avoid the ways you die.

9) Common traps (avoiding these already helps a lot)

Idempotency-Key exists but body drift is allowed → request_hash is mandatory
Trying to do external payment call “inside the same DB transaction” → breaks eventually
No state machine → you can’t safely resume
No ledger → you can’t explain or reconcile
Money as float → don’t

10) Scaling and ops: when the simple design hits limits

This design is intentionally DB-centered to prioritize correctness. It works well—until you scale.

10.1 Outbox/idempotency “hot table” + vacuum/bloat

High-frequency insert/update/delete creates dead tuples and vacuum pressure.

Practical progression:

small/medium: partial indexes + small-batch deletes + tuned autovacuum
large: partition by time and DROP PARTITION (avoid vacuum hell)
if still too heavy: move the queue outward (Kafka/SQS) but keep deterministic “planned effect” facts in DB (transactional outbox / CDC)

10.2 Reconciliation is hard (so design the key first)

Your job isn’t “implement a perfect reconciler.” It’s:

embed your internal charge_id into provider metadata if possible
store provider payment IDs
record reconcile diffs as ledger events before mutating state

10.3 Distributed writes change assumptions

This post assumes a single writer DB giving you strong serialization for lock/lease patterns.
If you move to multi-master / multi-region writes, redesign around ordering keys, not wall clock.

10.4 TTL and “identity durability”

Idempotency tables need a TTL policy.
But note the key difference:

If you have a strong business key (UNIQUE(customer_id, order_id)), idempotency is not your final line of defense.
If you don’t have a business key (e.g., top-ups where multiple identical operations are valid), you must introduce a durable operation_id / intent ID as the identity anchor.

Recap (TL;DR)

Double charges happen less from “bad billing logic” and more from execution wobble (retries/timeouts/async gaps).
The fix is building determinism in three layers:
- L1 — API contract in CI (stop meaning drift)
- L2 — DB boundary (make “twice” physically impossible)
- L3 — Execution patterns (idempotency + outbox + state machine + reconciliation)
The minimum survivable set is:
- Idempotency-Key + request_hash
- UNIQUE constraints
- Outbox
- State machine
- Ledger + Reconciliation

Once you have this, the rest is “just” running it on Cloud Run / Kubernetes and iterating with tests and logs.
Preventing double charges is not about “smartness.” It’s about determinism—and determinism grows through tests and receipts.

Closing

Double charges rarely come from “bad billing logic.”

They come from execution wobble—retries, timeouts, and async gaps—meeting a system that has no deterministic identity anchor.

To stop them, build determinism in three layers:

L1 contract in CI
L2 DB constraints that refuse “twice”
L3 execution patterns (idempotency + outbox + state machine + reconciliation)

In billing, determinism isn’t “nice architecture.”

It’s the thing that keeps your product alive.

Grow the Verifier, Not the Prompt: Run Production with 10 Golden Cases

kanaria007 — Mon, 06 Apr 2026 17:00:00 +0000

The moment you put an AI agent into a real workflow, two realities show up fast:

Models and prompts wobble (updates, infra, tools, inputs)
Most failures are “missing grounds,” not “rule violations.”

In the previous posts, we fixed the division of labor:

LLM generates a proposal (a plan)
Verifier deterministically returns ACCEPT / REJECT / DEGRADE, and may normalize the plan
Executor runs Typed Actions only (dry-run → approval → production)

Now the real question:

What do you “grow” so the system doesn’t collapse in ops?

My answer is simple:

Don’t start by tuning the prompt.
Start by freezing 10 golden cases for the verifier.

0) The premise (re-stated)

LLMs are probabilistic. Output variance is not evil.

What’s evil is executing variance.

Also: LLM-generated “explanations” (including chain-of-thought-like text) are not audit-grade grounds. What you should pin is:

input schema (what counts as admissible grounds)
policy_id / policy_version
deterministic rule-evaluation logs
evidence / trace IDs

1) Why “10 cases” is enough to start

You’re not trying to get coverage.

You’re trying to freeze the handful of patterns that kill you in production:

Boundaries: deadlines, caps, percentage steps, state transitions
Forbidden: privilege, SoD violations, legal holds, prohibited fields
Missing: approvals, identity verification, evidence links, observability
Ordering: no revoke, retry without idempotency, double execution
Exceptions become normal: DEGRADE missing → humans keep rescuing via interpretation

A small number of “representative accidents” eliminates most catastrophic behavior. After that, you grow from incidents and DEGRADE logs.

So: 10 is not a magic number.
It’s “the smallest number that forces an operational skeleton.”

2) What a golden case actually freezes

Do not freeze LLM output.
Freeze verifier output.

A golden case is a contract:

Schema’d input + proposed plan
Expected verifier outputs:
- verdict (ACCEPT / REJECT / DEGRADE)
- reasons (typed reason codes)
- missing (machine-readable missing list, for DEGRADE)
- normalized_plan (the executable, verified Typed Actions)

If the model changes, prompts change, tools change—your ops can still survive as long as the verifier continues to:

stop correctly,
request missing grounds correctly,
normalize into safe executable plans.

3) Minimal case format (portable)

YAML is nicer to read, but if you want a portable “single truth,” JSON is the safest. You can author in YAML and convert later.

3.1 One-case JSON example

{
  "name": "jit_access_missing_security_approval_degrade",
  "input": {
    "policy": { "policy_id": "iam-jit-access", "policy_version": "2026-01-20" },
    "access_request": {
      "request_id": "AR-2026-00077",
      "requester_user_id": "u-1234",
      "target_resource": "prod-db:billing",
      "requested_role": "db.readonly",
      "requested_duration_minutes": 60,
      "reason_code": "INCIDENT_RESPONSE",
      "incident_id": "INC-88921",
      "ticket_id": "T-2026-004512"
    },
    "approvals": { "manager_approved": true, "security_approved": false },
    "context": { "on_call": true, "break_glass": false },
    "evidence": { "runbook_id": "rbk-prod-db-read" }
  },
  "proposed_plan": {
    "actions": [
      {
        "name": "iam.grant_temporary_role",
        "params": {
          "user_id": "u-1234",
          "resource": "prod-db:billing",
          "role": "db.readonly",
          "duration_minutes": 60
        }
      },
      {
        "name": "iam.revoke_role",
        "params": {
          "user_id": "u-1234",
          "resource": "prod-db:billing",
          "role": "db.readonly"
        }
      }
    ]
  },
  "expect": {
    "verdict": "DEGRADE",
    "reasons": ["missing_security_approval"],
    "missing": ["approvals.security_approved"],
    "normalized_plan": []
  }
}

Key point: missing is not prose. It’s a machine-readable list (paths). That’s what makes DEGRADE operable.

4) Which 10 cases to pick (template)

A practical starting distribution:

ACCEPT: 3
- minimal happy path
- boundary-but-OK path
- happy path where normalization happens (proposal → normalized plan)
DEGRADE: 4
- missing approval
- missing evidence
- state uncertain
- observability missing (SLO/metric not available)
REJECT: 3
- clear forbidden (privilege/SoD/legal hold)
- window/time violation
- prohibited field or process violation

This set forces you to implement:

stopping safely (DEGRADE),
denying deterministically (REJECT),
producing an executable “true plan” (normalized_plan).

5) Ten starter cases (names + intent)

You can cut/paste these ideas into JSON files and expand your schemas:

JIT Access (3 cases)

1. `jit_access_accept_minimal` (ACCEPT)

{
  "name": "jit_access_accept_minimal",
  "input": {
    "policy": { "policy_id": "iam-jit-access", "policy_version": "2026-01-20" },
    "access_request": {
      "request_id": "AR-1",
      "requester_user_id": "u-1234",
      "target_resource": "prod-db:billing",
      "requested_role": "db.readonly",
      "requested_duration_minutes": 60,
      "incident_id": "INC-1",
      "ticket_id": "T-1"
    },
    "approvals": { "manager_approved": true, "security_approved": true },
    "context": { "break_glass": false }
  },
  "proposed_plan": {
    "actions": [
      {
        "name": "iam.grant_temporary_role",
        "params": {
          "user_id": "u-1234",
          "resource": "prod-db:billing",
          "role": "db.readonly",
          "duration_minutes": 60
        }
      },
      {
        "name": "iam.revoke_role",
        "params": {
          "user_id": "u-1234",
          "resource": "prod-db:billing",
          "role": "db.readonly"
        }
      }
    ]
  },
  "expect": {
    "verdict": "ACCEPT",
    "reasons": [],
    "missing": [],
    "normalized_plan": [
      {
        "name": "iam.grant_temporary_role",
        "params": {
          "user_id": "u-1234",
          "resource": "prod-db:billing",
          "role": "db.readonly",
          "duration_minutes": 60
        }
      },
      {
        "name": "iam.revoke_role",
        "params": {
          "user_id": "u-1234",
          "resource": "prod-db:billing",
          "role": "db.readonly"
        }
      }
    ]
  }
}

2. `jit_access_degrade_missing_security_approval` (DEGRADE)

{
  "name": "jit_access_degrade_missing_security_approval",
  "input": {
    "policy": { "policy_id": "iam-jit-access", "policy_version": "2026-01-20" },
    "access_request": {
      "request_id": "AR-2",
      "requester_user_id": "u-1234",
      "target_resource": "prod-db:billing",
      "requested_role": "db.readonly",
      "requested_duration_minutes": 60,
      "incident_id": "INC-2",
      "ticket_id": "T-2"
    },
    "approvals": { "manager_approved": true, "security_approved": false },
    "context": { "break_glass": false }
  },
  "proposed_plan": {
    "actions": [
      {
        "name": "iam.grant_temporary_role",
        "params": {
          "user_id": "u-1234",
          "resource": "prod-db:billing",
          "role": "db.readonly",
          "duration_minutes": 60
        }
      }
    ]
  },
  "expect": {
    "verdict": "DEGRADE",
    "reasons": ["missing_security_approval"],
    "missing": ["approvals.security_approved"],
    "normalized_plan": []
  }
}

3. `jit_access_reject_admin_role_without_break_glass` (REJECT)

{
  "name": "jit_access_reject_admin_role_without_break_glass",
  "input": {
    "policy": { "policy_id": "iam-jit-access", "policy_version": "2026-01-20" },
    "access_request": {
      "request_id": "AR-3",
      "requester_user_id": "u-1234",
      "target_resource": "prod-db:billing",
      "requested_role": "db.admin",
      "requested_duration_minutes": 60,
      "incident_id": "INC-3",
      "ticket_id": "T-3"
    },
    "approvals": { "manager_approved": true, "security_approved": true },
    "context": { "break_glass": false }
  },
  "proposed_plan": {
    "actions": [
      {
        "name": "iam.grant_temporary_role",
        "params": {
          "user_id": "u-1234",
          "resource": "prod-db:billing",
          "role": "db.admin",
          "duration_minutes": 60
        }
      }
    ]
  },
  "expect": {
    "verdict": "REJECT",
    "reasons": ["admin_role_requires_break_glass"],
    "missing": [],
    "normalized_plan": []
  }
}

Production Change Management (3 cases)

4. `change_degrade_missing_rollback_plan` (DEGRADE)

{
  "name": "change_degrade_missing_rollback_plan",
  "input": {
    "policy": { "policy_id": "prod-change-policy", "policy_version": "2026-01-10" },
    "change_request": {
      "change_type": "feature_flag_rollout",
      "flag_key": "new_invoice_flow",
      "to": { "percent": 10 },
      "rollback_plan_id": null
    },
    "guardrails": {
      "canary": { "step_percent": [10, 25, 50, 100] },
      "slo_gates": [{ "metric": "error_rate_5m", "op": "<=", "threshold": 0.01 }]
    },
    "approvals": { "owner_approved": true, "sre_approved": true }
  },
  "proposed_plan": {
    "actions": [
      {
        "name": "feature_flag.set_percent",
        "params": { "flag_key": "new_invoice_flow", "percent": 10 }
      }
    ]
  },
  "expect": {
    "verdict": "DEGRADE",
    "reasons": ["missing_rollback_plan"],
    "missing": ["change_request.rollback_plan_id"],
    "normalized_plan": []
  }
}

5. `change_reject_no_canary_steps` (REJECT)

{
  "name": "change_reject_no_canary_steps",
  "input": {
    "policy": {"policy_id": "prod-change-policy", "policy_version": "2026-01-10"},
    "change_request": {
      "change_type": "feature_flag_rollout",
      "flag_key": "new_invoice_flow",
      "rollback_plan_id": "rb-1"
    },
    "guardrails": {
      "canary": {"step_percent": [100]},
      "slo_gates": [{"metric": "error_rate_5m", "op": "<=", "threshold": 0.01}]
    },
    "approvals": {"owner_approved": true, "sre_approved": true}
  },
  "proposed_plan": {"actions": [{"name": "feature_flag.set_percent", "params": {"flag_key": "new_invoice_flow", "percent": 100}}]},
  "expect": {
    "verdict": "REJECT",
    "reasons": ["canary_steps_required"],
    "missing": [],
    "normalized_plan": []
  }
}

6. `change_accept_normalize_force_rollback_hook` (ACCEPT + normalize)

{
  "name": "change_accept_normalize_force_rollback_hook",
  "input": {
    "policy": {"policy_id": "prod-change-policy", "policy_version": "2026-01-10"},
    "change_request": {
      "change_type": "feature_flag_rollout",
      "flag_key": "new_invoice_flow",
      "risk_level": "MEDIUM",
      "rollback_plan_id": "rb-2026-0091"
    },
    "guardrails": {
      "canary": {"step_percent": [10, 25, 50, 100], "step_wait_minutes": 15},
      "slo_gates": [{"metric": "error_rate_5m", "op": "<=", "threshold": 0.01}],
      "rollback": {"auto_rollback_enabled": true}
    },
    "approvals": {"owner_approved": true, "sre_approved": true}
  },
  "proposed_plan": {
    "actions": [
      {"name": "feature_flag.set_percent", "params": {"flag_key": "new_invoice_flow", "percent": 10}},
      {"name": "slo_gate.check", "params": {"window_minutes": 15}}
    ]
  },
  "expect": {
    "verdict": "ACCEPT",
    "reasons": [],
    "missing": [],
    "normalized_plan": [
      {"name": "feature_flag.set_percent", "params": {"flag_key": "new_invoice_flow", "percent": 10}},
      {"name": "slo_gate.check", "params": {"window_minutes": 15}},
      {"name": "rollback.hook.ensure", "params": {"rollback_plan_id": "rb-2026-0091"}}
    ]
  }
}

Personal Data Erasure (3 cases)

7. `erasure_degrade_identity_not_verified` (DEGRADE)

{
  "name": "erasure_degrade_identity_not_verified",
  "input": {
    "policy": {"policy_id": "privacy-erasure-policy", "policy_version": "2026-01-05"},
    "erasure_request": {"subject_user_id": "C-1", "identity_verification": {"verified": false}},
    "holds": {"legal_hold": false}
  },
  "proposed_plan": {"actions": [{"name": "privacy.delete", "params": {"system": "crm", "subject_user_id": "C-1"}}]},
  "expect": {
    "verdict": "DEGRADE",
    "reasons": ["identity_verification_required"],
    "missing": ["erasure_request.identity_verification.verified"],
    "normalized_plan": []
  }
}

8. `erasure_reject_legal_hold` (REJECT)

{
  "name": "erasure_reject_legal_hold",
  "input": {
    "policy": {"policy_id": "privacy-erasure-policy", "policy_version": "2026-01-05"},
    "erasure_request": {"subject_user_id": "C-2", "identity_verification": {"verified": true}},
    "holds": {"legal_hold": true}
  },
  "proposed_plan": {"actions": [{"name": "privacy.delete", "params": {"system": "crm", "subject_user_id": "C-2"}}]},
  "expect": {
    "verdict": "REJECT",
    "reasons": ["legal_hold_blocks_erasure"],
    "missing": [],
    "normalized_plan": []
  }
}

9. `erasure_accept_normalize_retention_to_redact` (ACCEPT + normalize)

{
  "name": "erasure_accept_normalize_retention_to_redact",
  "input": {
    "policy": {"policy_id": "privacy-erasure-policy", "policy_version": "2026-01-05"},
    "erasure_request": {"subject_user_id": "C-3", "identity_verification": {"verified": true}},
    "holds": {"legal_hold": false, "accounting_retention_required": true}
  },
  "proposed_plan": {
    "actions": [
      {"name": "privacy.delete", "params": {"system": "billing", "subject_user_id": "C-3"}},
      {"name": "privacy.tombstone.write", "params": {"subject_user_id": "C-3"}}
    ]
  },
  "expect": {
    "verdict": "ACCEPT",
    "reasons": [],
    "missing": [],
    "normalized_plan": [
      {"name": "privacy.redact", "params": {"system": "billing", "subject_user_id": "C-3", "mode": "accounting_retention"}},
      {"name": "privacy.tombstone.write", "params": {"subject_user_id": "C-3"}}
    ]
  }
}

Underwriting packet (1 case)

10. `uw_degrade_missing_required_documents` (DEGRADE)

(This is explicitly about preventing the LLM from becoming the decision-maker.)

{
  "name": "uw_degrade_missing_employment_proof",
  "input": {
    "policy": {"policy_id": "credit-underwriting-policy", "policy_version": "2026-01-01"},
    "application": {"application_id": "APP-1", "requested_amount_jpy": 500000},
    "documents": {"identity_verified": true, "income_proof": {"provided": true}, "employment_proof": {"provided": false}},
    "fairness_controls": {"prohibited_fields_present": false}
  },
  "proposed_plan": {"actions": [{"name": "uw.emit_decision", "params": {"decision": "APPROVE", "reason": "looks good"}}]},
  "expect": {
    "verdict": "DEGRADE",
    "reasons": ["missing_required_documents"],
    "missing": ["documents.employment_proof.provided"],
    "normalized_plan": [
      {"name": "uw.request_more_documents", "params": {"missing": ["employment_proof"]}}
    ]
  }
}

The point is not “these domains.”
The point is the shape of failures: missing/forbidden/boundary/normalization.

6) A minimal golden harness (stdlib-only)

This is the part that makes everything real: run the cases in CI.

6.1 Directory layout

golden/
  cases/
    01_jit_access_accept.json
    02_jit_access_degrade_missing_approval.json
    ...
    10_uw_degrade_missing_docs.json
  run_golden.py
  verifier_stub.py   # replace with your real verifier

6.2 Harness (run_golden.py)

(Python 3.10+; stdlib only.)

from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict, List, Tuple

from verifier_stub import verify  # replace with your verifier


@dataclass(frozen=True)
class Expect:
    verdict: str
    reasons: Tuple[str, ...]
    missing: Tuple[str, ...]
    normalized_plan: Tuple[Dict[str, Any], ...]


def load_case(path: Path) -> Dict[str, Any]:
    with path.open("r", encoding="utf-8") as f:
        return json.load(f)


def normalize_plan(plan: Any) -> Tuple[Dict[str, Any], ...]:
    if plan is None:
        return ()
    if not isinstance(plan, list):
        raise TypeError(f"normalized_plan must be list[dict], got {type(plan)}")

    out: List[Dict[str, Any]] = []
    for i, a in enumerate(plan):
        if not isinstance(a, dict):
            raise TypeError(f"normalized_plan[{i}] must be dict, got {type(a)}")
        out.append(a)
    return tuple(out)


def to_expect(d: Dict[str, Any]) -> Expect:
    return Expect(
        verdict=str(d.get("verdict")),
        reasons=tuple(d.get("reasons", [])),
        missing=tuple(d.get("missing", [])),
        normalized_plan=normalize_plan(d.get("normalized_plan", [])),
    )


def diff(a: Any, b: Any) -> str:
    ja = json.dumps(a, ensure_ascii=False, sort_keys=True, indent=2)
    jb = json.dumps(b, ensure_ascii=False, sort_keys=True, indent=2)
    return f"--- expected\n{ja}\n--- actual\n{jb}"


def main() -> int:
    cases_dir = Path(__file__).parent / "cases"
    paths = sorted(cases_dir.glob("*.json"))
    if not paths:
        print("No golden cases found.", file=sys.stderr)
        return 2

    failed: List[str] = []

    for p in paths:
        case = load_case(p)
        name = case.get("name", p.name)
        inp = case["input"]
        proposed = case["proposed_plan"]
        exp = to_expect(case["expect"])

        actual = verify(inp, proposed)
        verdict = actual.get("verdict")
        if not isinstance(verdict, str) or not verdict:
            raise KeyError(f"verify() must return non-empty 'verdict' for case={name}")

        act = Expect(
            verdict=verdict,
            reasons=tuple(actual.get("reasons", [])),
            missing=tuple(actual.get("missing", [])),
            normalized_plan=normalize_plan(actual.get("normalized_plan", [])),
        )

        if exp != act:
            failed.append(name)
            print(f"\n[FAIL] {name}")
            print(diff(exp.__dict__, act.__dict__))

    if failed:
        print(f"\nFAILED {len(failed)}/{len(paths)} cases: {', '.join(failed)}", file=sys.stderr)
        return 1

    print(f"OK {len(paths)}/{len(paths)} cases")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

6.3 Verifier “plug-in” shape (verifier_stub.py)

from __future__ import annotations

from typing import Any, Dict


def verify(inp: Dict[str, Any], proposed_plan: Dict[str, Any]) -> Dict[str, Any]:
    """
    Replace this with your real verifier.

    Required return shape:
      verdict: "ACCEPT" | "REJECT" | "DEGRADE"
      reasons: [reason_code...]
      missing: [path...]
      normalized_plan: [typed_action...]
    """
    return {
        "verdict": "DEGRADE",
        "reasons": ["stub"],
        "missing": ["replace_with_real_verifier"],
        "normalized_plan": [],
    }

7) Run it in CI (minimal GitHub Actions)

name: golden
on: [push, pull_request]
jobs:
  golden:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python golden/run_golden.py

Now PRs will fail if:

you accidentally changed DEGRADE to ACCEPT,
normalization disappeared,
reason codes drifted without a deliberate update,
missing lists broke.

That’s how “agent ops” becomes engineering—not vibes.

8) How to grow the verifier (the operational loop)

Golden cases are not write-once artifacts. Grow them like this:

Aggregate DEGRADE logs (top missing paths / top reason codes)
If a missing pattern repeats, add one golden case
When a golden case breaks:

bug (unintended drift) → fix verifier
policy change (intended) → update case and record the change reason

Only after this skeleton exists does “prompt improvement” become meaningful.
Otherwise, prompt tuning becomes untestable, vibe-driven optimization.

9) Common traps

Trying to freeze LLM output Freeze verifier outputs instead. LLM proposals can wobble.
Trying to operate with REJECT only Real ops fails on missing grounds. Without DEGRADE, humans rescue everything forever.
Reason codes as free text If you can’t aggregate it, you can’t SLO it.
No normalization If the verifier can’t emit the executable “true plan,” ops gets pulled by whatever the LLM proposed.

Summary

Grow the verifier, not the prompt.
Freeze verifier outputs: verdict / reasons / missing / normalized_plan
Start with 10 golden cases that represent the ways you die in ops
Run them in CI
Grow from DEGRADE logs + incidents, not from vibes

If you treat an LLM as a proposer, you can even ask it to suggest new golden cases.
But first, build the harness and lock the initial ten.

Design DEGRADE (Defer) and Your Agent Becomes “Operations”

kanaria007 — Thu, 02 Apr 2026 17:00:00 +0000

Most agents don’t break because they violated a rule.

They break because they kept going while missing the grounds to decide: approvals, evidence links, unambiguous IDs, confirmed state, dependency health, etc.

If your system only has two outcomes—

ACCEPT (do it)
REJECT (don’t)

—then real work quickly devolves into “humans rescue everything by interpretation,” and audits become “we can’t reproduce why this happened.”

So if you want an agent to become operable in production, the first thing you should design is not REJECT, but:

DEGRADE (DEFER): a safe, re-enterable stop.
Not failure—a controlled branch that turns “we can’t decide yet” into a runnable workflow.

0) Reminder: LLMs propose. Deterministic systems decide.

Treat the LLM as a proposer (plans, drafts, suggested steps).
Pin audit-grade grounds somewhere else:

input schema (what counts as admissible grounds)
policy_id + policy_version
deterministic rule-evaluation logs
evidence / trace IDs

Your execution system should run typed actions only, after verification (dry-run → approval → production).

1) Why “REJECT-only” collapses in practice

1) Missing grounds get mislabeled as “violations”

If every “unknown / missing” becomes REJECT, operators constantly override via interpretation. Over time, tacit knowledge grows, not the system.

2) The agent starts “filling in” what’s missing

When many fields are missing, an LLM will often move toward making the story coherent—because that “feels helpful.” That’s exactly how you get execution on fake grounds.

3) Exceptions never become a process

What you actually need is a durable record of:

what was missing (missing keys / approvals / evidence)
what you requested next (request list)
which gate to retry after (next gate)

Without that, postmortems become rituals, not learning.

2) The DEGRADE design patterns

DEGRADE is a mechanical branch: “we can’t decide safely → stop.”
The trick is making the stop implementable and operable.

Pattern A: Return reason codes + missing list (not prose)

When you DEGRADE, don’t return a question. Return:

category reason code (why we stopped)
missing requirements list (what’s missing, as machine paths)

Examples:

MISSING_APPROVAL: approvals.security_approved
MISSING_EVIDENCE: change_request.dashboard_id
STATE_UNKNOWN: current_flag_state_unavailable

Pattern B: Provide DEGRADE-specific typed actions

DEGRADE is not “do nothing.” It’s a standard next step:

request_more_info
escalate
open_task
attach_evidence

The LLM can draft human-facing messages—but the verifier decides what must be requested.

Pattern C: Deterministically separate “missing” from “violations”

Draw a hard line:

REJECT = confirmed rule violation (forbidden role, outside allowed window, missing required SLO gates…)
DEGRADE = missing grounds / uncertain state / dependency down / contradictions
ACCEPT = all prerequisites satisfied

Pattern D: Make DEGRADE progress (not an infinite hold)

A good DEGRADE must have exits:

auto-create a task
route to an approval workflow
request evidence links
auto-escalate on timeout

Pattern E: Design DEGRADE reason codes as category + granularity + SLO

DEGRADE will grow. That’s fine.

What’s bad is: “we don’t know what kind of DEGRADE this is,” “we don’t know how long it’s been stuck,” “we can’t improve it.”

Minimum requirements:

Category: maps almost 1:1 to owner team / runbook / SLO
Examples: MISSING_APPROVAL, MISSING_EVIDENCE, STATE_UNKNOWN, DEPENDENCY_UNAVAILABLE, CONFLICT_DETECTED, POLICY_AMBIGUOUS.
Granularity: return a mechanical missing list (paths), not prose
Examples: approvals.sre_approved, change_request.rollback_plan_id, etc.
SLO metadata: DEGRADE is “a state with time,” so it needs timers
Suggested fields:

retry_after_seconds
escalate_after_seconds
owners (who owns this)
deadline_at (optional hard stop)

Pattern F: Make DEGRADE re-enterable

This is the final move that makes DEGRADE “operations”:

If missing grounds get filled, the same case can be retried through the same verifier and produce a fresh verdict.

Key techniques:

Fix a join key (request_id / change_id / case_id)
Make validation “pure-ish”: doc in → verdict out (If external state is required, stop with STATE_UNKNOWN / DEPENDENCY_UNAVAILABLE.)
Return a resume token + enforce idempotency in the executor
Represent human approval as typed actions (approval.collect, approval.record, case.resume)
Provide exactly one resume entrypoint (one API/job)

3) Example: Change management (production rollout) with DEGRADE

Production config changes are “small” but have huge blast radius—perfect for DEGRADE-first design.

3.1 Input schema (pin the grounds)

change_request:
  change_id: "CHG-2026-00112"
  service: "billing-api"
  environment: "prod"
  created_at: "2026-02-15T22:30:00+09:00"   # required time claim (missing => DEGRADE)
  change_type: "feature_flag_rollout"
  flag_key: "new_invoice_flow"
  to: { enabled: true, percent: 10 }
  window:
    start_at: "2026-02-16T01:00:00+09:00"
    end_at:   "2026-02-16T03:00:00+09:00"
  risk_level: "MEDIUM"
  rollback_plan_id: "rb-2026-0091"
  dashboard_id: "dash-billing-api"
  alert_policy_id: "alert-billing-slo"

approvals:
  owner_approved: true
  sre_approved: false

policy:
  policy_id: "prod-change-policy"
  policy_version: "2026-01-10"

guardrails:
  canary:
    step_percent: [10, 25, 50, 100]
    step_wait_minutes: 15
  slo_gates:
    - metric: "error_rate_5m"
      op: "<="
      threshold: 0.01

3.2 Typed actions (including DEGRADE exits)

change.plan.publish
feature_flag.set_percent
slo_gate.check
rollback.execute
audit.append
change.request_more_info (DEGRADE)
change.block (REJECT)

3.3 Minimal verifier (stdlib-only) that prefers DEGRADE over “guessing”

Below is the core idea:

missing approval / missing rollback plan / missing observability → DEGRADE
missing SLO gates / invalid canary steps / outside allowed window → REJECT
on DEGRADE, return:
- degrade_reasons (categories)
- missing (mechanical paths)
- SLO metadata (retry_after_seconds, escalate_after_seconds, owners)
- resume_token
- a typed action change.request_more_info

(Code below uses PEP 604 union types like DegradeMeta | None, so it requires **Python 3.10+.)

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from datetime import datetime
from enum import Enum, unique
from typing import Any, Dict, List, Sequence, Tuple


@unique
class VerdictLevel(Enum):
    ACCEPT = "ACCEPT"
    REJECT = "REJECT"
    DEGRADE = "DEGRADE"


@unique
class RejectReason(Enum):
    OUTSIDE_CHANGE_WINDOW = "outside_change_window"
    NO_SLO_GATES = "no_slo_gates"
    INVALID_CANARY_STEPS = "invalid_canary_steps"
    INVALID_STEP_WAIT = "invalid_step_wait"
    PROD_CHANGE_POLICY_MISSING = "prod_change_policy_missing"


@unique
class DegradeReason(Enum):
    # Categories that map to owners + SLOs
    MISSING_CHANGE_ID = "missing_change_id"
    MISSING_APPROVAL = "missing_approval"
    MISSING_ROLLBACK_PLAN = "missing_rollback_plan"
    MISSING_OBSERVABILITY = "missing_observability"
    MISSING_TIME_CLAIM = "missing_time_claim"
    MALFORMED_TIMESTAMP = "malformed_timestamp"
    DEPENDENCY_UNAVAILABLE = "dependency_unavailable"
    STATE_UNKNOWN = "state_unknown"
    CONFLICT_DETECTED = "conflict_detected"


@unique
class ActionName(Enum):
    CHANGE_REQUEST_MORE_INFO = "change.request_more_info"


@dataclass(frozen=True)
class TypedAction:
    name: ActionName
    params: Dict[str, Any]


@dataclass(frozen=True)
class DegradeMeta:
    missing: Tuple[str, ...]
    retry_after_seconds: int
    escalate_after_seconds: int
    resume_token: str
    owners: Tuple[str, ...]


@dataclass(frozen=True)
class Verdict:
    level: VerdictLevel
    reject_reasons: Tuple[RejectReason, ...] = ()
    degrade_reasons: Tuple[DegradeReason, ...] = ()
    degrade_meta: DegradeMeta | None = None
    normalized_plan: Tuple[TypedAction, ...] = ()


def _parse_iso8601(s: str) -> datetime:
    return datetime.fromisoformat(s)


def _sha256_hex(b: bytes) -> str:
    return hashlib.sha256(b).hexdigest()


def _make_resume_token(change_id: str, missing: Sequence[str], policy_version: str) -> str:
    payload = {"change_id": change_id, "missing": list(missing), "policy_version": policy_version}
    b = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
    return _sha256_hex(b)


# Category -> (retry_after, escalate_after, owners)
_DEGRADE_SLO: Dict[DegradeReason, Tuple[int, int, Tuple[str, ...]]] = {
    DegradeReason.MISSING_CHANGE_ID: (0, 10 * 60, ("owner",)),
    DegradeReason.MISSING_TIME_CLAIM: (0, 10 * 60, ("owner",)),
    DegradeReason.MISSING_APPROVAL: (0, 30 * 60, ("owner", "security", "sre")),
    DegradeReason.MISSING_OBSERVABILITY: (0, 30 * 60, ("sre",)),
    DegradeReason.MISSING_ROLLBACK_PLAN: (0, 30 * 60, ("owner", "sre")),
    DegradeReason.MALFORMED_TIMESTAMP: (0, 10 * 60, ("owner",)),
    DegradeReason.DEPENDENCY_UNAVAILABLE: (5 * 60, 30 * 60, ("sre",)),
    DegradeReason.STATE_UNKNOWN: (5 * 60, 30 * 60, ("sre",)),
    DegradeReason.CONFLICT_DETECTED: (0, 30 * 60, ("owner",)),
}


def _merge_degrade_meta(change_id: str, missing: List[str], reasons: List[DegradeReason], policy_version: str) -> DegradeMeta:
    retry = 0
    esc = 30 * 60
    owners: List[str] = []

    for r in reasons:
        r_retry, r_esc, r_owners = _DEGRADE_SLO.get(r, (0, 30 * 60, ("owner",)))
        retry = max(retry, r_retry)
        esc = min(esc, r_esc)
        for o in r_owners:
            if o not in owners:
                owners.append(o)

    token = _make_resume_token(change_id, missing, policy_version)
    return DegradeMeta(
        missing=tuple(missing),
        retry_after_seconds=retry,
        escalate_after_seconds=esc,
        resume_token=token,
        owners=tuple(owners),
    )


def _validate_canary_steps(steps: Any) -> bool:
    if not isinstance(steps, list) or len(steps) < 2:
        return False
    try:
        ints = [int(x) for x in steps]
    except Exception:
        return False
    if any(p <= 0 or p > 100 for p in ints):
        return False
    if any(ints[i] >= ints[i + 1] for i in range(len(ints) - 1)):
        return False
    return ints[-1] == 100


def validate_change_request(doc: Dict[str, Any]) -> Verdict:
    reject: List[RejectReason] = []
    degrade: List[DegradeReason] = []
    missing: List[str] = []

    policy = doc.get("policy", {})
    policy_id = policy.get("policy_id")
    policy_version = policy.get("policy_version")
    if not policy_id or not policy_version:
        return Verdict(VerdictLevel.REJECT, reject_reasons=(RejectReason.PROD_CHANGE_POLICY_MISSING,))

    change = doc.get("change_request", {})
    approvals = doc.get("approvals", {})

    change_id = change.get("change_id")
    if not change_id:
        degrade.append(DegradeReason.MISSING_CHANGE_ID)
        missing.append("change_request.change_id")
        meta = _merge_degrade_meta("(missing)", missing, degrade, policy_version)
        return Verdict(
            VerdictLevel.DEGRADE,
            degrade_reasons=tuple(degrade),
            degrade_meta=meta,
            normalized_plan=(TypedAction(ActionName.CHANGE_REQUEST_MORE_INFO, {
                "change_id": None,
                "missing": list(meta.missing),
                "resume_token": meta.resume_token,
                "owners": list(meta.owners),
                "retry_after_seconds": meta.retry_after_seconds,
                "escalate_after_seconds": meta.escalate_after_seconds,
            }),)
        )

    # Missing approvals => DEGRADE
    if approvals.get("owner_approved") is not True:
        degrade.append(DegradeReason.MISSING_APPROVAL)
        missing.append("approvals.owner_approved")

    risk = change.get("risk_level", "MEDIUM")
    if risk in ("HIGH", "CRITICAL") and approvals.get("sre_approved") is not True:
        degrade.append(DegradeReason.MISSING_APPROVAL)
        missing.append("approvals.sre_approved")

    # Missing rollback/observability => DEGRADE
    if not change.get("rollback_plan_id"):
        degrade.append(DegradeReason.MISSING_ROLLBACK_PLAN)
        missing.append("change_request.rollback_plan_id")

    if not change.get("dashboard_id") or not change.get("alert_policy_id"):
        degrade.append(DegradeReason.MISSING_OBSERVABILITY)
        missing.append("change_request.dashboard_id / alert_policy_id")

    # Missing SLO gates is a design violation => REJECT
    guardrails = doc.get("guardrails", {})
    slo_gates = guardrails.get("slo_gates", [])
    if not isinstance(slo_gates, list) or len(slo_gates) == 0:
        reject.append(RejectReason.NO_SLO_GATES)

    # Canary config must be valid => REJECT
    canary = guardrails.get("canary", {})
    if not _validate_canary_steps(canary.get("step_percent", [])):
        reject.append(RejectReason.INVALID_CANARY_STEPS)

    try:
        step_wait = int(canary.get("step_wait_minutes", 15))
        if step_wait <= 0:
            reject.append(RejectReason.INVALID_STEP_WAIT)
    except Exception:
        reject.append(RejectReason.INVALID_STEP_WAIT)

    # Window parsing failures => DEGRADE, outside window => REJECT
    try:
        start = _parse_iso8601(change["window"]["start_at"])
        end = _parse_iso8601(change["window"]["end_at"])
    except Exception:
        degrade.append(DegradeReason.MALFORMED_TIMESTAMP)
        missing.append("change_request.window.start_at/end_at")
        start = end = None

    if not change.get("created_at"):
        degrade.append(DegradeReason.MISSING_TIME_CLAIM)
        missing.append("change_request.created_at")
        now = None
    else:
        try:
            now = _parse_iso8601(change["created_at"])
        except Exception:
            degrade.append(DegradeReason.MALFORMED_TIMESTAMP)
            missing.append("change_request.created_at")
            now = None

    if start and end and now and not (start <= now <= end):
        reject.append(RejectReason.OUTSIDE_CHANGE_WINDOW)

    if reject:
        return Verdict(VerdictLevel.REJECT, reject_reasons=tuple(reject))

    if degrade:
        meta = _merge_degrade_meta(change_id, missing, degrade, policy_version)
        return Verdict(
            VerdictLevel.DEGRADE,
            degrade_reasons=tuple(degrade),
            degrade_meta=meta,
            normalized_plan=(TypedAction(ActionName.CHANGE_REQUEST_MORE_INFO, {
                "change_id": change_id,
                "missing": list(meta.missing),
                "resume_token": meta.resume_token,
                "owners": list(meta.owners),
                "retry_after_seconds": meta.retry_after_seconds,
                "escalate_after_seconds": meta.escalate_after_seconds,
            }),)
        )

    return Verdict(VerdictLevel.ACCEPT)

3.4 The key point

When DEGRADE happens, the verifier returns:

degrade_reasons: what category of missing grounds this is
missing: exactly what is missing (paths)
normalized_plan: a typed “exit action” like change.request_more_info

Then the LLM can do what it’s good at: drafting messages and proposing next steps—bounded by a deterministic missing list.

4) DEGRADE turns exceptions into “operations”

DEGRADE doesn’t just reduce accidents. It converts exception handling into a process:

missing grounds are logged → improvable
requests become typed actions → automatable
“stopping” becomes normal → safer systems

If you can’t place DEGRADE, you can’t operate the agent—because most real incidents are “missing grounds,” not “clear violations.”

Summary

The most common real-world failure mode is missing grounds, not rule violations.
DEGRADE is not failure; it’s a safe, re-enterable stop.
Make DEGRADE operable with:
- reason codes
- missing lists
- DEGRADE-specific typed actions
- SLO metadata
- re-entry (join key + resume token + idempotency)
Let the LLM do remediation assistance, not eligibility decisions.

If you can’t design DEGRADE first, your agent won’t be operable.

Don’t “Execute” the LLM: Typed Actions + Verifiers for Safe Business Agents

kanaria007 — Mon, 30 Mar 2026 17:00:00 +0000

More AI “agents” now look like they work in real systems.

But what actually makes them work is not just model capability—it’s a deterministic verifier + operations that decides what’s allowed to run.

In a previous post I used a refund example.
In this one I’ll intentionally pick scarier scenarios—ones that make senior engineers’ blood run cold—and show a minimal design pattern:

Propose (probability) → Verify (determinism) → Execute (authority + audit)
…so you never “execute the LLM.”

This article is a minimal pattern. It’s not a complete product spec.

0) Why this design exists (the premise)

LLMs are probabilistic. Output variance itself isn’t the problem.
The real problem is executing wobbly output directly.

So split the roles:

LLM: propose a plan
Verifier: deterministically accept/reject (and optionally normalize the plan)
Executor: runs only verified Typed Actions (dry-run → approval → production)

The closer you get to “execute free text,” the more accidents you’ll have.
The more you close execution behind types, the more operable the system becomes.

Also: the LLM’s “explanation-like text” (including chain-of-thought style reasoning) is not audit-grade grounds.
What you should pin as grounds are: input schema, policy ID/version, rule-evaluation logs, evidence/trace IDs.
Explanations are at best UI helper text.

1) The minimal architecture that removes most accidents

1) Structure the input (Schema)

Fix “what counts as grounds.”
Free text becomes optional notes; decision inputs are typed.

2) Make output Typed Actions

The LLM outputs an action list (a plan), not prose.

3) Verifier returns ACCEPT / REJECT / DEGRADE

ACCEPT: verified OK → return normalized plan
REJECT: rule violation → return reason codes → LLM can re-propose
DEGRADE: insufficient grounds → branch to “request missing info” (LLMs are good at this)

4) Execute only a verified plan

The executor accepts nothing but typed actions.
That’s the guardrail core.

flowchart LR
  I[Schema Input] --> LLM[LLM: propose plan]
  LLM --> P[Plan: Typed Actions]
  P --> V[Deterministic Verifier]
  V -->|ACCEPT| E["Execute (dry-run/approve/prod)"]
  V -->|REJECT| R[Return reason codes]
  V -->|DEGRADE| D[Request missing evidence]

2) Thought experiment 1: Just-In-Time production access (JIT Access)

“Need to read the prod DB during an incident.”
“Need to temporarily change a prod setting.”

If you let an LLM decide this, responsibility boundaries collapse—and everyone’s stomach drops.

So split roles:

LLM: summarize the situation, list missing info, propose a plan, draft request messages
Compiler-equivalent: deterministically decide eligibility / least privilege / expiration / approval / SoD
Executor: runs only approved typed actions (dry-run → approval → grant → auto-revoke)

1) Input schema (grounds)

access_request:
  request_id: "AR-2026-00077"
  requester_user_id: "u-1234"
  target_resource: "prod-db:billing"
  requested_role: "db.readonly"
  requested_duration_minutes: 60
  reason_code: "INCIDENT_RESPONSE"
  incident_id: "INC-88921"
  ticket_id: "T-2026-004512"
  created_at: "2026-02-15T22:10:00+09:00"

approvals:
  manager_approved: true
  security_approved: false      # if required, false => DEGRADE/REJECT
  approver_user_ids: ["u-9001"]

context:
  requester_team: "SRE"
  requester_level: "L2"
  on_call: true
  break_glass: false

policy:
  policy_id: "iam-jit-access"
  policy_version: "2026-01-20"

evidence:
  runbook_id: "rbk-prod-db-read"
  logs_link: "log://..."

2) Typed Actions (execution closed by type)

iam.grant_temporary_role
iam.revoke_role
ticket.comment
audit.append
access.request_more_info (for DEGRADE)

3) Deterministic rules (examples)

Required grounds: missing incident_id / ticket_id / policy_version → DEGRADE
Approvals: manager approval required; security approval required for certain roles → missing → DEGRADE
Two-person approval: unless break_glass, fewer than 2 approvers → DEGRADE
Least privilege: db.admin forbidden; only allowed in break_glass → REJECT
Expiration: max 120 minutes; unspecified → DEGRADE
SoD (segregation of duties): SRE must not hold “refund/payment-change” privileges on billing prod DB → REJECT
Auto-revoke required: if you can’t schedule revoke at grant time → REJECT

4) LLM outputs a plan proposal only

{
  "actions": [
    {
      "name": "iam.grant_temporary_role",
      "params": {
        "user_id": "u-1234",
        "resource": "prod-db:billing",
        "role": "db.readonly",
        "duration_minutes": 60
      }
    },
    {
      "name": "ticket.comment",
      "params": {
        "ticket_id": "T-2026-004512",
        "message": "JIT access requested for incident INC-88921. Duration 60m. Role db.readonly."
      }
    },
    {
      "name": "iam.revoke_role",
      "params": {
        "user_id": "u-1234",
        "resource": "prod-db:billing",
        "role": "db.readonly"
      }
    }
  ]
}

The verifier returns ACCEPT/REJECT/DEGRADE, and only the normalized plan is passed to execution
(e.g., normalize resource names, substitute role aliases, clip duration, force-add revoke).

3) Thought experiment 2: production change management (Change Management)

Feature-flag rollouts and config changes can look small but have massive blast radius.

Letting an LLM decide “is it okay to change prod?” is terrifying.
So again:

LLM: summarize, scope impact, propose execution plan, draft announcements
Compiler-equivalent: deterministically enforce approvals, canary steps, SLO gates, rollback readiness, time windows
Executor: runs only verified plan (dry-run → approval → staged rollout → monitoring → auto-rollback if needed)

1) Input schema (grounds fixed)

change_request:
  change_id: "CHG-2026-00112"
  requester_user_id: "u-1234"
  service: "billing-api"
  environment: "prod"
  change_type: "feature_flag_rollout"   # or "config_change"
  flag_key: "new_invoice_flow"
  from: { enabled: false, percent: 0 }
  to:   { enabled: true,  percent: 10 }
  window:
    start_at: "2026-02-16T01:00:00+09:00"
    end_at:   "2026-02-16T03:00:00+09:00"
  risk_level: "MEDIUM"
  rollback_plan_id: "rb-2026-0091"
  runbook_id: "rbk-billing-rollout"
  ticket_id: "INC-90121"               # incident link or change ticket
  created_at: "2026-02-15T22:30:00+09:00"

approvals:
  owner_approved: true
  sre_approved: false
  security_approved: false
  approver_user_ids: ["u-9001"]

guardrails:
  canary:
    step_percent: [10, 25, 50, 100]
    step_wait_minutes: 15
  slo_gates:
    - metric: "error_rate_5m"
      op: "<="
      threshold: 0.01
    - metric: "p95_latency_ms_5m"
      op: "<="
      threshold: 400
  rollback:
    auto_rollback_enabled: true
    rollback_on:
      - gate_violation
      - manual_trigger

observability:
  dashboard_id: "dash-billing-api"
  alert_policy_id: "alert-billing-slo"

policy:
  policy_id: "prod-change-policy"
  policy_version: "2026-01-10"

2) Typed Actions

change.plan.publish (dry-run preview)
feature_flag.set_percent
config.apply
slo_gate.check
rollback.execute
audit.append (store gate results + thresholds + timestamps)
change.request_more_info (DEGRADE)
change.block (REJECT)

3) Deterministic rules (examples)

Owner approval required: owner_approved=false → DEGRADE (request approval)
Prod window constraint: outside allowed window → REJECT (re-propose timing)
Staged rollout required: only 100% as a single step → REJECT
SLO gates required: slo_gates empty → REJECT
Rollback required: missing rollback_plan_id → DEGRADE
Observability required: missing dashboard_id / alert_policy_id → DEGRADE
High risk requires extra approval: risk_level=HIGH and sre_approved=false → DEGRADE

4) LLM outputs a staged plan proposal only

{
  "actions": [
    {"name": "change.plan.publish", "params": {"change_id": "CHG-2026-00112"}},
    {"name": "feature_flag.set_percent", "params": {"flag_key": "new_invoice_flow", "percent": 10}},
    {"name": "slo_gate.check", "params": {"window_minutes": 15}},
    {"name": "feature_flag.set_percent", "params": {"flag_key": "new_invoice_flow", "percent": 25}},
    {"name": "slo_gate.check", "params": {"window_minutes": 15}}
  ]
}

The verifier can stop on missing grounds (DEGRADE) or normalize the plan
(e.g., force-add rollback.execute, clip percent limits).

4) Thought experiment 3: personal data erasure (GDPR-like requests)

Erasure is not “just delete stuff.” The core is:

eligibility / exceptions
legal holds
evidence
irreversible deletion mechanics

You never want an LLM to decide deletion eligibility. So:

LLM: enumerate systems, propose steps, draft user-facing messages
Compiler-equivalent: deterministically enforce eligibility, verification, holds, retention rules, evidence bundling
Executor: runs only verified plan (dry-run → execute → bundle evidence)

1) Input schema (grounds)

erasure_request:
  request_id: "DEL-2026-0042"
  subject_user_id: "C-9182"
  subject_email: "user@example.com"
  received_at: "2026-02-15T19:05:00+09:00"
  request_channel: "web_form"
  identity_verification:
    method: "email_otp"
    verified: true
    verified_at: "2026-02-15T19:10:00+09:00"

holds:
  legal_hold: false
  fraud_investigation_hold: false
  accounting_retention_required: true  # retention required by accounting / law

scope:
  systems:
    - "crm"
    - "billing"
    - "support_tickets"
    - "analytics"
  data_categories:
    - "profile"
    - "support_messages"
    - "payment_metadata"              # may be non-deletable due to retention

policy:
  policy_id: "privacy-erasure-policy"
  policy_version: "2026-01-05"

audit:
  ticket_id: "PRIV-2026-0018"
  operator_user_id: "u-privacy-01"

2) Typed Actions

privacy.assess.deletion_eligibility
privacy.redact (mask/anonymize instead of delete)
privacy.delete
privacy.tombstone.write (tombstone to prevent restoration/double-processing)
legal.hold.apply / legal.hold.check
audit.append
subject.notify
privacy.request_more_info (DEGRADE)

3) Deterministic rules (examples)

Identity verification required: verified=false → DEGRADE
Legal hold: legal_hold=true → REJECT (deletion forbidden)
Retention-required categories: auto-convert delete → redaction (normalization)
Tombstone caution: do not store PII in tombstones; use irreversible hash/pseudo IDs
Audit fields required: missing policy_version / ticket_id → DEGRADE

4) LLM proposes a plan (delete + redact)

{
  "actions": [
    {"name": "privacy.assess.deletion_eligibility", "params": {"subject_user_id": "C-9182"}},
    {"name": "privacy.delete", "params": {"system": "crm", "subject_user_id": "C-9182"}},
    {"name": "privacy.redact", "params": {"system": "billing", "subject_user_id": "C-9182", "fields": ["name", "email"]}},
    {"name": "privacy.tombstone.write", "params": {"subject_user_id": "C-9182", "request_id": "DEL-2026-0042"}},
    {"name": "audit.append", "params": {"ticket_id": "PRIV-2026-0018"}}
  ]
}

What matters is not the LLM’s prose, but having auditable records of:

which policy/version
what was deleted vs retained
what tombstone was created

5) Thought experiment 4: credit underwriting (loans, etc.)

Underwriting is the clearest “do not let an LLM decide” domain.

This section is not advocating automated approval/denial.
It’s about designing a process so the LLM cannot become the decision maker.

You have institutional constraints:

adverse action reasons (explainability obligations)
discrimination risk (protected attributes / proxies / disparate impact)
regulation, audits, model governance

So split even harder:

LLM: structure application data, detect missing documents, draft instructions, organize the “underwriting packet”
Compiler-equivalent: decisions only via deterministic policy (rulebook / scorecards / audit logic)
Executor: if missing → DEGRADE; decisions emitted with audit-grade reason codes

1) Input schema (underwriting packet)

application:
  application_id: "APP-2026-00991"
  applicant_id: "A-5512"
  product: "personal_loan"
  requested_amount_jpy: 500000
  term_months: 24
  submitted_at: "2026-02-15T18:00:00+09:00"

documents:
  identity_verified: true
  income_proof:
    provided: true
    doc_id: "DOC-INC-7712"
  employment_proof:
    provided: false
  bank_statements:
    provided: true
    doc_id: "DOC-BANK-1142"

features:
  monthly_income_jpy: 320000
  employment_years: null
  existing_debt_jpy: 800000
  delinquency_flags:
    last_12m: 0

policy:
  policy_id: "credit-underwriting-policy"
  policy_version: "2026-01-01"

fairness_controls:
  prohibited_fields_present: false
  proxy_feature_flags:
    - "zip_code"         # potential proxy; usage must be deterministically governed
  audit_required: true

audit:
  case_id: "UW-2026-00440"
  operator_user_id: "u-uw-01"

Key point: do not let protected attributes (or inferred attributes) flow into the LLM decision path.
If they appear, block them at ingestion and make it auditable.

2) Typed Actions

uw.request_more_documents (DEGRADE)
uw.compute_scorecard (deterministic)
uw.apply_policy_rules (deterministic thresholds/exceptions)
uw.emit_decision (ACCEPT/REJECT/DEGRADE + reason codes)
audit.append
applicant.notify (LLM may draft, but reasons must come from reason codes)

3) Deterministic rules (examples)

Missing documents: employment_proof missing → DEGRADE
Prohibited/proxy misuse: prohibited/proxy features mixed into decision inputs → REJECT (process violation)
Decision requires reason codes: free-text-only reasons → REJECT
Explainability: denial must attach adverse action reason codes (deterministic)
Audit required: if audit_required=true, inability to produce an audit bundle (inputs/rules/outputs) → REJECT

4) LLM only does “missing detection” + guidance (no decision)

Example proposal (routes to DEGRADE):

{
  "actions": [
    {"name": "uw.request_more_documents", "params": {"application_id": "APP-2026-00991", "missing": ["employment_proof"]}},
    {"name": "audit.append", "params": {"case_id": "UW-2026-00440", "note": "Employment proof missing; request additional documents."}}
  ]
}

Approval/denial must come only from deterministic uw.compute_scorecard + uw.apply_policy_rules.

6) In real operations, DEGRADE matters more than REJECT

Across these examples, the most common failure mode is not rule violations—it’s missing grounds:

missing documents
missing data
uncertain state
missing policy prerequisites

So what you want the LLM to do is not “decide eligibility,” but:

identify missing information
draft evidence requests
propose next procedural steps

And what you want the verifier to do is:

detect missing grounds and stop safely (re-enterably).

If you can’t design DEGRADE first, the agent won’t be operable.

7) Common failure patterns (avoid these)

Avoiding these alone reduces accidents drastically:

executing free text (sending emails / creating tickets / updating DB rows)
fully delegating tool-call parameters to the LLM
treating unverified output as “success” (no durable logs)
only having REJECT paths, so humans must rescue everything via interpretation

Summary

LLMs are proposers. What businesses need is a deterministic verifier.
Close execution behind Typed Actions, and run only verified plans.
DEGRADE (deferral) design is as important as REJECT.
Even a minimal split—Propose → Verify → Execute—turns an “agent” into a reliable operational component.

Yes, I picked intentionally scary examples. But we’re also seeing more products trying to put AI at the core.

If you actually want probabilistic AI to touch the core of business operations, you’ll need “hard” domain design—responsibility boundaries, policies, and a serious rule-based domain compiler.

That’s a difficult and sometimes messy path—but also the kind of engineering that can be genuinely interesting.

Still… personally, I don’t want current LLMs anywhere near credit underwriting decisions.

Determinism Series: Siliconizing Decision-Making (Index)

kanaria007 — Sat, 28 Mar 2026 06:54:17 +0000

The goal of this series is simple:

If a decision can be turned into rules, we should “siliconize” it.
That is: move decisions from “interpretation-driven” to deterministic, replayable judgments.

“Determinism” here is not about “the correct time” or “powerful magic.”
What we really want is:

Ordering: don’t break which came first
Replay: later, with the same input + grounds + procedure, return to the same conclusion

Shared principles across the series

0) Label the target first (RML: which world is this decision in?)

Whether a decision can be “siliconized” is often determined before implementation—by the world (responsibility scope) the decision belongs to.

This series assumes a “world label” up front:

RML-1 (Closed World): not outside yet / safe to fail (simulation, dry-run)
RML-2 (Dialog World): counterparties exist / recover via dialog + compensation (sagas, retries, reconcile)
RML-3 (History World): irreversible history / correct-forward with refunds, explanations, audits (ledger)

Once you fix:

“Which world does this decision live in?”

…it becomes much harder for the design to drift:

how far you may automate
where human approval or correction events are needed
what must be kept as “non-erasable logs”

1) Split proposing from deciding

Whether it’s an LLM or a human, proposals can wobble—and that’s fine.
What must be stable is the side that decides accept/reject:

Propose (probabilistic): LLM/humans produce a plan
Verify (deterministic): rules return ACCEPT / REJECT / DEGRADE
Execute (authority + audit): only verified, typed operations may pass

2) Make “insufficient information” first-class (DEGRADE)

In real operations, the common failure is not “rule violations,” but missing grounds.

So the first branch to design is not REJECT, but:

DEGRADE (defer / hold / request missing)

DEGRADE is not “stop.” It’s stop in a re-enterable way:
enumerate missing fields mechanically, and make it operable under SLOs.

3) Pin the grounds into logs (make incidents replayable)

A “plausible explanation” is not auditable grounds.
What must be pinned is:

input schema
policy/version
reason codes
evidence IDs
an authority for ordering

Reading order (recommended)

From top to bottom, it naturally flows: agents → operations → distributed foundations.

1) The Real Reason AI Agents “Work” in Software (series intro)

Why “agents succeed” in software is not model intelligence, but deterministic guardrails (verifiers) already baked into the environment.

2) Don’t “Execute” the LLM: Typed Actions + Verifiers for Safe Business Agents

Never execute free text. Use Typed Actions + a deterministic verifier to prevent operational accidents.

3) Design DEGRADE (Defer) and Your Agent Becomes “Operations”

Once you can handle “missing,” you eliminate interpretation hell and turn workflows into re-enterable operations.

4) Grow the Verifier, Not the Prompt: Run Production with 10 Golden Cases

What must be stabilized is not the LLM output, but the verifier output:
verdict / reason / missing / normalized_plan. Includes how to “grow” it under CI.

Domain application (turn determinism into business reality)

5) Billing That Won’t Break: Deterministic Design to Kill Double Charges (API Contract × DB Boundary × Execution Patterns)

In “fatal if broken” billing domains, push design toward ordering, idempotency, auditability:
API contracts × DB boundaries × execution patterns.

6) Make A/B Tests Operable: Goal Surface × Deterministic Logs × Safe Automation (dry-run / shadow / canary / rollout)

Turn experiments from “one-off analyses” into operations:
connect multi-objective + constraints (goal surface) with DEGRADE, watermarking, and pinned logs.

Foundations (don’t break ordering and replay)

7) Deterministic Time: Don’t Break Ordering and Replay in Distributed Systems

Even without TrueTime, you can build “ordering authority” via seq / watermark / HLC.
Also covers late data, trace closure, and dual writes (outbox, event sourcing).

Series checklist (when you’re stuck, come back here)

If discussions don’t align: Which world is it (RML-1/2/3)?
If decisions wobble: did you separate proposal and verification?
If you keep getting blocked by missing info: are DEGRADE and missing first-class?
If it turns into meetings: are reason codes typed?
If you can’t explain it later: are inputs / policy / snapshots / ordering pinned to logs?
If long-running jobs break: are they state-machine based + re-enterable? (Fix dual writes via outbox / event sourcing.)
If ordering breaks in distributed systems: which authority did you choose—seq / watermark / HLC?

This determinism series is a parts kit for verification, evidence, and operations—but in practice, you often need to label the target world first.

RML-1: grow verifiers safely inside a room
RML-2: make workflows re-enterable via dialog + compensation (DEGRADE / action hints / retries)
RML-3: keep history, reconcile with correction events (ledgers / audits / explanations)

Once the “world” is aligned, it becomes much easier to see where you need:
deterministic logs, DEGRADE, watermarking, outbox/event sourcing, and more.

Closing

This series is not about “perfecting everything.”
It’s a parts kit for siliconizing the most fragile decisions first.

Start with just:

a minimal verifier, and
minimal pinned logging

Then grow it using SLOs and reason codes.

The long-term endgame is a system that can make decisions autonomously—and when it hits a wall, it uses the failure as training data:

unexpected events or missing inputs are recorded in deterministic logs,
the next step becomes cheaper to decide,
and if the decision is rule-able, you update the domain compiler and golden cases so the next run becomes automated.

Pushing full automation that far isn’t easy—but the operational load can be managed as engineering:
SLOs, reason codes, and staged adoption, not heroic effort.

At first, “Verifier + pinned logs” already pays off.
When you want more, add in order: domain compiler → golden case updates → proposal generation (LLM).

Epilogue — Toward Engineering with a Worldview

kanaria007 — Mon, 23 Mar 2026 15:00:00 +0000

When people write books about distributed systems, one of the first temptations is:

“I want to start with the coolest algorithm.”

A new consensus protocol.
A scalable queueing strategy.
A clean event schema.

But this book did almost the opposite.

Again and again, before we talked about what to use, we asked:

Which world are we operating in?

That question came first—every time.

1) What it means to have a “worldview”

I put Worlds in the title because the real goal wasn’t just technical knowledge.

I wanted to install a worldview for distributed systems.

When you:

write code,
design an API,
set alerts,
change Terms of Service,
respond to an incident,

…if, for just one second, you pause and ask:

“Which world am I touching right now?”

Then this book has already done half its job.

2) Not “the right answer,” but “the right question”

RML-1/2/3 labels,
Runbooks vs Playbooks,
Effect Ledgers, case-file templates—

None of these are “do it this way.”

Because the best approach changes completely depending on:

the company,
the domain,
the legal system,
and the team’s skill set.

What this book can realistically offer is closer to:

A shape of questions worth asking at least once.

Questions like:

“Is this feature really safe to keep at RML-2?”
“Is this incident OK to leave out of the History World?”
“Does the engineering team understand what this ToS clause implies for design?”

If those questions casually show up in reviews or hallway conversations, that alone is enough.

3) Real life still needs “make it work for now”

Of course, real teams live in reality:

you need a PoC before anyone will listen,
you need revenue for the org to survive,
and sometimes you need a patch more than a perfect design.

RML isn’t here to deny that.

It’s a small proposal:

If you’re going to “make it work for now,”
at least be conscious of which world you’re sacrificing.

Are you still contained in RML-1?
Have you spread into RML-2?
Have you already stepped into RML-3?

That single awareness changes both:

how much damage you take when things go wrong, and
how well you learn afterwards.

4) I want you to create your version of RML

In the book, for convenience, we used:

RML-1 / RML-2 / RML-3
Closed World / Dialog World / History World

But the names don’t matter.

Level A / B / C is fine.
World α / β / γ is fine.
Domain-specific labels are often better (“Internal World / Customer World / Society World,” etc.).

What matters is this:

Build a vocabulary that fits your product and organization perfectly.

You don’t need to import the book as-is.

Delete concepts.
Add worlds.
Take only the chapters you need.

The real goal is:

Create your own “Worlds of Distributed Systems.”

5) The chapters you’ll write next

The book ends at Chapter 11.

But the real work begins after that:

in someone’s design doc,
in the corner of an incident report,
on a workshop whiteboard—

where RML, History World, or Runbook/Playbook tables reappear in new forms.

Each time that happens in your environment:

that becomes your Chapter 12, Chapter 13, and beyond.

6) Final words

If you read this far, you’re probably the kind of person who can’t help but care about:

not just distributed system behavior,
but also responsibility, history, and organization.

In a good way.

Teams with more people like that tend to grow systems with a real worldview—systems that age well.

If this book helped you put even a small amount of language around that “I can’t stop thinking about responsibility” instinct, then that’s exactly what I hoped for.

Wishing your own Worlds of Distributed Systems grows into something solid and trustworthy.

Chapter 11 — A Field Recipe for RML: Start Small, Grow It

kanaria007 — Thu, 19 Mar 2026 15:00:00 +0000

The Worlds of Distributed Systems — Chapter 11 (Practical Adoption Guide)

“Nice theory… but what do we do tomorrow?”

After Chapter 10, the worldview may feel right—but you might still be unsure about:

where to start,
how far is “too much,”
and how to talk about this across teams.

This chapter is not “more theory.”
It’s a recipe / adoption guide for bringing RML into real work:

You can’t do everything at once.
But you can introduce it gradually—without overhauling the org.

1) Five tiny steps you can start tomorrow

These are intentionally small. You can do them as an individual—even without “organizational permission.”

1.1 Say the word “world” once in a conversation

In a design review or incident retrospective, ask this one question:

“Which world are we talking about—RML-1, RML-2, or RML-3?”

It’s OK if nobody has an answer yet.
The point is to flip the mental switch:

Make “world awareness” part of the discussion.

1.2 Add an `RML` column to a single backlog list

In Jira, Notion, a spreadsheet—anything.

Pick one epic or feature list.
Add a column called RML.
Write 1 / 2 / 3 for a few items, using your best guess.

Later, when you review it with the team, it becomes a productive “discussion starter”:

“Why is this RML-2?”
“Is this actually RML-3 once it touches money/contracts?”

1.3 Add `world` to a new exception class (just once)

Any language is fine. Keep it lightweight.

class RmlError extends Error {
  world: "RML1" | "RML2" | "RML3";

  constructor(args: { world: "RML1" | "RML2" | "RML3"; message: string }) {
    super(args.message);
    this.world = args.world;
  }
}

You don’t need perfect handling on day one.

Even just writing down:

“This throw assumes which world?”

…changes how engineers reason about failures.

1.4 Retrospect one recent incident with an RML tag

Don’t try to rewrite your entire incident process.

Take a single recent incident and add world markers to the timeline:

“Up to here was RML-2…”
“This point is where it first became RML-3.”

Just drawing that boundary makes the “real weight” visible.

1.5 Write one Runbook and one Playbook (one page each)

From Chapter 9:

Runbook = RML-2 (technical operations)
Playbook = RML-3 (organizational decision-making)

Pick one representative scenario for each:

Runbook: “Payment gateway outage (containment + retries + compensation)”
Playbook: “Incorrect billing occurred (who decides refunds + comms + legal steps)”

Two pages can align the team surprisingly fast.

2) Role-based “do only this” adoption guide

RML becomes easier when each role contributes a small part.

2.1 Application engineers

Add world / action to error types (even minimally)
Add idempotency keys to critical external calls
Ask once per feature: “What’s the RML of this?”

Goal:
Make RML appear in code and design reviews.

2.2 SRE / Platform engineers

Add rml.world / rml.action tags to logs/metrics/traces
Create one alert based on RML-3 signals
Prepare one Runbook (RML-2) + one Playbook (RML-3)

Goal:
Make RML flow into dashboards and incident response.

2.3 Product managers / POs

Allow (or encourage) adding an RML column to the backlog
Put “promotion” into roadmap discussion:
- “Do we promote this to RML-3 next quarter?”
Roughly track RML-3 costs (refunds, coupons, support time)

Goal:
Connect RML to product strategy and P&L reality.

2.4 Legal / Compliance

Define (with engineers + business) what counts as RML-3
Align ToS/contract compensation terms with actual RML-3 operations
Join at least one postmortem for a history-grade incident

Goal:
Align language for how the org enters the History World.

3) A sample RML kickoff meeting agenda (90 minutes)

If you want a “lightweight organizational start,” do one kickoff meeting.

Agenda (90 minutes)

Intro (10 min)

what RML is, in one slide
what you’ll call it internally (RML is fine)

Case sharing (20 min)

pick 1–2 recent incidents or near-misses
draw a timeline
mark where it shifts: “RML-2 → RML-3”

Role perspectives (20 min)

engineers: rollback/retry/compensation reality
business: customer experience + brand impact
legal: contracts/regulation + disclosure obligations

Draft “our org’s RML rules” (30 min)

list default RML-3 cases
define a “when unsure, do X” principle
pick the Playbook owner for RML-3 incidents

Next step (10 min)

choose one service/feature to start labeling
choose one incident type to apply the case-file template

The trick is not perfection.

Treat it as a meeting to decide the minimum that lets you move tomorrow.

4) FAQ (the questions teams actually ask)

Q1) “Our domain is basically all RML-3… doesn’t that make RML useless?”

In finance/health/public infrastructure, it can feel like “everything is History World.”

Even then, separating helps if you split:

World as a processing layer
World as a responsibility boundary

Example:

“Showing appointment candidates” → RML-1 (proposal-only)
“Reserving a slot” → RML-2 (adjustable via operations)
“Medical records / billing / official notes” → RML-3 (legal history)

Even in an RML-3 domain, you usually still have RML-1/2 layers inside it.

Q2) “Is labeling RML wrong itself risky?”

Early on, you will mislabel. That’s normal.

Treat labels as:

Best current hypotheses, updateable over time.

What’s riskier is the default state:

“We don’t know which world we’re in, so we guess mid-incident.”

It’s safer to label, learn, and refine than to operate without a shared map.

Q3) “We’re a small startup—we can’t do all this.”

Small orgs often need RML more, not less—because you can’t afford repeated history-grade mistakes.

Start with minimal scoping:

“Payments are RML-3 and we treat them seriously.”
“Everything else stays RML-2 unless proven otherwise.”

Early boundaries make later growth easier.

5) A small homework assignment

One small exercise to make this “your team’s book”:

Write down three RML-3-ish events in your environment.

Past incidents or imagined “worst days” are both fine.

For each, write:

what happened,
who was affected,
who owned responsibility,
what “history residue” remains.

Those three items become page one of:

Your team’s version of The Worlds of Distributed Systems.

That’s the end of the main content of this book—for now.

Epilogue — Toward Engineering with a Worldview

From here, you can:

remove chapters,
rename terms,
or refactor the entire worldview into your own.

If it helps you build systems that don’t just “work most of the time,” but are designed for trust, then the map did its job.

Chapter 10 — RML as Product Strategy: Designing Trust

kanaria007 — Mon, 16 Mar 2026 15:00:00 +0000

The Worlds of Distributed Systems — Chapter 10

“Will this feature work reliably?”
“Yes—but it depends on which world we’re willing to own.”

So far in this series:

Chapters 1–4 drew the map of the three worlds,
Chapters 5–7 drilled into RML-2 (Dialog World) patterns (failures, sagas, APIs),
Chapters 8–9 focused on RML-3 (History World) autonomy and case files.

In Chapter 10, we zoom out and place RML at the level that decides everything downstream:

RML = a “trust world map” for distributed systems.

Not just an engineering model—but a product strategy tool.

1) Use RML as a world map (not a “maturity level”)

RML is often read as “how mature your rollback is.”

But strategically, it’s more useful as:

A map of how far your product takes responsibility.

A compact restatement:

RML-1 — Closed World
Inside one room: tests, dry runs, simulations.
“A world where you can try freely because you can restart safely.”
RML-2 — Dialog World
Cross-service recovery: retries, compensation, reconciliation.
“A world where you can recover through conversation.”
RML-3 — History World
Money, law, social trust, irreversible records.
“A world where you correct forward, and you must be accountable.”

Now the strategic question becomes:

“Which world does this product step into?”
“Which world do we intentionally not step into (and delegate to others)?”

Once you answer that, these decisions become far easier to align:

feature scope and granularity
SLA / SLO promises
Terms of Service (ToS) and compensation boundaries
incident response structure and cross-functional ownership

2) Add an “RML column” to your product backlog

In Chapter 7, we put RML into APIs.
At the strategy level, an even earlier move works surprisingly well:

Add an RML column to your feature backlog.

2.1 A simple table

Feature	Description	RML	Notes
AI inference simulation	Show results only; don’t persist	1	production reads only
Internal review status update	internal users; DB updates	2	reversible via ops
Credit card payments	payment gateway + customer assets	3	refunds required
Tax-grade PDF export	may be submitted to authorities	3	versioning + correction log

Just adding this column changes the review conversation:

“Is this really RML-2, or is it actually RML-3?”
“If it’s RML-3, where is the refund/correction/explanation path?”

2.2 Feature “promotion” as a roadmap concept

You don’t need to build everything as RML-3 on day one.
A natural evolution looks like:

Prototype: RML-1 (simulation only)
Alpha: RML-2 (internal use / limited customers)
GA: promote some flows into RML-3

If you make this an explicit roadmap question:

“Do we promote this feature to RML-3 next quarter?”

…then planning becomes concrete:

when Legal must be involved
when sagas and idempotency must be redesigned properly
when you need an Effect Ledger and case files

3) RML and metrics: measuring “trust” by world

Product strategy always collapses into metrics.

With RML, you can separate “health” into three layers.

3.1 RML-1 metrics (Closed World)

Focus: Are we able to try safely?

test coverage / number of simulations
dry-run execution count and failure patterns
bugs discovered in RML-1 before production rollout

3.2 RML-2 metrics (Dialog World)

Focus: How much recovery can we automate through conversation?

saga success rate / compensation activation rate
RML-2 error counts (world = RML2) by code
auto-retry resolution rate vs human ops intervention rate
% of critical external calls with idempotency keys

3.3 RML-3 metrics (History World)

Focus: How often do we generate irreversible trust-cost events?

RML-3 incident count (quarterly)
per-incident:
- affected users
- monetary cost (refunds, credits, ops time)
time from detection → containment (MTTD / MTTC)
recurrence rate (same incident class reappears)

At this point, metrics connect directly to:

P&L (refunds and support cost)
brand impact
trust and retention

3.4 Split dashboards by world

Even a simple dashboard split changes management discussions:

“Error rate is high but RML-3 incidents are zero” → likely a Dialog World design/ops improvement problem
“Error rate is low but RML-3 incidents are rising” → your History World governance (Ch.8–9) is the real gap

4) RML across the product lifecycle

4.1 Planning & design

put RML labels into backlog and specs
decide “how far we own the world” as part of the requirements
for RML-3 candidates, involve Legal early

4.2 Implementation

Apply Dialog World patterns (Ch.5–7):

structured errors with world/action
idempotency keys + saga discipline
RML metadata in API boundaries
tests that validate promotion conditions (“when does this become RML-3?”)

4.3 Release & operations

observability tags: rml.world, rml.action
RML-3 alerts create incidents automatically
maintain Runbooks (RML-2) and Playbooks (RML-3)

4.4 Sunset & replacement

RML matters even when you shut things down:

retiring an RML-3 feature:
- what happens to the history (records, ledgers, retention)?
migrating systems:
- how do you shift RML-2 dialogs (APIs/sagas) safely?
- how do you share RML-3 responsibility during the transition?

A useful sentence for planning:

“Sunsetting a feature means withdrawing from a world.”

That changes the granularity of your sunset plan.

5) The minimum viable adoption set

If this feels heavy, you can adopt RML in stages.

Step 1: adopt the vocabulary only

say “this is RML-1 / RML-3-ish” in reviews
add an RML column to backlog
add an RML-World field to incident reports

No implementation changes needed yet.
You’re aligning the meta-language first.

Step 2: implement the minimum RML-2 patterns

add world/action to your structured error type (e.g., RmlError) (Ch.5)
add idempotency keys to critical external calls (Ch.6)
add X-RML-World / X-RML-Action to API responses (Ch.7)

Now you can:

tag telemetry by world
alert on RML-3-ish signals

Step 3: use case files only for confirmed RML-3 incidents

if the org agrees “this is RML-3,” then always write the Ch.9 case file.

This gives you visibility into:

history-grade costs (refunds, credits)
where your bottleneck is (detect/contain/decide)
where your contracts/runbooks/playbooks are drifting

6) Strategy-level anti-patterns

6.1 Treating RML as a compliance label only

If RML becomes “audit paperwork,” it disconnects from real engineering and ops.

Fix:

always connect it to implementation patterns (Ch.5–7) and governance (Ch.8–9)

6.2 “World inflation” (everything becomes RML-3)

If everything is treated as History World:

the org becomes permanently incident-fatigued
decision loops slow to a crawl

Fix:

pre-agree on “default-to-RML-3” conditions (Ch.8)
handle the rest in RML-2, and promote only when needed

6.3 Shipping a new business without deciding which world you own

A common failure mode:

launch quickly
the History World boundary is left undefined
incidents arrive before governance exists

Fix:

require a slide in every planning review:

“Which world does this business step into, and which worlds does it avoid?”

7) Final checklist: do you have a worldview?

7.1 World mapping

[ ] Can you label major features as RML-1/2/3?
[ ] Are RML-3 features agreed with Legal and Business?
[ ] Is “promotion” (RML-2 → RML-3) part of roadmap discussions?

7.2 Implementation & ops

[ ] StructuredError / ActionHint includes world/action
[ ] critical external calls use idempotency keys
[ ] observability supports rml.world / rml.action
[ ] APIs carry RML metadata (REST/GraphQL/gRPC)

7.3 Governance & incidents

[ ] RML-3 incident definition and escalation rules are documented
[ ] Legal/Business/SRE triangle ownership (RACI) is explicit
[ ] case file template includes world + Decision Log
[ ] Runbook (RML-2) and Playbook (RML-3) exist

7.4 Business & trust

[ ] You can estimate RML-3 incident costs (roughly)
[ ] ToS compensation scope matches actual RML-3 behavior
[ ] Reducing RML-3 incidents is treated as a strategic objective

Closing — “Which world is this rollback?”

Distributed systems discussions quickly become technical:

consistency models
queues and retries
sagas and compensation
APIs, gRPC, messaging…

All of that matters.

But the high-leverage pause—the one this series keeps returning to—is:

“Which world are we talking about?”

RML-1: a world where you can try safely
RML-2: a world where you recover through dialog
RML-3: a world where you carry history and correct forward

When the world changes:

rollback changes meaning
incident weight changes
product responsibility changes

So the next time you’re about to say:

“We can roll it back,” or
“We can compensate,”

pause for one beat and ask:

“Rollback in which world—
and which world’s history are we asking whom to carry?”

That question alone tends to upgrade a distributed system from “works most of the time” into something closer to designed trust.

Chapter 11 — A Field Recipe for RML: Start Small, Grow It

Chapter 9 — RML-3 Case Files: Aligning Your Incident Response Worldview

kanaria007 — Thu, 12 Mar 2026 15:00:00 +0000

The Worlds of Distributed Systems — Chapter 9

“When an incident happens…
what is supposed to be happening behind the screen?”

In Chapter 8, we reframed RML-3 (History World) as an autonomy domain operated by a triangle:

Legal / Compliance
Business
SRE / Engineering

In this chapter, we move one step closer to the field and answer:

When an RML-3 incident happens, what should the actual response flow look like?

The goal is not “the ideal incident process.”
It’s a practical case-file format that connects:

the RML-1/2/3 worldview, and
the incident response workflow teams actually run.

Think of it as:

A “case file” is a worldview log:
which world changed, when, and who decided what about it.

1) Alert vs Bug vs Incident — separate them by world

In day-to-day operations, teams mix these terms:

“An alert fired”
“We hit a bug”
“We had an incident”

RML makes the distinction crisp.

1.1 Alert

An alert is typically:

a monitoring threshold was exceeded
error rate increased
latency degraded

This usually lives in RML-1/2:

many are transient
there may be user impact
but once recovered, no history-grade residue remains

So alerts often resolve inside the RML-1/2 flow.

1.2 Bug (Defect)

A bug is:

behavior differs from spec
certain conditions cause errors
the system acts wrong—but hasn’t reached the History World

This is mostly RML-1/2 territory:

track it in tickets (JIRA, GitHub issues)
fix it with normal engineering cycles

1.3 Incident

In this series, we define an incident as:

A meaningful responsibility-bearing event that has reached the History World (RML-3).

Examples:

money moved (or should have moved but didn’t)
customers/third parties were affected in the real world
there is regulatory/contractual/social responsibility exposure

In RML terms:

world = "RML3", action = "escalate-history"
should be recorded in an Effect Ledger
requires an incident report / case file

This is what belongs in your “casebook.”

2) The RML-3 incident flow: a 6-step casebook

A practical response flow that teams can template as-is:

Detect
Contain
Understand
Decide
Act
Learn

The key improvement is:

Attach “which world this step lives in” to every phase.

2.1 Detect — discovery (RML-2 → RML-3 boundary)

Triggers typically include:

RML-2 code throws world="RML3" errors
gateway/SLO detects elevated “RML-3 errors”
CS/Sales escalates customer complaints

The most important move:

The moment you suspect “this might be RML-3,”
switch into the RML-3 workflow immediately.

Concrete checks:

[ ] incident ticket created with RML3 label
[ ] provisional Effect Ledger record created
[ ] you didn’t “handle it” via retries/compensation and hope nobody notices

2.2 Contain — stop the bleed (mostly RML-2)

Containment means:

prevent further History World damage from accumulating

Typical actions:

feature flag off / kill switch
block a tenant/region
stop new execution paths at RML-2 level (e.g., immediate cancels)

This is SRE/Engineering-led, but if business impact is large, bring in PO/leadership quickly.

2.3 Understand — build a single timeline (RML-1/2 analysis + RML-3 records)

Here you answer:

what happened (facts)
which world(s) it happened in (RML-1/2/3)
how much is already written as history (RML-3 residue)

You usually reconcile:

app logs (RML-1/2)
traces/metrics
Effect Ledger (RML-3)

Your output should be a single coherent timeline, not a pile of fragments.

2.4 Decide — the triangle meeting

This is where Chapter 8’s triangle becomes real:

Legal: minimum obligations (regulation/contracts/privacy)
Business: how far to go (customer trust/brand/cost)
SRE/Engineering: what is technically possible + risk trade-offs

Decision framing:

“How do we update history forward—without pretending we can delete it?”

Typical plan components:

refund / re-run / correction notice / feature disable / rollback-forward
customer notification vs press release vs ToS-based disclosure rules

2.5 Act — remediation + communication (History World updates)

Now you execute:

refunds
data corrections
customer emails/calls
internal and external explanations

Key point: these actions are new History World events.

You do not erase the original record. You add correction events on top of it.

2.6 Learn — prevention + governance improvements (back to RML-1/2)

Postmortem is where you convert RML-3 pain into RML-1/2 improvements:

technical design fixes
monitoring + runbook changes
process/governance gaps (ToS mismatch, escalation rules, ownership)

The objective:

Transform History World events into repeatable improvements—rather than shame artifacts.

3) Runbook vs Playbook (RML mapping)

Ops docs often come in two forms:

Runbook: step-by-step operation manual (How)
Playbook: scenario/decision flow (What/Who)

RML mapping makes this clean:

Runbook ≈ RML-2
- containment, rollback-forward, feature flags, commands
Playbook ≈ RML-3
- who must be involved, how decisions are made, comms strategy

A practical pattern:

RML-2 incidents stay in runbooks
the moment you detect RML3, you hand off to playbooks

4) A one-page case-file template (YAML)

Here is a “casebook template” you can actually use.

incident_id: INC-2025-0001
title: "Possible double charge"
rml_world: RML3
status: resolved # or ongoing, monitoring

detected_at: 2025-06-01T10:23:45Z
detected_by:
  - type: alert
    source: "payment_service.rml3_error_rate"
  - type: cs_ticket
    id: "CS-1234"

classification:
  severity: SEV-1 # align with your org’s scheme
  asset:
    - money
  scope:
    affected_users_estimate: 37
    affected_transactions_estimate: 41

timeline:
  - at: 2025-06-01T10:23:45Z
    world: RML2
    type: detection
    detail: "RML3 error rate crossed threshold"
  - at: 2025-06-01T10:25:00Z
    world: RML2
    type: containment
    detail: "Feature flag off to stop new payment execution"
  - at: 2025-06-01T11:10:00Z
    world: RML3
    type: impact_assessed
    detail: "37 users / 41 transactions impacted"

decisions:
  summary: >
    Full refund to all impacted users + apology coupon.
    No public press release required.
  decided_by:
    - role: legal
      name: "..."
    - role: business
      name: "..."
    - role: sre
      name: "..."

actions:
  technical:
    - type: refund
      world: RML3
      count: 41
    - type: feature_fix
      world: RML2
      detail: "Require idempotency keys for payment saga steps"
  communication:
    - type: user_email
      world: RML3
      template_id: "refund_apology_v2"
    - type: internal_post
      world: RML2
      channel: "#incident"

lessons_learned:
  technical:
    - "Make idempotency keys mandatory for payment saga steps"
  process:
    - "Add CS handoff procedure when RML3 alerts fire"
  governance:
    - "Review ToS compensation clause vs actual practice"

Why this works:

the timeline includes world: RML1/2/3
actions also include which world they update
decisions record who decided what
lessons split into technical/process/governance

This makes later review possible:

where are we slow? detect/contain/decide?
what bottleneck is repeated? technical or governance?

5) A concrete story: payment saga → RML-3 incident

A common path into RML-3 is an RML-2 mistake.

Example anti-pattern:

// Anti-pattern: retry without an idempotency key (duplicates can happen)
type PaymentGateway = {
  charge: (req: { paymentId: string; amount: number }) => Promise<void>;
};
declare const paymentGateway: PaymentGateway;

// (Assume `RmlError` is the structured error type from Chapter 5.)
async function charge(paymentId: string, amount: number) {
  try {
    await paymentGateway.charge({ paymentId, amount });
  } catch (e) {
    throw new RmlError({
      world: "RML2",
      severity: "error",
      action: "retry-with-backoff",
      code: "PAYMENT_GATEWAY_TEMPORARY_ERROR",
      message: "Temporary payment error",
      cause: e,
    });
  }
}

Client follows retry-with-backoff, retries a few times.
Without idempotency, the gateway may charge multiple times.

This is how:

an RML-2 “retry policy” bug
becomes an RML-3 money incident
which then returns as an RML-2 design fix:
- require idempotency keys
- enforce with SDK/lint/tests

It’s the full “world loop” in one example.

6) Anti-patterns: when case files stop working

6.1 Writing the report only “for appearance”

If the case file is written after the fact to look clean:

timeline diverges from reality
decisions are retroactively rationalized

Fix:

fill the timeline in real time
write decision logs at the moment decisions happen

6.2 Treating an RML-3 incident as a “lighter” bug

This is cultural:

the org doesn’t want to admit history-grade impact
so it gets down-scoped as “just an RML-2 defect”

Result:

later: “why wasn’t this reported/escalated?”
trust damage compounds

Fix:

predefine what counts as RML-3
when unsure, treat gray as RML-3 and downscope later

6.3 Only writing the technical root cause

If postmortems omit governance/process/ToS aspects, the same class of incident repeats in new shapes.

Fix:

always keep:
- lessons_learned.technical
- lessons_learned.process
- lessons_learned.governance

7) Checklist: how ready is your RML-3 casebook?

Definitions

[ ] Can you explain alert vs bug vs incident using RML?
[ ] Is your RML-3 incident definition documented?
[ ] Do you have a “gray → treat as RML-3” principle?

Flow

[ ] Do you have Detect/Contain/Understand/Decide/Act/Learn?
[ ] Do world="RML3" signals auto-create incident tickets?
[ ] Do you have kill switches / feature flags for containment?

Template

[ ] Do case files include a world per timeline event?
[ ] Do decisions capture “who decided what”?
[ ] Are lessons split across technical/process/governance?

Docs

[ ] Runbooks exist for RML-2 containment steps
[ ] Playbooks exist for RML-3 decision + comms steps
[ ] Lessons feed back into runbooks/playbooks

Culture

[ ] Legal/Business/SRE attend RML-3 postmortems
[ ] You can roughly estimate RML-3 costs (refunds, credits, ops)
[ ] RML-3 incidents are treated as learning opportunities

Closing — case files are worldview logs

This chapter’s one-liner:

An RML-3 case file is a log of worldview:
which world changed, what residue remained, who decided what, and how you corrected forward.

RML-1/2 explain how the system behaved internally
RML-3 records what became history
the organization decides how to update history forward
and then you rewrite RML-1/2 design to prevent repeat incidents

Chapter 10 — RML as Product Strategy: Designing Trust

Chapter 8 — Autonomy in the History World: The Legal–Business–SRE Triangle

kanaria007 — Mon, 09 Mar 2026 15:00:00 +0000

The Worlds of Distributed Systems — Chapter 8

“Technically, we can fix it.
Legally, it’s already considered ‘done.’”

The moment you enter RML-3 (History World), engineering alone is no longer sufficient.

Suddenly, these roles show up at the table—at the same time:

Legal / Compliance
Business (Product Owner / Leadership)
SRE / Platform / Engineering

And the conversation shifts into questions like:

“Is this an incident or just a bug?”
“Do we need to apologize? Refund? What does the contract require?”
“How much do we disclose, and to whom?”
“What does ‘prevention’ mean here, and how do we prove it?”

This chapter reframes RML-3 as:

Not a technical problem, but an autonomy problem.

In other words: how an organization governs History World decisions—who decides what, at what level, and with what responsibilities.

1) Why RML-3 becomes organizational design

In RML-1 / RML-2, the main actor is usually engineering:

RML-1: process-local rollback, tests, dry-runs
RML-2: sagas, compensation, backoff & retry, reconciliation

Many problems can be solved inside an engineering team.

But in RML-3, that collapses quickly. The moment you touch:

movement of money
information that already reached customers
regulation, law, contracts, audits
security/privacy obligations

…you can’t “make it as-if-it-never-happened” by technical rollback.

From here on, History World must be operated as a shared domain:

Legal: rules, contracts, liability, privacy, reputation constraints
Business: customer relationship, brand behavior, cost and trade-offs
SRE/Engineering: facts, causality, technical limits, prevention measures

So:

RML-3 autonomy = how these three parties split decision-making and accountability for history-grade events.

2) The triangle model of the History World

Here’s the simplest map that keeps teams sane:

            Legal / Compliance
                 ▲
                 │  (rules / contracts / social responsibility)
                 │
Business ◀────────┼────────▶ SRE / Engineering
(customer / P&L)  │          (facts / system reality)

Each vertex brings a different “truth”:

Legal optimizes for constraint satisfaction (what you must/must not do)
Business optimizes for relationship and viability (brand, customer trust, cost)
SRE/Engineering optimizes for reality alignment (what happened, why, what can be changed)

When a History World incident happens, you’re effectively negotiating:

what level to record it at
who must be informed
what remediation is acceptable
how public communication should look
what prevention commitments are real vs performative

That negotiation is not a bug. It is History World governance.

3) Lifecycle of an RML-3 incident

Most history-grade events follow a repeatable lifecycle:

Detection
Initial triage
Impact assessment
Remediation
Record & communication
Prevention

3.1 Detection often starts from RML-2 escalation

If you adopted Chapters 5–7, you already have the clean “entry point”:

application code throws an error labeled like world: "RML3", action: "escalate-history"
observability/gateways/incident tooling detect and route it

Key principle:

The moment you enter History World should be explicit in code and telemetry—not discovered later by humans.

3.2 Initial triage: is it really RML-3?

This phase separates:

“It looked scary but self-corrected in RML-2” vs
“This is truly a History World event”

Examples:

double charge occurred and money actually moved → RML-3 confirmed
double charge appeared but one side auto-canceled before settlement → could be RML-2 bug / UX issue

Practical rule: when uncertain, treat gray cases as RML-3 until proven otherwise. It’s safer to downgrade than to underreact.

3.3 Impact assessment: how “heavy” is the history?

Three axes help:

Scope: affected users / transactions
Asset: money, PII, legal exposure, operational integrity
Time: how long the impact persisted

Example intuition:

1 user, small mistaken charge → small RML-3 (CS-led)
hundreds of invoices mis-sent → medium RML-3 (Business + SRE-led)
potential confidential leak → large RML-3 (Legal + leadership-level)

4) Escalation rules: when to treat it as RML-3 by default

Teams lose time when every incident starts with:

“Is this really RML-3?”

So create shared rules.

4.1 Default-to-RML-3 cases

Treat these as RML-3 unless you can prove otherwise:

Financial discrepancies
- double charges, overcharges, missing refunds
Externally visible misdelivery
- sending data to the wrong party (email/notifications/statements)
Potential regulatory/contract breach
- missed deadlines, deletion/retention obligations, audit failures
Security incident suspicion
- authz bugs, privilege escalation, data exposure risk

Common thread:

You have an obligation to explain later—what happened, to whom, and what you did about it.

That’s History World.

4.2 Cases that may stay in RML-2

Possibly RML-2-contained:

temporary sync failures that later reconciled correctly
issues resolved entirely by retries/compensation with no external visibility
staging accidents with no production data or external effects

Even here, the “may” depends on shared criteria agreed with Legal/Business—not engineering vibes.

5) A usable RACI table for RML-3

A lightweight RACI (Responsible / Accountable / Consulted / Informed) makes autonomy explicit.

Phase	Legal	Business (PO/Leadership)	SRE/Engineering	CS/Support
Detection	I	I	R/A	I
Initial triage	C	C	R/A	I
Impact assessment	C/A	C/A	R	C
Remediation policy decision	A	A	C	C
Remediation execution (technical)	I	I	R/A	C
Remediation execution (customer)	C	A	I	R/A
Record & reporting	A	C	R	C
Prevention measures	C	A	R	C

This does not need to be “perfect.”
What matters is that, in an incident, nobody is guessing:

who owns facts and technical limits
who owns external responsibility and contractual constraints
who owns customer-facing behavior and cost trade-offs

6) The information infrastructure that makes autonomy real

Governance without records becomes “trust me bro.”

6.1 Effect Ledger + Incident Report + Decision Log

To operate History World, you need artifacts that survive time:

Effect Ledger: what external effects occurred (charges, notices, statements, access grants)
Incident Report:
- summary, scope, root cause (technical + process), remediation, prevention
Decision Log:
- who decided what, when, based on what information/policy

History World autonomy includes:

Not only “we can explain what happened,”
but “we can explain who decided the response, and why.”

6.2 Reduce “shadow operations”

A common anti-pattern:

refunds/corrections done manually in an admin UI
no durable record of who did what
later you can’t reconstruct the timeline confidently

Better target:

admin actions also append to the Effect Ledger
every manual operation is linked to an incident or ledger record

7) Two lenses that matter in the boardroom: P&L and ToS

7.1 P&L (Profit & Loss): History World costs are real

RML-3 remediation becomes money:

refunds
goodwill coupons/credits
external legal/consulting fees
staffing/ops costs for support and comms

If you keep good History World records, you can later answer:

“How much did RML-3 incidents cost us this year?”
“How much was refunds vs ops vs goodwill?”
“Which product area is repeatedly generating history-grade costs?”

That clarity improves prioritization—and makes prevention investment legible.

7.2 ToS / contracts: the minimum line vs the brand line

Legal contracts often define:

what is guaranteed vs best-effort
what compensation is required
what is excluded/limited

This enables a clean split:

ToS line: the legal minimum you must do
Brand line: what Business chooses to do beyond minimum to preserve trust

Important: this is not “ToS justifies anything.”
It’s a way to make the autonomy boundary explicit.

8) Organizational anti-patterns (and fixes)

8.1 “We can roll back” as an argument against Legal

Engineering: “We can revert the DB.”
Legal: “But users already received the email.”

Fix: present the honest residue:

“We can roll back within RML-2 up to here.
What remains as history is: X, Y, Z.”

8.2 Treating incidents as “just bug tickets”

If RML-3 events are filed like normal bugs:

scope/impact/refunds aren’t recorded
later you can’t tell whether something was RML-2 or RML-3

Fix:

manage RML-3 as a distinct incident type
add fields like RML world, external impact, refund required, external comms required

8.3 “Someone will handle it” culture

If SRE ends up doing everything (triage, reporting, prevention) while Legal/Business only rubber-stamp later, autonomy becomes fragile.

Fix:

enforce the RACI in the postmortem process
require cross-functional attendance for history-grade incidents

9) Practical checklist (History World autonomy)

Rules & process

[ ] Do we have a shared definition of “RML-3 incident”?
[ ] Are escalation conditions documented and agreed cross-functionally?
[ ] Is the incident lifecycle owned via an explicit RACI?

Information infrastructure

[ ] Do we have templates for Effect Ledger + Incident Report + Decision Log?
[ ] Are manual/admin operations recorded and linked to history artifacts?
[ ] Do world=RML3 signals trigger incident creation and routing?

Culture

[ ] Do Legal/Business/SRE attend postmortems for RML-3 events?
[ ] Do we talk in “which world, how far rollback goes,” not “we can rollback”?
[ ] Do we treat RML-3 incidents as improvement inputs, not shame events?

Closing — whose world is History World?

RML-3 cannot be operated by:

engineering alone
legal alone
business alone

History World is the organization’s autonomy domain.

The question is not whether you’ll face RML-3 incidents. You will.

The question is whether, when it happens, your organization can:

identify it cleanly
decide responsibly
remediate coherently
explain credibly
improve deterministically

Next chapter:

Chapter 9 — RML-3 Case Files: Aligning your incident-response worldview

Chapter 7 — API & Client Design: Exposing RML Labels Across the Boundary

kanaria007 — Thu, 05 Mar 2026 15:00:00 +0000

The Worlds of Distributed Systems — Chapter 7

In Chapters 5–6, we made RML-2 concrete:

Attach world / severity / action to failures
Connect them to observability and on-call policy
Treat sagas as retryable conversations, where retry-with-backoff, start-compensation, and escalate-history are state machine triggers
Treat idempotency keys as the lifeline of retries

This chapter brings that worldview to the most important interface in real systems:

The boundary between API and client.
“What world is burning behind this HTTP 500?”

We’ll design APIs and clients so they can share RML semantics and enforce correct behavior—rather than leaving everything to guesswork.

1) HTTP status codes don’t tell you which world you’re in

A typical API error looks like this:

HTTP/1.1 500 Internal Server Error
Content-Type: application/json

{ "message": "Internal Server Error" }

From the client’s perspective, this tells you almost nothing:

Was it invalid input (RML-1-ish)?
Was it a transient downstream failure (RML-2 retryable)?
Did we hit a history-bound rule (RML-3 escalation / no retries)?

In RML terms:

“Which world is this error from: RML-1 / RML-2 / RML-3?”
“What should I do next?”

This chapter’s goal:

Make API/client boundaries RML-aware
Let clients interpret Action Hints
Prevent the worst outcome: automation-by-guessing

2) Put RML in API responses: body vs headers vs status

Think of an API response as three layers:

HTTP status
HTTP headers
JSON body

Each has a natural role.

2.1 JSON body: the core error object

Body is what humans and clients read first.

A practical shape:

{
  "code": "PAYMENT_ALREADY_SETTLED",
  "message": "This payment is already settled and cannot be canceled.",
  "world": "RML3",
  "action": "escalate-history",
  "details": {
    "paymentId": "pay_123",
    "settledAt": "2025-12-01T10:23:45Z"
  }
}

code: stable branching key (logging, UI mapping, analytics)
message: human-facing message (UI/logs)
world / action: the RML metadata
details: domain-specific context

Public APIs note: you may choose to expose fewer fields in the body (e.g., omit world/action) and put RML only in headers. Internal APIs can usually include everything.

2.2 HTTP headers: RML metadata + retry control

Headers are ideal for machine-readable policy and cross-cutting behavior:

HTTP/1.1 409 Conflict
Content-Type: application/json
X-RML-World: RML3
X-RML-Action: escalate-history
Retry-After: 0
X-Idempotency-Key: saga:12345:charge
X-Request-ID: req-abcde

X-RML-World: RML1 / RML2 / RML3
X-RML-Action:
- retry-local
- retry-with-backoff
- start-compensation
- escalate-history
- abort
Retry-After: suggested delay (especially for retry-with-backoff)
X-Idempotency-Key: the retry lifeline (when relevant)
X-Request-ID: ties UI and support workflows to traces

Now clients can behave consistently:

RML3 + escalate-history → never auto-retry, show support path / ticket ID
RML2 + retry-with-backoff → retry using the same idempotency key
RML1 + abort → show validation feedback immediately

2.3 HTTP status: keep it coarse

Status codes are a coarse bucket. That’s fine.

A useful mapping (not universal, but practical):

world	action	status (typical)	example
RML1	abort	400 / 422	validation failure
RML2	retry-with-backoff	502 / 503	transient downstream outage
RML2	start-compensation	409 / 500	saga step failed mid-flow
RML3	escalate-history	409 / 422 / 403	settled payment; forbidden reversal

Don’t try to encode everything in status codes.
Encode the meaning in X-RML-World and X-RML-Action.

3) Internal API vs BFF vs Public API: different exposure levels

3.1 Internal microservice APIs

Assumption: callers understand RML.

So you can return full metadata:

{
  "code": "STOCK_RESERVE_FAILED",
  "message": "Failed to reserve stock.",
  "world": "RML2",
  "action": "start-compensation",
  "details": { "sku": "SKU-1234", "warehouse": "TOKYO-1" }
}

BFFs, orchestrators, and gateways can interpret this directly.

3.2 BFF (Backend for Frontend) as translator

BFF sits between:

RML-aware backend world
UI that must be understandable to humans

Typical responsibilities:

Receive world/action from downstream
Translate to a UI-level error model:

which field to highlight
whether to auto-retry
whether to show a “contact support” path
1. Optionally soften messaging for end users

Example translation:

Downstream returns:

{
  "code": "PAYMENT_ALREADY_SETTLED",
  "world": "RML3",
  "action": "escalate-history",
  "message": "Payment already settled; cannot cancel."
}

BFF outputs to UI:

End-user message: “This payment can’t be canceled here. Please contact support.”
Internal behavior: enqueue incident/case + attach requestId

3.3 Public APIs

For external developers, you must decide what becomes part of the external contract.

A common approach:

Body: code/message/details only
Headers: X-RML-World, X-RML-Action, Retry-After, X-Request-ID

External clients can still follow simple rules:

RML2 + retry-with-backoff → retry safely
RML3 + escalate-history → do not retry; escalate to human flow

4) The RML-aware client (SDK) pattern

If you expose X-RML-* but nobody reads it, nothing changes.

So the core move is:

Provide an RML-aware client wrapper and standardize its use.

4.1 Parse RML from headers

type World = "RML1" | "RML2" | "RML3";
type Action =
  | "retry-local"
  | "retry-with-backoff"
  | "start-compensation"
  | "escalate-history"
  | "abort";

type RmlMeta = { world?: World; action?: Action };

function asWorld(v: string | null): World | undefined {
  if (v === "RML1" || v === "RML2" || v === "RML3") return v;
  return undefined;
}

function asAction(v: string | null): Action | undefined {
  if (
    v === "retry-local" ||
    v === "retry-with-backoff" ||
    v === "start-compensation" ||
    v === "escalate-history" ||
    v === "abort"
  ) return v;
  return undefined;
}

function parseRml(res: Response): RmlMeta {
  return {
    world: asWorld(res.headers.get("X-RML-World")),
    action: asAction(res.headers.get("X-RML-Action")),
  };
}

4.2 Enforce Action Hints

function sleep(ms: number) {
  return new Promise<void>((r) => setTimeout(r, ms));
}

function backoffWithJitter(baseMs: number, attempt: number) {
  const capped = Math.min(baseMs * 2 ** attempt, 10_000);
  const jitter = Math.random() * (capped * 0.2); // ~ +0–20%
  return capped + jitter;
}

export async function fetchWithRmlHandling(
  input: RequestInfo,
  init?: RequestInit,
  opts?: { maxRetries?: number; baseDelayMs?: number }
): Promise<Response> {
  const maxRetries = opts?.maxRetries ?? 3;
  const baseDelayMs = opts?.baseDelayMs ?? 500;

  let attempt = 0;

  while (true) {
    const res = await fetch(input, init);
    if (res.ok) return res;

    const { world, action } = parseRml(res);

    // If no RML metadata: behave like a normal fetch client.
    if (!world || !action) return res;

    // Never auto-retry history-bound failures.
    if (action === "escalate-history" || world === "RML3") return res;

    if (action === "retry-local" || action === "retry-with-backoff") {
      if (attempt >= maxRetries) return res;
      attempt++;

      const retryAfterHeader = res.headers.get("Retry-After");
      const retryAfterMs = retryAfterHeader ? Number(retryAfterHeader) * 1000 : 0;

      const delay =
        action === "retry-with-backoff"
          ? (retryAfterMs || backoffWithJitter(baseDelayMs, attempt))
          : 0;

      if (delay > 0) await sleep(delay);
      continue;
    }

    // For saga compensation triggers, return the response to caller (or hook your saga runner here).
    if (action === "start-compensation") return res;

    // Default: do not invent behavior.
    return res;
  }
}

Key point: action hints only matter if you enforce them via shared code.

5) UX by world: don’t show “RML3 error” to users

RML metadata is primarily for clients/ops—not end users.

A healthy UX mapping:

RML-1 (validation / local issues)
- highlight fields, inline errors
- “Please fix and resubmit”
RML-2 (transient dialog failures)
- “Temporary issue. Retrying…”
- keep spinner, auto-retry with backoff
- after N retries: “Please try later”
RML-3 (history-bound)
- “This can’t be completed here. Contact support.”
- show requestId / case ID
- quietly create an incident/case on the backend

6) Gateway governance: centralize retry policies and detect bad clients

If you emit RML headers, gateways can enforce org-level safety.

6.1 Centralized retry rules

If X-RML-World=RML2 and X-RML-Action=retry-with-backoff:
- gateway retries a limited number of times
If X-RML-World=RML3:
- gateway forbids retries

6.2 Detect clients ignoring Action Hints

If you see:

API responds RML3 + escalate-history
same client continues calling at high frequency

That’s a sign they ignore the contract. Gateways can:

rate limit
block
notify developers

7) GraphQL / gRPC: where to carry `world/action`

RML is not REST-specific.

7.1 GraphQL (use `extensions`)

{
  "data": null,
  "errors": [
    {
      "message": "Payment already settled; cannot cancel.",
      "path": ["cancelPayment"],
      "extensions": {
        "code": "PAYMENT_ALREADY_SETTLED",
        "world": "RML3",
        "action": "escalate-history",
        "requestId": "req-abcde"
      }
    }
  ]
}

7.2 gRPC (metadata + rich status details)

Use:

gRPC status codes (FAILED_PRECONDITION, UNAVAILABLE, etc.)
metadata/trailers: x-rml-world, x-rml-action
optional structured error details message

The idea is the same: clients must be able to route behavior based on world/action.

8) Anti-patterns

8.1 You defined `X-RML-*`, but nobody reads it

Result: you changed nothing. Fix it by:

providing an RML-aware SDK wrapper
enforcing “no raw fetch/http client” via linting or code review rules

8.2 “World” semantics drift across services

If one service uses RML3 to mean “non-retryable policy violation” and another uses RML3 to mean “temporary restriction,” clients become confused and unsafe.

Fix it by:

having a single org-level RML meaning table
documenting endpoint guarantees like “This endpoint never returns RML3”

8.3 Exposing RML jargon directly to end users

“RML3 error occurred” is meaningless to users.

Fix it by:

translating at the BFF/UI layer
showing meaningful messages + a support path + request ID

9) Checklist

API side

[ ] JSON body includes code, message, and optional details
[ ] Headers carry X-RML-World, X-RML-Action, Retry-After, X-Request-ID
[ ] RML3 + escalate-history returns a non-retryable status (often 409/422/403)
[ ] Idempotency is contractually supported where retries exist

Client / BFF side

[ ] Use an RML-aware shared client wrapper
[ ] Behavior branches on world/action (retry / compensate / escalate)
[ ] UX differs per world (validation vs retry vs support)
[ ] Public API docs explain RML headers clearly

Gateway / SRE side

[ ] Central retry policy exists for RML2 + retry-*
[ ] Retry forbidden for RML3
[ ] Detection exists for clients ignoring Action Hints

Closing — carrying the worldview across boundaries makes responsibility easier

The core takeaway:

HTTP status codes alone can’t express RML semantics
Add world/action in headers (and optionally body)
Build an RML-aware client wrapper so Action Hints are enforced
Translate RML semantics into UX, gateway policy, and incident workflows
Apply the same idea across REST, GraphQL, and gRPC

At the end of Part II, we now have the full “Dialog World” stack:

Chapter 5: Failure design + observability + governance
Chapter 6: Sagas and compensations as retryable conversations
Chapter 7: API/client boundary carrying the worldview

Next, Part III moves the focus fully into RML-3 (History World)—where distributed systems meet legal responsibility, money, and social trust.

DEV Community: kanaria007

Make A/B Tests Operable: Goal Surface Deterministic Logs Safe Automation (dry-run / shadow / canary / rollout)

0) Fix one failure story first

1) Evaluation is not a scalar: Goal Surface (multi-objective + constraints)

1.1 What is a Goal Surface?

1.2 Minimal Goal Surface format (JSON policy)

2) Make failures replayable: a minimal deterministic log spec

2.1 Minimal fields you should not skip

2.2 NDJSON append-only example

3) Safe automation boundary: dry-run / shadow / canary / rollout

3.1 Recommended stages

3.2 Stop conditions must be deterministic

3.3 Turn DEGRADE into an operational metric (SLO)

3.3.1 Reason code taxonomy (aggregate-able granularity)

3.3.2 Two SLOs that actually work

3.3.3 Make DEGRADE re-enterable (resume-friendly)

4) Minimal implementation (stdlib-only): gate evaluation + append logs + replay

4.1 Stable input fingerprint: input_digest (sha256)

4.2 Gate evaluation: Goal Surface → Verdict

4.3 Append-only NDJSON logging

4.4 Stage driver (minimal)

4.5 Replay (“what happened?” becomes queryable)

4.6 The missing link from PoC to production (the real constraints)

4.6.1 Treat metrics_snapshot as evidence (with freshness)

4.6.2 Don’t mix “promotion (winner)” with “safety gate”

4.6.3 Rollback atomicity: automate only reversible changes

4.6.4 Align by watermark contract, not “perfect 5-min windows”

4.6.5 Implement the driver as an orchestrator (state machine)

4.6.5.1 Split into two layers: Orchestrator vs Evaluator

4.6.5.2 DEGRADE is not “stop”—it’s “stop in a re-enterable way”

4.6.6 Make Promotion Gate mandatory for 100% rollout (not only safety gates)

4.6.6.1 Canary uses Safety Gate; 100% requires Promotion Gate

4.6.6.2 Make “100% promotion conditions” explicit in policy

5) Closing: three pieces, one operable system

Billing That Won’t Break: Deterministic Design to Kill Double Charges (API Contract DB Boundary Execution Patterns)

1) The classic accident: how double charges actually happen

2) The conclusion: determinism is built in 3 layers

L3 (execution): converge to the same outcome, even with retries

L2 (database): make double execution impossible

L1 (API): stop contract drift in CI

3) L3: Five deterministic patterns that kill most double-charge bugs

Pattern 1) Idempotency-Key (pin request identity)

Pattern 2) Unique constraint + Upsert (DB refuses “twice”)

Pattern 3) Outbox (move external effects outside the DB transaction)

Why “exactly-once” is a myth (and why Outbox + Reconciliation is not optional)

Pattern 4) State machine (track how far you got)

The truth table: internal state vs provider state (what you actually do)

Pattern 5) Reconciliation (assume drift will happen)

4) L2: A minimal PostgreSQL boundary (DDL)

5) L1: Freeze the API contract in CI (minimal)

5.1 Minimal OpenAPI excerpt

5.2 Minimal breaking-change checker (stdlib-only, JSON OpenAPI)

6) Minimal implementation sketch (Kotlin + JDBC + PostgreSQL)

6.1 request_hash (prevent “same key, different body”)

6.2 Key JDBC constraint: avoid “aborted transaction” traps

7) Outbox worker (FOR UPDATE SKIP LOCKED)

8) Ten golden cases (grow determinism like a verifier)

9) Common traps (avoiding these already helps a lot)

10) Scaling and ops: when the simple design hits limits

10.1 Outbox/idempotency “hot table” + vacuum/bloat

10.2 Reconciliation is hard (so design the key first)

10.3 Distributed writes change assumptions

10.4 TTL and “identity durability”

Recap (TL;DR)

Closing

Grow the Verifier, Not the Prompt: Run Production with 10 Golden Cases

0) The premise (re-stated)

1) Why “10 cases” is enough to start

2) What a golden case actually freezes

3) Minimal case format (portable)

3.1 One-case JSON example

4) Which 10 cases to pick (template)

5) Ten starter cases (names + intent)

JIT Access (3 cases)

1. jit_access_accept_minimal (ACCEPT)

2. jit_access_degrade_missing_security_approval (DEGRADE)

3. jit_access_reject_admin_role_without_break_glass (REJECT)

Production Change Management (3 cases)

4. change_degrade_missing_rollback_plan (DEGRADE)

5. change_reject_no_canary_steps (REJECT)

4.1 Stable input fingerprint: `input_digest` (sha256)

4.6.1 Treat `metrics_snapshot` as evidence (with freshness)

1. `jit_access_accept_minimal` (ACCEPT)

2. `jit_access_degrade_missing_security_approval` (DEGRADE)

3. `jit_access_reject_admin_role_without_break_glass` (REJECT)

4. `change_degrade_missing_rollback_plan` (DEGRADE)

5. `change_reject_no_canary_steps` (REJECT)

6. `change_accept_normalize_force_rollback_hook` (ACCEPT + normalize)

7. `erasure_degrade_identity_not_verified` (DEGRADE)

8. `erasure_reject_legal_hold` (REJECT)

9. `erasure_accept_normalize_retention_to_redact` (ACCEPT + normalize)

10. `uw_degrade_missing_required_documents` (DEGRADE)