The LinkedIn post I published three days ago triggered something I didn't expect.
Not disagreement — validation. Senior engineers from payments companies, healthcare platforms, and financial infrastructure all said the same thing in different words: they've watched brilliant developers ship changes that worked perfectly in staging and detonated in production, not because the engineers were careless, but because their entire training rewards forward motion while reliability work rewards the opposite.
This post is the practical follow-up. The opinion was: developers and SREs think differently, and you can't fix that by handing developers a Terraform module and calling it ownership. The practice is: here's how you encode SRE paranoia into automated gates so the system enforces what culture can't.
The Core Problem With "Shift Left"
A shift-left mindset means SREs can embed reliability principles from Dev to Ops, baking reliability and resiliency into each process, app, and code change. That's the theory. The practice is that "baking reliability in" almost always means asking developers to think like SREs — to internalize failure mode thinking as a natural habit. Dynatrace
That doesn't work at scale. It works for the senior engineer who has been paged at 3 AM enough times. It doesn't work for the engineer who has never been on-call for a service they built.
In a traditional DevOps environment, the developer who wrote the code is often the one paged, focusing on a hotfix to restore the pipeline. In an SRE-driven environment, the SRE team manages the incident using a pre-defined playbook, focusing on automated remediation to bring the system back within its SLO parameters while the developers continue their sprint. Full-Stack Techies
The SRE-driven model works because the SRE's paranoia is encoded into the playbook, the SLO, and the error budget — not because the developer has learned to think differently. The developer doesn't need to internalize the mindset. The system has already internalized it for them.
That's the design principle. Encode the paranoia. Don't outsource it to culture.
What SRE Paranoia Actually Looks Like in Code
An SRE reviewing a production change asks five questions that a developer typically doesn't:
What's the rollback path, and have we tested it recently? Not "does a rollback exist" — does the team have a recent drill showing it takes under 5 minutes?
What happens during deployment, not just after? Connections in flight. Transactions mid-way through. Cache state that doesn't match the new schema.
What does this change look like at 10x load with one dependency degraded? Not at nominal load with everything healthy.
What's the blast radius if this goes wrong at 2 AM with one engineer on-call? Can one person contain it, or does it need three?
What's the SLO burn rate impact of a 1% error rate for 10 minutes? Does the team have that calculation, or will they be running it during the incident?
None of these questions require the developer to become an SRE. They require the CI/CD pipeline to refuse to proceed until these questions are answered — in code, not in conversation.
Introducing SRE Paranoia Gates
An SRE Paranoia Gate is an automated production readiness check that encodes one specific failure-mode question as a machine-enforceable constraint. The gate runs in CI/CD. It produces a pass/fail signal. It has a named owner — an SRE who wrote it and is accountable for its accuracy.
Each gate maps to one of the five questions above:
python# agentsre/sre_paranoia_gates.py
from dataclasses import dataclass, field
from typing import List, Optional, Callable
from datetime import datetime, timezone, timedelta
from enum import Enum
import json
class GateResult(Enum):
PASS = "pass"
FAIL = "fail"
WARN = "warn" # Won't block but logs for SRE review
@dataclass
class ParanoiaGateCheck:
"""
Result of one SRE Paranoia Gate evaluation.
Every gate check is logged — pass or fail.
Passes build confidence. Failures block deployment.
Warns surface to SRE review without blocking.
"""
gate_id: str
gate_name: str
result: GateResult
reason: str
sre_owner: str
evidence: Optional[dict] = None
checked_at: str = field(
default_factory=lambda: datetime.now(timezone.utc).isoformat()
)
def to_dict(self) -> dict:
return {
"gate_id": self.gate_id,
"gate_name": self.gate_name,
"result": self.result.value,
"reason": self.reason,
"sre_owner": self.sre_owner,
"evidence": self.evidence,
"checked_at": self.checked_at,
}
class SREParanoiaGateRunner:
"""
Run all registered SRE Paranoia Gates before a production deployment.
Gates encode the five questions a seasoned SRE asks before any change:
Gate 1: Is the rollback path tested and under 5 minutes?
Gate 2: Is in-flight traffic handled during deployment?
Gate 3: Has this been tested at degraded-dependency load?
Gate 4: Is the blast radius survivable by one on-call engineer?
Gate 5: Is the SLO error budget healthy enough to absorb a 1% error rate?
All gates must pass for deployment to proceed.
WARN gates surface to SRE review but don't block.
One FAIL blocks the entire deployment.
"""
def __init__(self, service_name: str, sre_owner: str):
self.service_name = service_name
self.sre_owner = sre_owner
self._gates: List[Callable] = []
def register(self, gate_fn: Callable) -> None:
"""Register a gate function. Each gate returns ParanoiaGateCheck."""
self._gates.append(gate_fn)
def run_all(self, context: dict) -> dict:
"""
Run all registered gates against deployment context.
Args:
context: Deployment metadata including service config,
rollback info, blast radius, SLO state, load test results.
Returns:
Summary with all gate results and deployment decision.
"""
results = []
blocked = False
for gate_fn in self._gates:
try:
check = gate_fn(context, self.sre_owner)
results.append(check)
if check.result == GateResult.FAIL:
blocked = True
except Exception as e:
# Gate evaluation failure = FAIL, not skip
results.append(ParanoiaGateCheck(
gate_id="gate_error",
gate_name=gate_fn.__name__,
result=GateResult.FAIL,
reason=f"Gate evaluation raised exception: {str(e)}",
sre_owner=self.sre_owner,
evidence={"exception": str(e)}
))
blocked = True
return {
"service": self.service_name,
"deployment_approved": not blocked,
"gates_run": len(results),
"gates_passed": sum(1 for r in results if r.result == GateResult.PASS),
"gates_failed": sum(1 for r in results if r.result == GateResult.FAIL),
"gates_warned": sum(1 for r in results if r.result == GateResult.WARN),
"results": [r.to_dict() for r in results],
"evaluated_at": datetime.now(timezone.utc).isoformat(),
"decision": (
"APPROVED — all SRE paranoia gates passed."
if not blocked
else "BLOCKED — one or more SRE paranoia gates failed. "
"Fix the flagged conditions before proceeding."
)
}
The Five Gates — Implemented
python# Gate 1: Rollback tested and under 5 minutes
def gate_rollback_readiness(context: dict, owner: str) -> ParanoiaGateCheck:
"""
Developers assume rollback exists. SREs verify it works in under 5 minutes.
The question isn't 'do we have a rollback' — it's 'did we test it recently?'
"""
rollback = context.get("rollback", {})
last_drill_date = rollback.get("last_tested_at")
p95_minutes = rollback.get("p95_duration_minutes")
if not last_drill_date:
return ParanoiaGateCheck(
gate_id="SPG-001",
gate_name="Rollback Readiness",
result=GateResult.FAIL,
reason="No rollback drill date recorded. Rollbacks untested are rollbacks that fail at 3 AM.",
sre_owner=owner
)
days_since = (datetime.now(timezone.utc)
- datetime.fromisoformat(last_drill_date)).days
if days_since > 30:
return ParanoiaGateCheck(
gate_id="SPG-001",
gate_name="Rollback Readiness",
result=GateResult.FAIL,
reason=f"Rollback last tested {days_since} days ago. Require < 30 days.",
sre_owner=owner,
evidence={"last_tested_at": last_drill_date, "days_since": days_since}
)
if p95_minutes and p95_minutes > 5:
return ParanoiaGateCheck(
gate_id="SPG-001",
gate_name="Rollback Readiness",
result=GateResult.FAIL,
reason=f"Rollback p95 is {p95_minutes}m. SRE target is < 5 minutes.",
sre_owner=owner,
evidence={"p95_minutes": p95_minutes}
)
return ParanoiaGateCheck(
gate_id="SPG-001",
gate_name="Rollback Readiness",
result=GateResult.PASS,
reason=f"Rollback tested {days_since} days ago, p95 {p95_minutes}m.",
sre_owner=owner
)
Gate 2: In-flight traffic handling
def gate_in_flight_traffic(context: dict, owner: str) -> ParanoiaGateCheck:
"""
Developers test what happens after deployment. SREs test what happens during.
Connections in flight, transactions mid-way, cache state mismatches.
"""
deploy = context.get("deployment", {})
graceful_shutdown = deploy.get("graceful_shutdown_seconds", 0)
drain_configured = deploy.get("connection_draining_enabled", False)
schema_backward_compat = deploy.get("schema_backward_compatible", None)
failures = []
if graceful_shutdown < 30:
failures.append(
f"Graceful shutdown is {graceful_shutdown}s — "
"recommend minimum 30s for in-flight requests to complete."
)
if not drain_configured:
failures.append(
"Connection draining not enabled — "
"load balancer will drop in-flight requests on pod termination."
)
if schema_backward_compat is False:
failures.append(
"Schema change is NOT backward compatible — "
"both old and new code will be running simultaneously during rollout. "
"Migration plan required."
)
if failures:
return ParanoiaGateCheck(
gate_id="SPG-002",
gate_name="In-Flight Traffic Safety",
result=GateResult.FAIL,
reason=" | ".join(failures),
sre_owner=owner,
evidence={"deployment_config": deploy}
)
return ParanoiaGateCheck(
gate_id="SPG-002",
gate_name="In-Flight Traffic Safety",
result=GateResult.PASS,
reason="Graceful shutdown, connection draining, schema compatibility verified.",
sre_owner=owner
)
Gate 3: Degraded-dependency load testing
def gate_degraded_load_test(context: dict, owner: str) -> ParanoiaGateCheck:
"""
Developers test at nominal load with all dependencies healthy.
SREs test at peak load with one critical dependency degraded.
The real question: what happens when the upstream rate-limits you
during a retry storm you caused?
"""
load_test = context.get("load_testing", {})
tested_at_peak = load_test.get("tested_at_peak_load", False)
tested_with_degraded_dep = load_test.get("tested_with_degraded_dependency", False)
retry_storm_tested = load_test.get("retry_storm_simulation", False)
if not tested_at_peak:
return ParanoiaGateCheck(
gate_id="SPG-003",
gate_name="Degraded Load Testing",
result=GateResult.FAIL,
reason="No peak load test recorded. Staging at 10% traffic ≠ production at 100%.",
sre_owner=owner
)
if not tested_with_degraded_dep:
return ParanoiaGateCheck(
gate_id="SPG-003",
gate_name="Degraded Load Testing",
result=GateResult.WARN,
reason=(
"Peak load tested but not with a degraded dependency. "
"Add fault injection to your load test suite. "
"This is a warning — but it's where most production incidents live."
),
sre_owner=owner
)
return ParanoiaGateCheck(
gate_id="SPG-003",
gate_name="Degraded Load Testing",
result=GateResult.PASS,
reason="Peak load and degraded-dependency scenarios both tested.",
sre_owner=owner,
evidence={"load_test_config": load_test}
)
Gate 4: Blast radius survivability
def gate_blast_radius_survivability(context: dict, owner: str) -> ParanoiaGateCheck:
"""
The SRE question: if this goes wrong at 2 AM with one engineer on-call,
can they contain it alone? Or does it need three engineers and a war room?
"""
blast = context.get("blast_radius", {})
downstream_count = blast.get("downstream_service_count", 0)
contains_payment_path = blast.get("contains_payment_path", False)
single_engineer_containable = blast.get("single_engineer_containable", None)
has_circuit_breaker = blast.get("circuit_breaker_configured", False)
if not has_circuit_breaker and downstream_count > 3:
return ParanoiaGateCheck(
gate_id="SPG-004",
gate_name="Blast Radius Survivability",
result=GateResult.FAIL,
reason=(
f"Service has {downstream_count} downstream dependencies "
"and no circuit breaker. "
"Failure will cascade. One engineer cannot contain this at 2 AM."
),
sre_owner=owner,
evidence={"downstream_count": downstream_count}
)
if contains_payment_path and single_engineer_containable is False:
return ParanoiaGateCheck(
gate_id="SPG-004",
gate_name="Blast Radius Survivability",
result=GateResult.FAIL,
reason=(
"Change touches payment path and requires multiple engineers to contain. "
"Require change window with full team available."
),
sre_owner=owner
)
return ParanoiaGateCheck(
gate_id="SPG-004",
gate_name="Blast Radius Survivability",
result=GateResult.PASS,
reason="Blast radius survivable by on-call rotation. Circuit breakers configured.",
sre_owner=owner
)
Gate 5: SLO error budget check
def gate_error_budget(context: dict, owner: str) -> ParanoiaGateCheck:
"""
The question developers don't ask: how much error budget remains?
If budget is < 20%, shipping anything non-trivial is a reliability bet
the team hasn't explicitly made.
"""
slo = context.get("slo", {})
budget_remaining_pct = slo.get("error_budget_remaining_pct", 100.0)
is_critical_change = context.get("is_critical_change", False)
if budget_remaining_pct < 5.0:
return ParanoiaGateCheck(
gate_id="SPG-005",
gate_name="SLO Error Budget",
result=GateResult.FAIL,
reason=(
f"Error budget at {budget_remaining_pct:.1f}% — critically low. "
"No changes permitted until budget recovers. "
"This is not optional. The SLO contract with your users says so."
),
sre_owner=owner,
evidence={"budget_remaining_pct": budget_remaining_pct}
)
if budget_remaining_pct < 20.0 and is_critical_change:
return ParanoiaGateCheck(
gate_id="SPG-005",
gate_name="SLO Error Budget",
result=GateResult.FAIL,
reason=(
f"Error budget at {budget_remaining_pct:.1f}% and change is marked critical. "
"Require SRE lead approval before proceeding."
),
sre_owner=owner
)
if budget_remaining_pct < 20.0:
return ParanoiaGateCheck(
gate_id="SPG-005",
gate_name="SLO Error Budget",
result=GateResult.WARN,
reason=(
f"Error budget at {budget_remaining_pct:.1f}%. "
"Below 20% — proceed with caution. "
"SRE team should be aware this change is consuming headroom."
),
sre_owner=owner
)
return ParanoiaGateCheck(
gate_id="SPG-005",
gate_name="SLO Error Budget",
result=GateResult.PASS,
reason=f"Error budget at {budget_remaining_pct:.1f}% — sufficient headroom.",
sre_owner=owner
)
Running the Full Gate Suite
python# Usage in your CI/CD pipeline
runner = SREParanoiaGateRunner(
service_name="payments-service",
sre_owner="platform-sre-team"
)
Register all five gates
runner.register(gate_rollback_readiness)
runner.register(gate_in_flight_traffic)
runner.register(gate_degraded_load_test)
runner.register(gate_blast_radius_survivability)
runner.register(gate_error_budget)
Build context from deployment metadata
context = {
"rollback": {
"last_tested_at": "2026-06-01T09:00:00Z",
"p95_duration_minutes": 3.5
},
"deployment": {
"graceful_shutdown_seconds": 60,
"connection_draining_enabled": True,
"schema_backward_compatible": True
},
"load_testing": {
"tested_at_peak_load": True,
"tested_with_degraded_dependency": False, # WARN
"retry_storm_simulation": False
},
"blast_radius": {
"downstream_service_count": 4,
"contains_payment_path": True,
"single_engineer_containable": True,
"circuit_breaker_configured": True
},
"slo": {
"error_budget_remaining_pct": 42.0
},
"is_critical_change": False
}
result = runner.run_all(context)
print(json.dumps(result, indent=2))
In CI/CD: fail the pipeline if not approved
if not result["deployment_approved"]:
raise SystemExit("SRE Paranoia Gates blocked deployment. See gate results above.")
Why This Works Better Than Culture
The teams winning on reliability in 2026 are not the ones with the most sophisticated AI stack. They are the ones that paired intelligent tooling with genuine engineering culture and did the hard work of changing how ownership flows, not just how alerts fire. Sherlocks AI
SRE Paranoia Gates are how ownership flows change in practice. The SRE doesn't have to be in every deployment review meeting. The SRE's questions are already in the pipeline. A developer shipping a change either answers them — by filling in the deployment context — or the gate blocks the deployment and the developer learns why those questions matter.
That's shift left done correctly. Not "teach developers to think like SREs." Encode what SREs think into the system itself.
The code is in agentsre/sre_paranoia_gates.py. MIT licensed.
Ajay Devineni | AWS Community Builder | Senior SRE/Platform Engineer
github.com/Ajay150313/agentsre | dev.to/ajaydevineni
Top comments (0)