- 73% of Slack's customer-facing incidents were triggered by Slack's own code deploys
- 90% reduction in customer impact hours from peak to January 2025
- Q1 improvement happened before any code shipped — from communication alone
- Manual → automatic rollbacks — removing human latency from the most common recovery path
- 10-minute threshold — where customer tolerance shifts from "blip" to "incident"
- Culture change came before code change; both were necessary
73% of Slack's customer-facing incidents were being triggered by Slack itself — by its own code deploys. The team stopped treating each outage as a one-off and started treating deploy safety as a program, with metrics, milestones, and automated rollbacks. Eighteen months later, customer impact hours were down 90%.
The Story
It's mid 2023 and we've identified some opportunities to improve our reliability. Fast forward to January 2025. Customer impact hours are reduced from the peak by 90% and continuing to trend downward.
— Slack Engineering, via 'Deploy Safety: Reducing customer impact from change', slack.engineering
Slack's reliability story in 2023 had an uncomfortable truth at its centre: the biggest source of customer-facing incidents was not external infrastructure failures, not traffic spikes, not adversarial attacks — it was Slack's own code deploys. A measurement of the incident dataset showed that 73% of customer-facing incidents were change-triggered, primarily code deployments. This number reframes the reliability problem entirely. You can harden infrastructure, add redundancy, and build better monitoring — but if most incidents are self-inflicted, the highest-leverage intervention is improving how you ship code.
The Deploy Safety Program began in mid-2023 with a key insight: measuring reliability improvement by waiting for incidents to occur creates a long feedback loop that is difficult to optimise. The team shifted to a leading-indicator metric — customer impact hours from high-severity change-triggered incidents — that could be tracked continuously without waiting for the next major outage. This metric served as the north star throughout the 18-month programme, allowing the team to see improvement (or regression) before the data showed up in annual availability reports.
Problem
73% of Incidents Self-Inflicted by Deploys
Slack measured that 73% of customer-facing incidents were triggered by change — primarily code deployments across hundreds of internal services. Manual remediation processes added latency between deploy and recovery. Interruptions exceeding 10 minutes were disproportionately damaging to customer trust.
Cause
Manual Detection and Remediation Too Slow
The existing approach relied on engineers detecting deploy-related regressions from monitoring dashboards and making manual rollback decisions. This added human latency — time to be paged, time to investigate, time to decide — to every incident. At Slack's deploy frequency across hundreds of services, the accumulated human latency was significant.
Solution
Automatic Rollbacks + Safety Culture
The Deploy Safety Programme introduced automated rollback triggers: when deploy-time metrics crossed defined thresholds, a rollback was automatically initiated without waiting for engineer intervention. The programme also invested in safety culture — normalising rollbacks as the right response rather than a failure indicator.
Result
90% Reduction by January 2025
Customer impact hours were down 90% from peak by January 2025, with the trend continuing downward. The peak of impact occurred between February and April 2024 — before automatic rollbacks were introduced. Once automatic rollbacks were live, the data showed dramatic improvement.
The Fix
The Deploy Safety Programme: Engineering + Culture
The Deploy Safety Programme was not purely a technical programme. Its first-quarter improvements came from communication — telling engineering teams what the metric was, why it mattered, and what behaviours were contributing to it. The technical work (automated alerts, automatic rollbacks, improved deploy signals) came later. Culture change came before code change, and the culture change produced measurable improvement even before the tooling was in place.
- 73% — fraction of customer-facing incidents triggered by Slack's own code changes; the measurement that transformed reliability into a deployment problem
- 90% — reduction in customer impact hours from peak (Feb–Apr 2024) to January 2025
- Q1 — quarter of work when improvement appeared before any code changes — purely from communicating the programme goals to engineering teams
- Auto — rollback execution mode after the programme's key technical milestone; removing human latency from the most common incident recovery path
# Simplified deploy safety automatic rollback logic
# Real implementation uses Slack's internal deploy orchestration system
class DeploySafetyMonitor:
def __init__(self, service: str, deploy_id: str):
self.service = service
self.deploy_id = deploy_id
self.baseline = self._capture_pre_deploy_metrics()
def monitor_post_deploy(self, window_minutes: int = 10):
"""Monitor service health for 10 minutes after a deploy.
Automatically roll back if metrics regress beyond thresholds.
Target: resolve most issues within the 10-minute customer tolerance window."""
start_time = time.time()
while time.time() - start_time < window_minutes * 60:
current = self._get_current_metrics()
# Check error rate regression
if current.error_rate > self.baseline.error_rate * ERROR_RATE_THRESHOLD:
self._automatic_rollback(
reason=f"Error rate {current.error_rate:.1%} exceeded "
f"threshold (baseline: {self.baseline.error_rate:.1%})"
)
return # rollback initiated — no human required in recovery path
# Check p99 latency regression
if current.p99_latency > self.baseline.p99_latency * LATENCY_THRESHOLD:
self._automatic_rollback(
reason=f"p99 latency {current.p99_latency}ms exceeded threshold"
)
return
time.sleep(30) # check every 30 seconds during bake period
# Monitoring window passed — deploy is baked, mark stable
self._mark_deploy_stable(self.deploy_id)
def _automatic_rollback(self, reason: str):
deploy_orchestrator.rollback(self.deploy_id)
# P2, not P1 — automatic rollback is the mitigation; engineer reviews after
pagerduty.notify(
severity='P2',
message=f'Auto-rollback: {self.service} {self.deploy_id}\n{reason}'
)
The non-linear progress curve: why it looked like it wasn't working
The programme's impact chart showed non-linear improvement. The first quarter produced reduction before any code changes shipped — from communication alone. Then came the peak of impact in early 2024 before automatic rollbacks were deployed, suggesting things were getting worse. Then a dramatic drop after automatic rollbacks went live. This non-linear curve is common in reliability programmes: communication changes behaviour, new tooling is built without full effect, the tooling deploys and impact drops sharply. Reading the curve correctly requires understanding what was deployed when — and maintaining confidence in the work based on leading metrics even when trailing metrics haven't yet reflected it.
The threshold calibration problem
Automatic rollbacks require precise threshold calibration — thresholds too sensitive trigger unnecessary rollbacks on normal traffic variance, eroding engineer trust in the system. Thresholds too loose miss real regressions. Slack's approach was per-service threshold calibration based on historical metric variance, with ongoing tuning as services' traffic patterns evolved. This calibration work is ongoing — it doesn't end when the automation is deployed. Getting thresholds wrong in either direction undermines the entire programme.
The 10-minute threshold and evolving customer expectations
Slack's data showed that customer tolerance for interruptions changed significantly at around 10 minutes. Shorter interruptions were treated as acceptable blips; longer ones were treated as incidents that impacted workflows and generated support tickets. Designing automatic rollbacks to trigger fast enough to resolve most issues within the 10-minute window became a key design constraint. Slack's blog post also notes that the introduction of Agentforce in 2025 raised customer expectations further — Slack being used as an AI-assisted work tool made even shorter interruptions more disruptive. The 10-minute threshold will continue to shrink as Slack becomes more tightly integrated into customer workflows.
Architecture
Slack's deploy safety architecture evolved from a manual-first system to an automated-first system over 18 months. The before state: engineers deploy → monitoring alerts fire → engineer investigates → engineer decides to roll back → engineer executes rollback. The after state: engineers deploy → monitoring compares against pre-deploy baseline → automatic rollback fires if thresholds are crossed → engineer is paged with context after the rollback has already happened. The human is in the loop — but as a reviewer of an automated decision, not as a prerequisite to recovery.
Before: Manual Deploy Remediation Path
View interactive diagram on TechLogStack →
Interactive diagram available on TechLogStack (link above).
After: Automatic Deploy Rollback Path
View interactive diagram on TechLogStack →
Interactive diagram available on TechLogStack (link above).
Lessons
Measure what's causing incidents before investing in what might fix them. Slack's discovery that 73% of incidents were change-triggered completely reframed their reliability investment. Without measurement, they might have invested in infrastructure redundancy while the primary driver — their own deploys — continued unchecked.
Trailing metrics (metrics that measure outcomes after they occur) tell you how things went. Leading metrics (metrics that indicate direction of travel before outcomes are fully visible) tell you whether what you're doing is working. Run both. Use leading metrics to steer the programme, and trailing metrics to confirm you've arrived.
Culture change can produce measurable improvement before code changes do. Slack saw improvement in the first quarter of the programme from communication alone — before any technical work shipped. When engineers understand what behaviour is costing customers, many of them change their behaviour voluntarily. Don't skip the cultural investment in favour of jumping straight to tooling.
Automatic rollbacks are not a replacement for good engineering — they are a safety net that reduces the cost of imperfect engineering. Every team ships bugs; the question is how quickly the system detects and recovers from them. Automatic rollbacks compress the detection-to-recovery time from tens of minutes to seconds, dramatically reducing customer impact for the most common class of incidents.
Rollback is the correct first response to a deploy regression. Forward-fixing maintains customer impact during investigation. Rolling back immediately restores service, then gives engineers the time and safety to understand the issue properly. Normalising rollback as the correct response — not a failure — is as important as building the tooling to do it automatically.
Engineering Glossary
Automatic rollback — a system that detects deploy-related metric regressions and automatically reverts the deployment without waiting for engineer intervention. Removes human latency (page → investigate → decide → execute) from the most common incident recovery path.
Change-triggered incident — an incident caused by a code deployment, configuration change, or other intentional system change rather than by external infrastructure failure or traffic anomaly. Slack found 73% of incidents fell into this category.
Customer impact hours — the programme's north-star metric: total hours of high-severity customer-impacting incidents attributable to change-triggered events. Tracked continuously rather than only in annual reports; provides a leading indicator of reliability improvement direction.
Forward-fix — deploying a code fix to address a regression rather than rolling back to the last known good state. Maintains customer impact during the investigation and fix cycle. Generally the wrong first response to a deploy regression.
Leading indicator — a metric that indicates the direction of travel before outcomes are fully visible. Customer impact per deploy, rollback rate, and manual-to-auto rollback conversion rate are leading indicators for deploy safety. Contrasted with trailing indicators like annual availability, which tell you how things went after the fact.
This case is a plain-English retelling of publicly available engineering material.
Read the full case on TechLogStack →
(Interactive diagrams, source links, and the full reader experience)
TechLogStack — built at scale, broken in public, rebuilt by engineers.
Top comments (0)