Production failures: the things that page you · 1. The three in the morning page

Production failures: the things that page you · Chapter 1 of 24 · Reliability · new chapter every Friday morning

By the end of this chapter: Read an unfamiliar dashboard under pressure and decide what to look at first.

Why this matters

When a pager goes off at three in the morning, adrenaline floods your system. Heart rate increases, cognitive focus narrows, and the brain defaults to pattern-matching rather than analytical reasoning. If you do not have a structured method for reading system state, this physiological response will force you down a rabbit hole. You will spend forty minutes investigating a minor CPU spike on a background worker while the primary database is completely unreachable.

Dashboards are almost always built by the engineers who wrote the application, which means they are designed to confirm the system is working, not to diagnose it when it is broken. They are dense with implementation details: garbage collection pauses, cache hit rates, queue depths, and thread counts.

Reading an unfamiliar dashboard under pressure requires ignoring ninety percent of what is on the screen. You must know exactly which three or four charts to look at first to isolate the fault domain. Without this discipline, you will guess at the root cause based on whichever chart looks the most erratic, prolonging the outage and exhausting yourself in the process.

Before you start

To execute the worked example and follow the technical reasoning in this chapter, you must have:

• Docker and Docker Compose installed on your local machine.
• Python 3.10 or higher installed.
• A basic understanding of HTTP status codes (specifically the difference between 200, 400, and 500 series).
• Familiarity with the concept of a time-series metric (a value recorded at a specific timestamp).

Bounding the problem from the outside in

The most common mistake in incident response is starting the investigation at the component you understand best, rather than the component closest to the user. If you are a database engineer, you look at database locks. If you are a frontend engineer, you look at the CDN.

You must evaluate the system from the outside in. The "outside" is the absolute edge of your infrastructure—usually a load balancer, an ingress controller, or an API gateway. The "inside" is the deepest layer of persistence.

When you open a dashboard, locate the metrics for the edge first. You are looking to answer a single, binary question: is traffic reaching our infrastructure at all? If the load balancer shows zero incoming requests, there is no point looking at application logs or database queries. The failure is upstream—perhaps a DNS failure, a revoked TLS certificate, or a severed transit link.

If traffic is reaching the edge, you move one layer inward to the application. If the application is receiving traffic but failing, you move to its dependencies. You only look at internal component metrics (like memory usage or thread counts) once you have proven that the layer above it is receiving requests and generating errors.

The three signals that dictate the investigation

Once you have identified the layer you are investigating, you must ignore custom business metrics and focus entirely on the RED signals: Rate, Errors, and Duration. These three metrics describe the experience of the consumer calling that layer.

Rate is the volume of requests per second. You are looking for sudden, sharp changes. A gradual increase is organic load; a vertical spike is a retry storm or a malicious attack. A sudden drop to zero means a pipe has broken upstream.

Errors are the rate of failed requests. In HTTP systems, this means 5xx status codes. You must explicitly filter out 4xx codes during initial triage. A spike in 4xx errors usually means a client deployed a bug and is sending malformed requests. A spike in 5xx errors means your infrastructure is failing to handle valid requests.

Duration is latency. You must never look at average latency. Averages hide catastrophic failures. If an endpoint serves 99 requests in 10 milliseconds, and one request hangs for 10,000 milliseconds before timing out, the average latency is 109 milliseconds. On a dashboard, 109ms looks like a healthy system. You must look at the 99th percentile (p99) latency. In that same scenario, the p99 latency is 10,000ms, which immediately tells you that a subset of users is experiencing total failure.

Correlating the signals to isolate the fault

Reading a dashboard is not about looking at one chart; it is about observing how Rate, Errors, and Duration move in relation to one another at the exact minute the alert fired. Their correlation tells you where to look next.

Scenario A: Rate remains flat, Duration spikes, Errors spike.
Traffic did not change, but suddenly the system got slow, and then it started throwing errors. This is the signature of resource exhaustion. A downstream dependency (like a database or a third-party API) slowed down. Your application threads waited for that dependency, holding open connections until they hit a timeout limit, at which point they returned 500 errors to the user. Do not look at application code; look at the downstream dependencies.

Scenario B: Rate spikes, Duration spikes, Errors spike.
This is a capacity failure. The system was overwhelmed by a sudden surge in traffic. The application ran out of CPU, memory, or network bandwidth trying to serve the load. The fix is usually to shed load, scale up, or block the offending traffic source.

Scenario C: Rate drops, Errors remain flat, Duration remains flat.
The system is fast and error-free, but nobody is using it. During peak hours, this is impossible. This means clients are failing to reach you. The fault domain is strictly upstream: DNS, BGP routing, or a misconfigured external firewall.

To see these correlations, you must control the time window of the dashboard. By default, dashboards often show the last 24 hours. A critical failure that began five minutes ago is invisible on a 24-hour graph; it looks like a single, tiny pixel. When you open the dashboard, immediately change the time window to start ten minutes before the alert fired, and end at the current time. This maximizes the visual contrast between "healthy" and "broken."

A worked example

This example simulates a production service that suddenly experiences a downstream dependency failure. We will spin up a Python application and a Prometheus instance to scrape its metrics. We will then trigger the failure and use PromQL (Prometheus Query Language) to triage the symptoms exactly as you would on a dashboard.

Create a directory named triage-example and create three files inside it.

1. app.py
This is our web server. It exposes metrics and a /simulate_outage endpoint.

import time
import random
import threading
from http.server import HTTPServer, BaseHTTPRequestHandler
from prometheus_client import start_http_server, Counter, Histogram

# RED Metrics
REQUEST_COUNT = Counter('http_requests_total', 'Total HTTP Requests', ['method', 'endpoint', 'status'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'HTTP Request Latency', ['endpoint'])

outage_active = False

class MetricsHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        global outage_active
        start_time = time.time()

        if self.path == '/simulate_outage':
            outage_active = True
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"Outage simulation started.")
            return

        # Simulate normal traffic vs outage traffic
        if outage_active:
            # Downstream dependency is hanging, causing latency spikes and timeouts
            time.sleep(random.uniform(2.0, 5.0))
            status = 500
        else:
            # Normal operation
            time.sleep(random.uniform(0.01, 0.05))
            status = 200

        REQUEST_COUNT.labels(method='GET', endpoint='/api/data', status=status).inc()
        REQUEST_LATENCY.labels(endpoint='/api/data').observe(time.time() - start_time)

        self.send_response(status)
        self.end_headers()
        self.wfile.write(b"Response")

def generate_background_traffic():
    """Simulates users constantly hitting the API."""
    import urllib.request
    while True:
        try:
            urllib.request.urlopen('http://localhost:8080/api/data', timeout=10)
        except Exception:
            pass
        time.sleep(0.5)

if __name__ == '__main__':
    # Start Prometheus metrics server on port 8000
    start_http_server(8000)

    # Start background traffic generator
    threading.Thread(target=generate_background_traffic, daemon=True).start()

    # Start main application server on port 8080
    server = HTTPServer(('0.0.0.0', 8080), MetricsHandler)
    print("Server running. Metrics on port 8000, App on port 8080.")
    server.serve_forever()

2. prometheus.yml
This configures Prometheus to scrape our Python application.

global:
  scrape_interval: 2s

scrape_configs:
  - job_name: 'python_app'
    static_configs:
      - targets: ['app:8000']

3. docker-compose.yml
This wires the application and Prometheus together.

version: '3.8'
services:
  app:
    build: 
      context: .
      dockerfile_inline: |
        FROM python:3.10-slim
        RUN pip install prometheus_client
        COPY app.py .
        CMD ["python", "app.py"]
    ports:
      - "8080:8080"
      - "8000:8000"

  prometheus:
    image: prom/prometheus:v2.45.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

Execution and Triage:

1. Open your terminal in the triage-example directory and run: docker-compose up -d
2. Wait 30 seconds for background traffic to generate baseline metrics.
3. Open Prometheus in your browser at http://localhost:9090.
4. In the query bar, type the following to check the Rate and click "Execute", then switch to the "Graph" tab: rate(http_requests_total[10s]) You will see a steady line representing normal traffic.
5. In a new terminal window, trigger the failure: curl http://localhost:8080/simulate_outage
6. Wait 15 seconds.
7. Now, triage the system using the RED method by running these three queries in Prometheus:

Query 1: Errors
rate(http_requests_total{status="500"}[10s])
You will see errors spiking from zero to a high rate.

Query 2: Rate (Total Traffic)
sum(rate(http_requests_total[10s]))
You will notice the total rate of requests has actually dropped slightly. The clients are blocked waiting for the slow server.

Query 3: Duration (p99 Latency)
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[10s]))
You will see latency jump from ~0.05 seconds to over 4.0 seconds.

By reading these three signals, you have successfully triaged the failure: Rate is steady/dropping, Duration spiked, Errors spiked. This is Scenario A. You know instantly that a downstream dependency is hanging and exhausting your application's capacity to serve new requests.

Where people get stuck

Symptom: You stare at a dashboard full of CPU and memory charts, trying to find the anomaly, but everything looks slightly jagged and you cannot tell what is normal.
Fix: Stop looking at resource metrics. Switch immediately to the load balancer or API gateway dashboard and look at the HTTP status codes and latency. CPU is a resource; it only matters if it impacts the user. Find the user impact first, then look at the resources of the specific component serving those users.

Symptom: The alert says the API is down, but the application dashboard shows zero errors and extremely low latency.
Fix: You are looking at the wrong fault domain. If the application shows no errors and low traffic, the traffic is not reaching the application. Move upstream. Check the ingress controller, the WAF (Web Application Firewall), or the DNS routing metrics.

Symptom: A customer complains that the system is timing out, but the dashboard shows an average latency of 45 milliseconds.
Fix: Change the dashboard aggregation from average (or mean) to p99 or p99.9. Averages smooth out outliers. If 99% of your traffic is fast and 1% is timing out at 30 seconds, the average will hide the failure entirely. You must measure the worst-case experience.

Your tasks

1. Adjust the time window: In the Prometheus UI from the worked example, find the time controls (usually in the top right). Change the graph window from the default (1 hour) to 5 minutes. Observe how the transition from "healthy" to "broken" becomes much sharper and easier to read. 'Done' when you can clearly see the exact second the outage began on the graph.
2. Isolate the 4xx errors: Modify app.py so that when outage_active is true, it returns a 400 status code instead of 500, and keeps latency low (0.01s). Restart the docker container, trigger the outage, and write a PromQL query that proves the system is failing due to client errors, not server latency. 'Done' when your p99 latency query shows a flat line, but your error query shows a spike.
3. Simulate an upstream failure: Modify the generate_background_traffic function in app.py to completely stop sending requests when outage_active is true. Restart the system, trigger the outage, and use PromQL to diagnose it. 'Done' when you can prove via the rate() query that traffic has dropped to zero, while errors and latency remain flat.

---

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.

Production failures: the things that page you

Chapter 1 of 24. New chapter every Friday morning.
Next: Thundering herd: when the cache expires all at once.

· The full syllabus and every chapter so far
· Subscribers also get the condensed notes for this chapter, the running
recap of everything the series has covered, and the extended guidance:
subscribe

---

Written by Amit Chakraborty — founding engineer and senior architect: React Native, AI and RAG systems, production architecture. Portfolio · LinkedIn · GitHub.

Need this built, reviewed or taught to your team? Get in touch or email [e-mail]. Available for senior and founding engineering roles, consulting and training, remote worldwide.

Comments

[Baptiste Le Bouquin]

The "dashboards are built to confirm health, not to diagnose failure" line is painfully accurate. I run automation across a fleet of VPS agents and my early dashboards were all green-lights: they answered "is the system up?" and were useless at 3am when the actual question is "what changed?".

What moved the needle for me: page on state transitions, not on error rates. A dependency that has been degraded for weeks is background noise; a component that was healthy 4 minutes ago and isn't now is almost always the story. Curious how you're structuring the 24 chapters — are you covering alert design itself, or keeping the scope to reading an existing dashboard under pressure?