DEV Community

Cover image for Incident Postmortem Culture: How to Conduct a Blameless Analysis
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

Incident Postmortem Culture: How to Conduct a Blameless Analysis

#career #indiehacker

Core Principles of Blameless Analysis

A blameless postmortem is a structured approach to identifying the cause of an incident, its impact, and the steps needed to prevent its recurrence. The main goal is to build an environment of trust within the team, turning mistakes into learning opportunities. The foundation of this culture rests on three core principles: Transparency, Process-Orientation, and Continuous Improvement.

Transparency ensures that all relevant data (logs, metrics, alerts) is shared and accessible to everyone. Process-orientation standardizes the steps; for example, defining the incident, building a timeline, conducting a root cause analysis, and documenting the action plan. Continuous improvement means systematically integrating the lessons learned from each postmortem into the product, architecture, and operational processes. These three principles enable teams to focus on solutions rather than pointing fingers.

Steps of the Postmortem Process

The postmortem process follows a specific template and includes the following steps:

  1. Defining the Incident – Gathering basic information such as the start and end times of the incident, the affected services, and the number of impacted users.
  2. Creating a Timeline – Chronologically listing key event points (alerts, escalations, interventions).
  3. Root Cause Analysis – Examining technical data and metrics to determine which components, configurations, or human errors were responsible.
  4. Action Plan – Defining what changes will be made to prevent recurrence, who will be responsible, and the timeline for implementation.
  5. Documentation and Sharing – Documenting all findings in Confluence, Jira, or another central repository and sharing them with relevant teams.
  6. Monitoring and Feedback – Monitoring the results after implementing the action plan and re-evaluating if necessary.

Completing each step helps teams resolve the issue across the entire ecosystem rather than treating it as an isolated point of failure.

Example Scenario: Database Outage

This section, labeled example scenario, presents a reproducible lab scenario in a controlled environment rather than a real-world incident.

Scenario Definition: In a production environment running PostgreSQL 13.5, a sudden spike in the number of checkpoints in pg_stat_bgwriter due to high I/O load causes checkpoint_timeout to trigger excessive memory consumption, resulting in a service outage.

Step 1 – Incident Definition

# Incident duration
journalctl -u postgresql -b | grep -E 'checkpoint|timeout' | tail -n 5
Enter fullscreen mode Exit fullscreen mode

Example Output

2026-08-20 14:32:07.123 UTC [12345] LOG:  checkpoint starting
2026-08-20 14:32:11.323 UTC [12345] LOG:  checkpoint finished: wrote 1200 buffers, 8.1 MB in 4.2 s
Enter fullscreen mode Exit fullscreen mode

Step 2 – Timeline

| Time | Event | Description |
|-------|------|----------|
| 14:30 | pg_stat_bgwriter spike | High I/O density |
| 14:32 | Checkpoint initiation | System approaching memory limit |
| 14:32:07 | Timeout | Service outage |

Step 3 – Root Cause

The default PostgreSQL value of checkpoint_timeout=5min is insufficient under high I/O on a server with 2GB of RAM. The max_wal_size and min_wal_size settings are also incompatible with checkpoint_timeout.

Step 4 – Action Plan

  • Increase checkpoint_timeout to 10m.
  • Set max_wal_size to 1GB and min_wal_size to 256MB.
  • Increase shared_buffers to 512MB.
  • Set vacuum_cost_delay for autovacuum to 20ms.
  • Apply changes using pg_ctl reload.

Step 5 – Rollback

Using a previously created pg_basebackup backup, the state at 14:29:00 is restored.

pg_basebackup -D /var/lib/pgsql/13/data -Ft -z -P -U replication_user
# Restore backup
pg_ctl stop -D /var/lib/pgsql/13/data
rsync -a /var/lib/pgsql/13/backup/14_29_00/ /var/lib/pgsql/13/data/
pg_ctl start -D /var/lib/pgsql/13/data
Enter fullscreen mode Exit fullscreen mode

After the rollback, a normal I/O profile is observed in pg_stat_bgwriter.

Trade-Off and Edge Case Analysis

The adjustments made in the scenario above balance performance gains against data integrity risks.

  • Performance Gain: Increasing checkpoint_timeout to 10m reduces checkpoint frequency, which lowers I/O pressure.
  • Risk: Longer checkpoint intervals can lead to increased disk space consumption due to growing WAL files.
  • Edge Case: During low I/O periods, even if checkpoint_timeout is set to 10m, the system may still trigger a checkpoint if the max_wal_size limit is exceeded. In this case, the wal_keep_size setting needs to be reviewed.

Additionally, increasing shared_buffers raises memory usage; this can impact the memory requirements of other services running concurrently. Therefore, monitoring the memory profile and CPU usage is critical.

Rollback and Verification Strategies

Rollback is not limited to restoring data backups; rolling back configuration changes is equally important.

  1. Configuration Version Control – The postgresql.conf and pg_hba.conf files are managed with Git. Before making any changes, run git commit -m "Update checkpoint settings".
  2. Rollback Command – Revert to the previous version with git checkout HEAD~1 postgresql.conf and apply with pg_ctl reload.
  3. Verification – Check if the checkpoint frequency has returned to its previous level in pg_stat_bgwriter and pg_stat_activity. 
psql -U postgres -c "SELECT checkpoints_timed, checkpoints_req FROM pg_stat_bgwriter;"
Enter fullscreen mode Exit fullscreen mode

Expected Output

 checkpoints_timed | checkpoints_req
-------------------+----------------
                12 |              0
Enter fullscreen mode Exit fullscreen mode
  1. Monitoring – Monitor I/O and memory metrics via Prometheus and Grafana; if a max_wal_size breach is observed, re-enable the alert.

The rollback process must be verified in a test environment to measure downtime and validate recovery steps.

Actionable Steps and Responsibility Matrix

Assigning the actions determined during the postmortem to specific owners is how the "blameless" approach is put into practice. The most common method used is the RACI (Responsible, Accountable, Consulted, Informed) matrix. This matrix shows who is directly responsible (R), the ultimate decision-maker (A), who needs to be consulted (C), and who needs to be informed (I) for each action item.

{
  "actionItem": "increase checkpoint_timeout",
  "responsible": "DBA_Team",
  "accountable": "Tech_Lead",
  "consulted": ["Ops_Team", "Security_Engineer"],
  "informed": ["Product_Manager", "Support_Team"]
}
Enter fullscreen mode Exit fullscreen mode

This JSON example expresses the responsibility distribution of an action in a single block. Two critical steps play a key role when preparing this matrix:

  1. Clarifying the Action List – What each step is, its measurable goal, and its completion criteria (Definition of Done) must be clearly defined.
  2. Reviewing Person/Group Assignments – Verification is conducted with multiple team members to ensure responsibilities do not overlap and do not create an excessive workload.

Once the matrix is complete, the team has a single, clear answer to "who will do what." This eliminates the risk of "ambiguity of responsibility," which is the most common reason for post-incident action item delays. Furthermore, documenting responsibilities makes it easier to trigger a fast rollback plan if the same error occurs again in the future.

Verification and Monitoring Protocols

Verifying that the system behaves as expected after implementing actions is the most critical phase of the postmortem process. Verification should be two-layered: Configuration verification and Performance/Monitoring verification.

Configuration Verification

Modified files (e.g., postgresql.conf) must be committed to the version control system (Git) and run through automatic lint checks via the CI pipeline. The following Ansible playbook example demonstrates applying a specific version of the postgresql.conf file to target servers.

- name: Apply PostgreSQL checkpoint settings
  hosts: db_servers
  become: true
  vars:
    pg_conf_path: /var/lib/pgsql/13/data/postgresql.conf
    checkpoint_timeout: "10min"
  tasks:
    - name: Ensure postgresql.conf is present
      copy:
        src: files/postgresql.conf
        dest: "{{ pg_conf_path }}"
        owner: postgres
        group: postgres
        mode: '0640'
    - name: Set checkpoint_timeout
      lineinfile:
        path: "{{ pg_conf_path }}"
        regexp: '^#?checkpoint_timeout'
        line: "checkpoint_timeout = '{{ checkpoint_timeout }}'"
    - name: Reload PostgreSQL configuration
      command: pg_ctl reload -D /var/lib/pgsql/13/data
      become_user: postgres
Enter fullscreen mode Exit fullscreen mode

After running the playbook, verify that the setting was actually loaded using the following command:

psql -U postgres -c "SHOW checkpoint_timeout;"
Enter fullscreen mode Exit fullscreen mode

Performance and Monitoring Verification

Applying the configuration is not enough; the impact of the change on the system must also be monitored. By tracking the "checkpoint_duration_seconds" metric in Prometheus and Grafana, you can compare the new values against the previous average. The Mermaid diagram below visualizes this verification flow.

Diagram

In this flow, if the targets set as KPIs (e.g., a 30% reduction in checkpoint duration, a 20% drop in I/O wait times) are not met, the system automatically transitions to the rollback step. This keeps the "trial and error" process under control and preserves system stability.

Risk Management and Continuous Improvement Loop

Every action carries a risk; identifying and monitoring these risks beforehand guarantees the sustainability of the postmortem culture. Risk management consists of three stages: Risk Identification, Risk Monitoring, and Risk Mitigation.

  1. Risk Identification – Potential impacts associated with action items (e.g., increased memory consumption, exceeding disk space limits) are added to a risk register. This register is maintained as a ticket created with a "Risk" label in Jira.
  2. Risk Monitoring – Created risk tickets are mapped to relevant metrics (CPU%, RAM%, disk usage). For example, when shared_buffers is increased, the vmstat output is monitored simultaneously; if an anomaly is detected, an alert is triggered.
  3. Risk Mitigation – Based on monitoring results, an "incremental increase" strategy is applied. For example, shared_buffers is increased by 10% at a time, monitored for a week, and then the next increase is made. This iterative approach reduces the risk of system crashes that a major configuration change might bring.

The continuous improvement loop is appended to the Lessons Learned document at the end of every postmortem, and this document serves as a reference for the next incident. This way, the team not only fixes past mistakes but also systematically evolves their processes to prevent the same error from resurfacing.

Conclusion

A blameless postmortem culture ensures that mistakes are viewed as systemic shortcomings rather than individual failures. Structured steps, transparent data sharing, and rollback strategies accelerate the team's learning loop. In our example scenario, an I/O-based outage was successfully resolved by reconfiguring PostgreSQL's checkpoint settings, and permanent improvements were applied to prevent recurrence. This approach provides a framework not just for post-incident reporting, but also for preventive engineering.

Official Sources

Top comments (0)