DEV Community

Cover image for No external attack found: a blameless postmortem template for the Salesforce incidents we cause ourselves
Juan Manuel Garrido
Juan Manuel Garrido

Posted on

No external attack found: a blameless postmortem template for the Salesforce incidents we cause ourselves

Today I posted, in Spanish, about companies where nobody dares to tell the boss they are wrong. I paired it with a cartoon: a spokesperson at a podium with the Microsoft logo tells the press, "We found no evidence of an external attack. We are perfectly capable of taking down our own systems without anyone's help."

The line from my post that I keep coming back to is this one: sometimes we look for threats outside when the real risk is a culture where nobody dares to raise their hand. The full piece, with what I recommend to anyone who manages a team, is here: https://discover.egafutura.com/nadie-se-anima-a-decirte-que-estas-equivocado/?utm_source=devto&utm_medium=referral&utm_campaign=amplify

On a Salesforce team, that culture shows up the morning after something breaks in production. The first question is usually who deployed. A blameless postmortem asks a different one: how did our process let this through, and did anyone see it coming?

This is the template I suggest, plus where to get the facts from the org instead of from memory.

The template

# Incident: <one-line title>

Date, duration, impact (users, records, processes affected)

## Timeline
Facts with timestamps, taken from the org and from tickets.

## What we knew before
- Who raised a concern, where (meeting, ticket, chat) and when
- What happened to it: answered, deferred, ignored

## Contributing factors
Roles and process steps, not names.

## Changes
| Action | Owner | Due date |
Enter fullscreen mode Exit fullscreen mode

Two rules make it blameless in practice. Names can appear in the timeline, because the timeline is facts. They do not appear in contributing factors. And every action has an owner and a date. "Be more careful next time" is not an action.

Build the timeline from the org

Setup changes. Setup Audit Trail records configuration changes and can be queried:

SELECT CreatedDate, CreatedBy.Name, Section, Action, Display
FROM SetupAuditTrail
WHERE CreatedDate = LAST_N_DAYS:7
ORDER BY CreatedDate
Enter fullscreen mode Exit fullscreen mode

With Salesforce CLI:

sf data query --target-org prod --result-format csv \
  --query "SELECT CreatedDate, CreatedBy.Name, Section, Action, Display FROM SetupAuditTrail WHERE CreatedDate = LAST_N_DAYS:7 ORDER BY CreatedDate" \
  > audit-trail.csv
Enter fullscreen mode Exit fullscreen mode

Deployments. Setup > Deployment Status lists recent deployments with who started them, when, and whether they were validations or real deploys. Copy the relevant ones into the timeline with their IDs.

Automation errors. Flow and Apex error emails carry exact timestamps and the element that failed. If nobody on the team receives them, check Process Automation Settings and the Apex exception email recipients in Setup. That alone can be a finding for the Changes table.

The section that matters: what we knew before

This is where the cartoon comes in. In a team where nobody raises a hand, the warning may have existed and never reached the person who decided. Maybe someone noticed the flow that touched too many records, the test class that failed now and then, or the deploy scheduled for Friday evening.

So search before you write: tickets, pull request comments, chat threads around the change. Then ask people one by one, not in the group meeting:

  • Did you see anything before this that worried you?
  • Did you say it? If not, what stopped you?

If someone raised it and the warning got lost, write that plainly. It is not an accusation against the person who decided. It is the most useful line in the document, because it tells you where your process drops warnings.

A few days ago I wrote a related piece from the other side, for the person whose warning keeps getting ignored: When a Salesforce recommendation keeps getting ignored, write the risk down and set a limit.

Run the meeting so people actually talk

  • The most senior person in the room speaks last.
  • Read the "What we knew before" section aloud before discussing causes.
  • End with the Changes table filled in, or schedule the follow-up before people leave.

At EGA Futura we build a native ERP for Salesforce, so an incident in the org can stop invoicing or inventory, not just a dashboard. If your team is new to the platform, our free live Salesforce classes (in Spanish) cover the basics.

The spokesperson in the cartoon at least said it out loud: the problem was ours. A good postmortem gets to that sentence in writing, with a date next to each fix.

Top comments (0)