DEV Community

Sub Engel
Sub Engel

Posted on

It broke again: a short about the same bug coming back, and the incident note I write now

Episode 2 of our Rozz shorts is about a bug that comes back. Customers got three payment emails for one invoice. Rozz opens the Incidents dashboard, then the incident note, and works through it: symptom, root cause, fix, prevention.

The note on Rozz's screen is a real one from Dev Second Brain, the Obsidian vault template for developers. It's one of the example incident notes that comes with it: "Duplicate payment emails after deploy". A new step made a webhook handler slower than the provider's timeout. The provider retried, the handler wasn't idempotent, and each retry sent another email.

The Incident template has the same four parts the episode walks through:

  • Symptom: what users, alerts or logs actually showed, with room to paste the log lines
  • Root cause: the real cause, not just what set it off. Keep asking "why did that happen?"
  • Fix: what changed, and the PR or commit
  • Prevention: checkboxes for the test or alert that would have caught it

It asks you for a severity when you create it, files the note under 40 Incidents/ with the date in the name, and tracks status as open, mitigated, then resolved.

The Incidents dashboard (Dataview) is where "it broke again" gets easier. It lists incidents that aren't resolved yet, every unchecked Prevention item, and a log grouped by cause and by service. If the same service keeps coming up, you'll see it there. Prevention work you never got to also stays on the list until you check it off.

You need Obsidian (free) plus the free Templater and Dataview plugins. It's all plain Markdown files on your own disk.

Episode 3, "Standup amnesia", goes up tomorrow.

Dev Second Brain is $19 on Gumroad, one person per copy, personal and commercial use allowed: https://subengel.gumroad.com/l/uoybkd

More of what we make: https://engelailabs.com

Top comments (0)