The Perfect Post-Mortem: Turning Failure into Learning (Template Included)

The outage is over. The site is up. You are exhausted. You want to forget it happened. Don’t.

This is the most valuable moment for your engineering team. If you don’t learn from this outage, you will repeat it.

The Rule: Blameless

The goal of a Post-Mortem (or Incident Review) is Process Improvement, not Punishment.

• Bad: “Dave pushed a bad config.”
• Good: “The CI/CD pipeline allowed a bad config to be pushed without validation.”

If you blame Dave, Dave will hide his mistakes next time. If you blame the process, you fix the system.

The Template

Copy this into your Notion/Confluence.

1. Summary

• Impact: Who was affected? (e.g., “50% of Checkout requests failed”).
• Duration: Start time to End time.
• Severity: SEV-1.

2. Timeline

• 10:00 UTC - Deployment triggered.
• 10:05 UTC - Alerts fired (High Latency).
• 10:10 UTC - PagerDuty woke up Alice.
• 10:15 UTC - Alice rolled back the deployment.
• 10:20 UTC - Recovery confirmed.

3. Root Cause Analysis (The 5 Whys)

1. Why did the site fail? Database connection limit reached.
2. Why? The new code opened a new connection for every request.
3. Why? The developer missed using the connection pool.
4. Why? The code review didn’t catch it.
5. Why? We don’t have automated linting for database patterns.

Root Cause: Lack of automated static analysis for connection pooling.

4. Action Items (Jira Tickets)

• Add sqlclosecheck linter to CI pipeline. (Owner: Bob, Due: Dec 30).
• Update “New Hire” documentation regarding DB pools. (Owner: Alice).
• Lower the connection timeout on the Load Balancer.

Publish It

Radical transparency builds trust. Publishing your post-mortem (sanitized) to your customer-facing blog shows that you are mature, honest, and improving.