The blameless postmortem
Own a production incident you caused without drama, or facilitate a postmortem that finds causes, not culprits.
Case study: The config change at 6 p.m.
Pranav, a DevOps engineer in Noida, pushed a config change on a Friday evening that dropped the checkout service for 47 minutes. In the postmortem, he apologized for five minutes, called himself careless, and hid the fact that the deploy pipeline had no review step for config. The action item was "Pranav will be more careful." Three months later, someone else did the same thing. At a San Francisco team, an engineer named Ali presented a similar outage as a timeline: what he did, what he expected, what actually happened, and what in the system allowed it. The fix was a config validation check that has blocked six bad changes since.
Takeaway: People make mistakes. Systems let them through. Present the facts, own your part, and fix the system.
In many Indian teams, an outage feels like personal shame, so people either over-apologize or hide details. Blameless postmortems, common in global tech, assume good intent and focus on what in the system allowed the error.
Presenting your incident
Goal: Present a clear, honest timeline, own your actions in one sentence, and help the team find the system fix.
Step by step
- Own it once. One clear sentence of ownership. Not five minutes of apology.
- Walk the timeline. What you did, when, and what you expected to happen.
- Show the gap. What in the system allowed the mistake to reach production.
- Propose fixes. Concrete actions with owners, not "be more careful".
Say this, not that
| I pushed the config change that caused this outage, and I want to walk through exactly what happened. | I'm so sorry, I'm very careless, it's all my fault. | Over-apologizing makes everyone uncomfortable and hides the useful facts. |
| At six oh five, I changed the connection pool size. I expected it to apply to staging only. | Timeline plus intent helps people see the real gap. | |
| What actually happened is that the same file is used for production, and nothing in the pipeline flagged it. | ||
| The gap is that config changes skip code review and have no validation step. | I will be more careful next time. | Carefulness is not a fix. A check is. |
| I'd suggest two actions: a review step for config, and a validation check. I'm happy to own the first. |
Model conversation
- Jessica
- Thanks, everyone. Ali, could you walk us through the incident?
- You
- Sure. I pushed the config change that caused the outage, and I want to walk through exactly what happened.
- Jessica
- Thanks. Go ahead.
- You
- At six oh five I changed the connection pool size. I expected it to apply to staging only. It applied to production, and checkout failed for forty seven minutes.
- Jessica
- Why did you think it was staging only?
- You
- The file names look environment specific, but one file is shared. And config changes skip code review, so nobody else saw it.
- Jessica
- That's really useful. What would you suggest?
- You
- Two actions: a review step for config, and an automated check that blocks shared file changes. I'm happy to own the first.
Facilitating the postmortem
Goal: Keep it blameless and calm, get the real timeline, and leave with system fixes that have owners.
Step by step
- Set the tone. State that this is blameless: we look at systems, not people.
- Build the timeline. Ask what happened and what people expected, step by step.
- Ask how, not who. "How did this reach production?" not "Who approved this?"
- Find the system gap. Keep asking what made the mistake easy to make.
- Agree actions. Specific fixes, each with an owner and a date.
Say this, not that
| Before we start, a reminder that this is blameless. We're here to understand the system, not to judge anyone. | Saying it out loud gives people permission to be honest. | |
| Aditya, could you walk us through what you saw and what you expected at each step? | ||
| How did the change reach production without a review? | Who approved this change? | "Who" questions make people defend. "How" questions find causes. |
| Thanks. Let's not focus on Friday timing. What would have caught this on any day? | Redirect blame back to the system. | |
| So we have two actions. Who can own the validation check, and by when? | Let's all be more careful. | Actions without owners never happen. |
Model conversation
- You
- Thanks for coming. A quick reminder: this is blameless. We're here to understand the system, not to judge anyone. Aditya, could you walk us through what you saw and expected?
- Aditya
- I changed the pool size, expecting staging. It hit production. I'm really sorry, it's my mistake.
- You
- Thank you for being so open. How did the change reach production without anyone else seeing it?
- Aditya
- Config changes don't go through code review. And the staging and production files share one base file.
- You
- That's the real gap. What would have caught this, on any day of the week?
- Aditya
- A required review for config, and a check that flags changes to shared files.
- You
- Great. Who can own the validation check, and by when?
Practice it out loud
Reading is half of it. Fluent listens to you say it and scores your clarity, free, in your browser.