Running an incident

Observability and operations Mid-level Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Order the steps

Order the first steps of an incident.

  • Declare the incident and name a commander
  • Post an update and schedule the postmortem
  • Confirm recovery on user-facing metrics
  • Mitigate: roll back or turn off the flag
  • Check recent deploys and config changes

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A E D C B

  1. Declare the incident and name a commander Clear ownership.
  2. Check recent deploys and config changes The most likely cause.
  3. Mitigate: roll back or turn off the flag Restore service first.
  4. Confirm recovery on user-facing metrics Errors and latency back to normal.
  5. Post an update and schedule the postmortem Close the loop.

30-second answer

Declare it and name an incident commander who coordinates rather than debugs. Open one channel and post short status updates on a schedule. Mitigate first: roll back the latest deploy, flip the feature flag, fail over or shed load, before hunting the root cause. Check the impact with your SLO dashboards, communicate to users through the status page, and keep a timeline. Afterwards, run a blameless postmortem.

Practise saying it · 35 s. No recording, just a timer

Handle an incident.

Say your answer out loud, then check which of these points you covered:

  • Declare, commander
  • One channel, updates
  • Mitigate first
  • Check recent changes
  • Postmortem

Deep dive

Roles: commander, operations lead, communications lead, scribe.

Most incidents follow a change; check recent deploys and config first.

Severity levels decide who gets paged and how often to update.

Practise with game days.

Common mistakes

  • Debugging for an hour before trying a rollback.
  • Ten people changing production at once.
  • No customer communication.

Senior take & follow-ups

Roles, one channel, mitigate first, communicate, timeline, postmortem.

Follow-up questions

Think of your answer first, then open the hint.

  1. Mitigate or find root cause first?

    Show hint

    Mitigate; restore service, then investigate.

  2. Why a commander who doesn't debug?

    Show hint

    Someone has to keep the overall picture and the communication going.

Similar on iOS: A release that causes data loss The iOS page handles an incident inside the app; here the response is for the backend fleet.

Related questions

Suggest a fix if something here is wrong or out of date.