An incident is running and the service is degraded. You have a pager, a group chat, and a strong pull to understand what is happening before you touch anything.
That pull is worth examining, because it is the reason well-run teams take an hour to do something that would have taken five minutes.
Restore service before you understand it
This is the core principle and it is worth stating plainly: mitigation beats diagnosis, every time, without exception.
An incident that is still running is an active cost to users, to the business, and to your on-call engineer’s sleep. A postmortem will produce the same understanding tomorrow at no additional cost. Restoring service now and investigating later strictly dominates investigating first, unless the mitigation itself is dangerous.
You are not debugging. You are buying time cheaply so that the eventual debugging can be done properly, without an audience and without a pager counting down.
What the first fifteen minutes should look like
Minutes 0–2: establish the blast radius
Three questions, in this order:
- Who is affected? All users, one region, one tenant, or just an internal tool?
- How bad is it? Slow, degraded, or completely down? This determines urgency, not the other way round.
- What changed recently? Deploys, config changes, certificate expiry, dependency update, traffic increase.
If someone has recently made a change, the fastest mitigation is frequently to undo it. You do not need to know why it broke to know it is the most recent variable.
Minutes 2–5: pick the boring option
When the fastest safe action is not obvious, take the one that returns you to a known-good state. Roll back the deploy. Shift traffic. Fail over. Scale up. Disable the feature flag.
Resist anything that requires understanding the new behaviour. A live experiment on production while users are affected is how a forty-minute incident becomes a four-hour one.
Minutes 5–10: communicate
Say what you know, what you are doing, and when you will update next. Even if all three are “investigating”.
The silence is what generates the anxiety. A short message every ten minutes, including the boring ones, is what stops a technical incident becoming an organisational one.
Minutes 10–15: verify, then hand over
Confirm the service is actually healthy rather than quiet, and check the graphs you did not look at during the panic. Then decide: continue, or escalate with a clear summary of what you have done and ruled out.
Escalating with a good summary is a skill, not an admission of failure. “Deploy at 14:02, rolled back at 14:09, error rate back to baseline at 14:11, now investigating why the deploy passed CI” saves the next person twenty minutes.
What makes it worse
Some habits consistently turn a manageable incident into a bad one.
Changing multiple things at once. Every simultaneous change destroys your ability to attribute the effect. One change, observe, then the next.
Debugging in the incident channel. The channel is for status and decisions. Deep investigation belongs in a side thread, summarised back. Otherwise the people making decisions cannot see what is happening.
Silent fixing. Someone restarts a service at 2am, says nothing, and service comes back. The team learns that the system is opaque and that people are taking undocumented risks on its behalf. This one corrodes trust permanently.
Declaring victory too early. “Fixed” before the graphs confirm recovery, followed by a second incident when the underlying condition recurs.
The instinct to diagnose before acting is a debugging instinct, and it is the wrong one here. You are an operator right now. Operators restore service and investigate later.
Why the first fifteen minutes reveal everything
How a team handles the opening quarter hour of an incident is a direct readout of how its systems are built:
- If nobody knows what changed, there is no deployment history or change log.
- If rollback is not obvious, deployment is not reversible.
- If nobody can say what is affected, there is no meaningful alerting.
- If communication needs permission, there is no incident process.
That is the useful insight. Making the first fifteen minutes calm is not primarily a people problem or a process-training problem. It is mostly a consequence of whether the platform supports fast, boring, reversible decisions.
What good looks like
A team doing this well has four things, and none of them are about seniority:
- A deploy that can be undone in under five minutes, rehearsed rather than assumed
- Alerts that fire on user-visible symptoms, so the blast radius is already known when the page arrives
- A runbook for the common cases, written when nobody was stressed
- Permission to roll back without asking anyone
Give a team those four things and the first fifteen minutes stop being the hard part. Take them away, and no amount of process or training compensates.