Use this runbook when latency, refusal, resource queues or recovery loops suggest that effective demand has outrun finite service capacity. The first objective is not full diagnosis. It is to protect critical work while preventing the response from adding more load.
Declare early if: queue age rises across two measurement intervals; retry traffic exceeds 10% of attempts; a hard resource or dependency quota exceeds 85%; or two capacity controls are interacting unexpectedly. Tune these provisional gates to the service.
0–5 minutes · Bound the incident
- Name the user-visible critical path and the work allowed to fail closed, fail open, delay or degrade.
- Assign incident lead, operations owner and communications owner. Record every temporary control with owner and expiry.
- Freeze nonessential deploys, batch starts and experiments. Do not automatically disable safe rollback.
- Capture a before-state: throughput, latency, errors, queue age/depth, retry volume, concurrency and dependency health.
5–15 minutes · Locate queued work
| Layer | Inspect | Saturation evidence | Misleading comfort signal |
|---|---|---|---|
| CPU | Run queue, steal, throttling, per-core load | Runnable work and latency rise together | Average CPU below 100% |
| Memory | Live set, allocation, reclaim, GC, OOM | Reclaim or collector work crowds out service | Free bytes after killing workers |
| Disk / logs | Latency, queue, bytes, inodes, amplification | Writes wait or metadata cannot allocate | Capacity in bytes when inodes are gone |
| Network | Connections, ports, retransmits, backlog | Allocation/wait failures and retransmission | Aggregate bandwidth below link rate |
| Application pools | Active, waiting, timeout, service time | Wait time grows; completions fall | Pool fully utilized |
| Queues | Age of oldest, ingress, egress, expiry | Drain rate below arrival rate | Depth falls due to expiration |
| People / control plane | Approvals, access, API quotas, deploy queue | Recovery action waits or fails | Primary data plane looks stable |
15–30 minutes · Stop amplification
| Mechanism | Containment | Guardrail | Success signal |
|---|---|---|---|
| Client or service retries | Increase backoff, add jitter, reduce attempt budget | Preserve necessary at-most-once semantics | Retry share and duplicate work fall |
| Fan-out / expensive input | Cap per-request work, reject or defer bulk classes | Return explicit partial/refusal status | Downstream operations per request fall |
| Autoscaling / recovery | Pause the loop or cap change rate | Maintain minimum healthy capacity | Control-plane and warm-up demand fall |
| Error logging | Sample, aggregate or rate-limit repeated errors | Retain unique signatures and security events | Write rate drops without losing incident shape |
| Optional features | Disable via tested dynamic control | Record owner, scope and expiry | Critical-path service rate recovers |
| Background work | Pause intake; preserve durable cursor | Estimate reconciliation debt | Oldest critical work drains |
Decision gates
Scale only when all are true
- The constrained resource scales with the proposed unit.
- Warm-up and control-plane work do not consume the same bottleneck.
- The dependency can accept the increased concurrency.
- A measurable stop condition and rollback owner exist.
Use break-glass access only when all are true
- Ordinary access is unavailable or too slow for the declared impact.
- A human approver, incident identifier and audit trail exist.
- The action is narrow, reversible where possible, and has a second observer.
- Credentials or grants have an explicit expiry.
Stabilization checklist
- Arrival rate is at or below sustainable completion rate for critical work.
- Age of oldest critical item declines for three consecutive intervals.
- Retries and duplicate work are bounded.
- No temporary action is silently amplifying another scarce path.
- Ambiguous or dropped work is quantified and assigned.
- Temporary controls, permissions and flags have owners and removal conditions.
After the incident · Preserve adaptive evidence
Save the hypothesis timeline, control changes, missing access, ambiguous signals, near-misses and actions that bought time. Separate improvements to the base envelope from improvements to adaptive capacity. For each corrective action, ask what new resource it consumes and how it might participate in a future cascade.
Do not copy thresholds blindly. The provisional gates in this runbook are prompts. Establish service-specific values through load tests, dependency contracts and incidents.