You operate Atlas, a multi-tenant document-analysis service. Monday traffic is 35% above forecast. The exercise is not a capacity quiz: it tests whether your team can recognize interacting limits, preserve optionality and reorganize when the original operating model stops working.
Facilitator setup · 15 minutes
- Roles: incident lead, operations, application, dependency liaison, communications and observer.
- Provide only the initial dashboard in Inject 0. Reveal later evidence when requested or at the marked time.
- Score observable coordination, not whether participants guess the hidden mechanism.
- Timebox the scenario to 55 minutes and the debrief to 25 minutes.
Inject 0 · Normal, but closer to the edge T+00
P95 latency is 420 ms against a 500 ms objective. CPU is 72%. Queue depth is stable. A new bulk-import client begins sending 18,000-item manifests, which are valid and within the documented API limit.
Prompt: What evidence distinguishes high utilization from saturation? What must remain available if load rises?
Inject 1 · A locally sensible response T+08
P95 reaches 2.6 s. HTTP timeouts rise. Autoscaling adds workers. Database connections climb to 94% of pool capacity; the queueʼs oldest item is now 80 seconds old although depth fluctuates.
Evidence on request: each manifest item fans out to three object-store reads. Clients retry twice after 2 seconds. New workers open fresh connections during warm-up.
Decision gate: Continue scaling, cap concurrency, reject bulk work, alter retries, or choose another action. State what scarce resource the action consumes.
Inject 2 · The recovery path joins the incident T+18
Verbose error logging was enabled to aid diagnosis. Log throughput rises 14×. Two nodes hit inode warnings; garbage-collection CPU rises. Deploying a configuration change fails because the deployment controller has exhausted an API quota while autoscaling.
Constraint: normal deployment and normal privilege elevation are unavailable. Break-glass access exists but requires two-person approval and an incident identifier.
Decision gate: What gets disabled, delayed or bypassed? Who owns the risk? How will the team know whether the action helped?
Inject 3 · Conflicting signs T+30
CPU falls to 55%, yet interactive latency worsens. Queue depth falls, but age of oldest work rises. A dependency dashboard is green. Customer support reports duplicate imports.
Evidence on request: workers are blocked waiting for database connections; expired items leave the head of one queue while older work remains in a lower-priority partition. Retried requests are not idempotent.
Prompt: Which metric was an attractive red herring? Rewrite the incident model in one sentence.
Inject 4 · Stabilization is not recovery T+42
Bulk intake is paused, retry budgets are zeroed, and interactive traffic is recovering. There are 11,000 ambiguous import operations, elevated manual-support load and a feature flag that cannot safely remain off overnight.
Decision gate: Define stable state, recovery owner, reconciliation plan, customer message and conditions for re-enabling work.
Debrief
- Where did demand get amplified?
- Which mechanism intended to help became another consumer?
- Which control existed but was not usable quickly?
- When did the team change its operating strategy rather than merely add capacity?
- What did observers learn about expertise, authority and coordination?
- Which corrective action might create a future incident?