Learning objectives
By the end of 75 minutes, learners can:
- Distinguish utilization, saturation and failure.
- Trace amplification through retries, fan-out, logging and automated recovery.
- Explain base capacity versus graceful extensibility.
- Propose one safety valve and evaluate its tradeoff.
- Describe incident evidence that preserves responder reasoning.
Lesson sequence
| Minutes | Activity | Instructor move | Evidence of learning |
|---|---|---|---|
| 0–8 | Boundary prediction | Ask: "What fails first in a healthy service under 4× demand?" | Learners name a finite resource and observable queue. |
| 8–20 | Mini-lesson: saturation | Use Frank–Starling and competence-envelope frames. | Learners distinguish a boundary from an error. |
| 20–35 | Feedback-loop tracing | Give the retry scenario below. | Groups label reinforcing edges. |
| 35–50 | Incident comparison | Assign Bluesky, Slack or Waymo. | Groups identify demand, finite path and adaptation. |
| 50–65 | Control design | Require one guardrail and reversal signal. | Proposals state what the control consumes. |
| 65–75 | Retrieval quiz | Use questions before revealing answers. | At least 6/8 with corrected reasoning. |
Exercise A · Complete the loop
A dependencyʼs P99 rises above the caller timeout. Clients retry twice. Workers remain occupied while waiting. Autoscaling adds callers.
- Draw four nodes and arrows that make the reinforcing loop explicit.
- Mark the first quantity that queues.
- Choose one edge to cut. State a user-visible cost.
- Name a metric that can improve while the incident worsens.
Exercise B · Classify the response
For each, decide whether it expands the base envelope, creates optionality, activates adaptive capacity, or does more than one:
- Add headroom to a connection pool after load testing.
- Give incident commanders a tested, expiring traffic-block control.
- Teach responders how to isolate one tenant without redeploying.
- Write a postmortem that preserves abandoned hypotheses and red herrings.
Quiz
- Can a resource be saturated below 100% reported utilization? Explain.
- Why can a bounded queue be a safety feature?
- What made Slackʼs autoscaling response relevant to saturation?
- What is the risk of logging every error during a failure?
- How does a slow leak hide behind deployment?
- What separates a feature flag from usable adaptive capacity?
- Why does the talk resist "prevent every incident" as a learning strategy?
- What should a postmortem preserve besides corrective actions?