Systems fail under load, not only from code defects
A defect can trigger an incident, but overload is often what turns a local issue into a broad outage. The pattern is common: a dependency slows down, requests stay in flight longer, concurrency rises, queue growth starts, and tail latency climbs. Then retries arrive and push the system across a stability boundary.
From the outside, this looks sudden: “everything was fine until it wasn't.” Internally, the failure was building for minutes through saturation and backlog debt.
The mechanics of overload
Overload starts when arrival rate exceeds effective service rate for long enough. Effective service rate drops not just from traffic spikes, but from lock contention, cache misses, dependency slowdown, garbage collection, or connection pool exhaustion.
As utilization approaches saturation, latency becomes nonlinear. A small load increase can produce a large tail-latency jump. That extra latency keeps work in system longer, which further increases concurrency and queue depth. This is a positive feedback loop, not a linear degradation curve.
Average latency hides this. You need saturation signals: in-flight concurrency, queue age, pool wait time, timeout rate, and per-dependency error budget burn.
Amplification loops: retries, fan-out, shared dependencies
Three multipliers show up repeatedly in real incidents:
- Retry amplification: callers retry timeouts without budget limits, creating retry storms.
- Fan-out amplification: one request fans out to many downstream calls, so partial slowdown multiplies total work.
- Shared dependency collapse: multiple services rely on the same datastore, cache, or auth service, so one bottleneck destabilizes many paths at once.
A single slow dependency can therefore destabilize the entire request path even if every individual service is “healthy” by CPU average.
For a deeper API-centric framing, see API Backpressure Explained Simply. For queue-level mechanics, see Queue Design Under Load. For duplicate-work behavior, see Retry Strategies and Idempotency.
Why failures appear sudden even when they were building
Control planes and autoscalers react on minute-scale loops. Burst overload can destabilize a request path in seconds. By the time additional capacity comes online, queues may already hold expired work and clients may already be in retry mode. Autoscaling is useful, but it is too slow to be your only overload strategy.
Resource exhaustion usually appears in specific pools before host-level metrics look catastrophic:
- thread pools saturated by blocking calls
- connection pools exhausted by long dependency waits
- socket and file descriptor pressure
- memory growth from unbounded queues or large in-flight payloads
- CPU burn from timeout handling and duplicate request processing
This is why “we had spare CPU” does not prove the system had spare capacity.
What stable systems do differently
Stable systems are explicit about limits and refusal. They do not pretend capacity is infinite.
- Backpressure: communicate saturation quickly to upstream callers.
- Admission control: enforce front-door limits before deep queues form.
- Bounded queues: trade unlimited buffering for predictable latency behavior.
- Load shedding: drop low-value work first.
- Graceful degradation: preserve core user paths while optional features degrade.
- Deadline propagation: avoid completing stale work that cannot satisfy callers.
This is the operating model behind Backpressure in Distributed Systems: Stability, Correctness, and Graceful Degradation.
Engineering conclusion
Reliability work is not only about preventing defects. It is about managing system dynamics under pressure. The architecture choice that matters most during incidents is often not algorithmic correctness; it is whether the system can bound work, refuse excess demand, and recover quickly once pressure drops.
In practice, resilient systems survive because they reject work intentionally, isolate blast radius, and keep overload from propagating across service chains.
References
- Google Cloud. (n.d.). Addressing cascading failures. https://cloud.google.com/architecture/addressing-cascading-failures
- Amazon Web Services. (n.d.). Timeouts, retries, and backoff with jitter. AWS Builders' Library. https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
- Reactive Streams. (2015). Reactive Streams specification. https://www.reactive-streams.org/
- The Linux Foundation. (n.d.). OpenTelemetry semantic conventions. https://opentelemetry.io/docs/specs/semconv/
Closing summary
Summary: Systems fail under load when they lack explicit controls to bound work, manage amplification, and preserve critical paths. Reliability is not only about defects; it is about designing for pressure.