What it is
Summary: A circuit breaker monitors dependency outcomes and transitions between closed, open, and half-open states to prevent failure amplification.
In closed state, traffic flows. When timeout/error thresholds are exceeded, the breaker opens and immediately fails new calls. After a cooldown, half-open probes test recovery with controlled volume before closing again.
Why it matters
Summary: Without breakers, slow dependencies consume caller threads, connection pools, and retry budgets until healthy paths also fail.
The first dependency fault is rarely the full outage. The outage is usually created by repeated timeouts and retries across multiple services.
What problem it solves
Summary: Circuit breakers contain dependency collapse and prevent timeout-driven resource starvation in callers.
- Limits repeated timeout accumulation in service-to-service hops.
- Protects worker pools and connection pools from blocking saturation.
- Reduces retry storms that can keep a recovering dependency down.
- Creates clean failure semantics for fallback and degradation paths.
How it works
Summary: Breakers combine outcome classification, thresholds, and controlled recovery probes.
State model
- Closed: normal traffic with rolling error/timeout counters.
- Open: fail fast without touching dependency.
- Half-open: allow limited probes to validate recovery.
Signals and thresholds
- Error ratio by class (timeouts, 5xx, transport failures).
- Latency percentile breaches over a rolling window.
- Minimum request volume before evaluating open criteria.
Circuit breakers are not retry policy
Summary: Retries and breakers can conflict unless retry budgets, jitter, and stop conditions are designed together.
Naive clients retry immediately while the breaker is open, creating useless traffic and log noise. Retries should honor breaker-open responses, use bounded attempts, and back off with jitter.
Circuit breakers as a failure containment primitive
Summary: Place breakers at each dependency boundary where failure can propagate to a broader blast radius.
Breakers belong in service clients, not only at ingress. Pair with timeout budgets, bulkheads (isolated worker pools), and explicit fallback behavior for non-critical paths.
What it is not
Summary: A circuit breaker is not load balancing, not quota enforcement, and not dynamic queue control.
It does not solve fairness or high sustained demand. Those belong to rate limiting and backpressure.
Tradeoffs and constraints
Summary: Poor thresholds can create false-open events or delayed opening that still allows collapse.
- False-open risk if thresholds ignore transient errors during deploys.
- Late-open risk if windows are too long for fast incidents.
- Half-open stampede risk if probe volume is not capped.
- Hidden coupling when many services share one downstream breaker policy.
Where it fits in the system
Summary: Breakers sit on dependency edges and complement backpressure, not replace it.
Typical layering: ingress limits and admission control first, bounded queues and backpressure in workers, then circuit breakers per dependency call, and finally graceful degradation decisions for user-visible behavior.
Production scenarios
Summary: Breakers are most valuable where dependency slowness can fan out across large fan-in workloads.
- Control-plane orchestration: isolate one unhealthy inventory dependency so command execution remains available for unaffected workflows.
- Notification dispatch: open breaker on a failing provider API and route traffic to alternate channels where policy allows.
- Global migration tooling: preserve rollout control APIs even while one regional integration endpoint degrades.
Design guidance
Summary: Tune with production latency distributions, dependency error classes, and explicit fallback contracts.
- Define which errors should trip the breaker and which should not.
- Bound all dependency call timeouts before breaker thresholds are evaluated.
- Instrument open-rate, half-open recovery success, and timeout ratio per dependency.
- Load-test failure scenarios where retries and breaker behavior interact.
Closing summary
Summary: Circuit breakers keep dependency faults local by failing fast, preserving resources, and creating controlled recovery checkpoints.