What this page covers
This is a practical guide to the failure modes that appear when cache-aside or CDN caching sits in front of asynchronously replicated data. The goal is not to ban caching. The goal is to make staleness explicit, measurable, and bounded so users do not observe contradictory system behavior.
Why engineers get surprised by this combination
Teams often model eventual consistency at the storage layer only, then treat the cache as a pure performance optimization. In production, cache freshness policy is part of the consistency model. If primary-to-replica lag is 1-3 seconds and cache TTL is 30 seconds, your user-visible staleness window is usually much closer to cache TTL than replication lag.
This gap is where incidents happen: dashboards disagree with detail pages, users read old policy state after a successful update, or tenant control-plane actions appear to "randomly" revert until caches expire.
Common production failure modes
1) Dual staleness windows (replica lag + cache TTL)
A write commits on the leader, but readers may still hit lagging replicas. If that stale value is then cached, staleness persists even after replicas catch up.
2) Cache-aside read-after-write races
Writer updates storage and invalidates cache asynchronously. Concurrent reads repopulate cache from stale replicas before invalidation propagates, extending inconsistency.
3) Multi-layer cache drift
Origin cache, service cache, and CDN each have independent eviction and purge behavior. A "successful" invalidation in one layer can still leave stale payloads in another region or edge POP.
4) Stampedes that mask correctness issues
Under miss storms, systems add aggressive TTL or stale-if-error behavior to protect origin. This can improve availability while silently increasing stale-read frequency beyond business tolerance.
Design principles that actually help
- Define a staleness budget: state an explicit max freshness lag per endpoint (for example: "profile read may be stale up to 5s").
- Version cache entries: include version or logical timestamp in cache keys when write frequency is high and correctness matters.
- Separate read classes: serve latency-optimized reads from cache, but route strict read-after-write flows to primary/leader paths.
- Invalidate with ownership: tie purge logic to the write path that owns entity state instead of scattering invalidations across callers.
- Measure stale reads: instrument freshness age, cache age-at-hit, and replica apply lag as first-class SLO inputs.
Concrete scenario: tenant policy updates
Suppose a control plane stores tenant policy in a quorum store and exposes read APIs behind a regional cache.
- Tenant updates policy at
t0; write commits on leader. - A read at
t0+200mshits a replica that has not applied the update yet. - Cache-aside logic writes that stale response with TTL 20s.
- Subsequent reads across the region remain stale until TTL expiry, despite storage convergence at
t0+1.5s.
From the user perspective, this looks like control-plane non-determinism. The fix is usually policy-specific: leader reads on mutation paths, shorter/conditional TTL for hot entities, and versioned invalidation keyed by policy revision.
Tradeoffs and decision points
| Choice | Benefit | Cost / Risk |
|---|---|---|
| Long global TTLs | Lower origin load | Longer stale windows and harder debugging |
| Short TTLs + jitter | Better freshness and smoother refill | Higher backend read pressure |
| Leader-only reads for critical flows | Stronger read-after-write behavior | Higher latency and hotspot risk |
| Versioned keys | Safer invalidation semantics | More key churn and memory pressure |
When this matters most
- Control-plane operations where users expect immediate reflectivity after writes.
- Billing, quotas, feature flags, and entitlement decisions.
- Incident periods, where protective cache policy changes can accidentally degrade correctness.
Related concepts
- What Is Eventual Consistency? — baseline model and terminology.
- Why Eventual Consistency Breaks Systems — where teams usually mis-specify guarantees.
- Designing Read-Your-Writes Guarantees — tightening user-visible consistency on critical flows.
- Retries, Idempotency, and Duplicate Writes — how retry behavior compounds stale-state risk.
- Backpressure in Distributed Systems — load controls that interact with cache refill and failover paths.
Conclusion
Cache policy is consistency policy. Treat freshness budgets, invalidation ownership, and read-path selection as explicit architecture decisions, not implementation details. That is the difference between "eventual consistency is acceptable" and recurring correctness incidents in production.