Part of the Distributed Coordination series.
Start here: What Is a Distributed Lock? (With Examples)
1) Process crash after acquisition
What goes wrong: ownership lingers until lease expiry, delaying takeover.
Missing property: immediate owner cleanup.
Concrete example: nightly compaction owner crashes after grabbing lease and before persisting progress marker. New owner is blocked until TTL, then must recompute safety checks before continuing.
Technically, this is expected in lease systems: ownership cleanup is time-based, not crash-interrupt based. If TTL is long, recovery latency is long even when the system is otherwise healthy.
Mitigation summary: tune TTL to recovery goals and make work re-entry idempotent.
Severity: efficiency workloads: medium. correctness workloads: medium/high.
2) Lease expiry during long work
What goes wrong: ownership transfers while old owner is still active.
Missing property: bounded critical section duration.
Concrete example: schema migration step takes 40s but lease is 20s under load. A second worker acquires and starts same step while first worker is still committing chunked updates.
This happens when teams size TTL from median latency instead of tail latency plus jitter. Long-tail network and IO stalls turn “usually safe” leases into overlapping execution windows.
Mitigation summary: split long work and revalidate ownership at each mutation boundary.
Severity: efficiency: medium. correctness: high.
3) GC pause or runtime stall
What goes wrong: paused owner resumes with stale authority.
Missing property: stale-writer rejection at the write boundary.
Concrete example: stop-the-world GC freezes worker for 12 seconds while lease TTL is 8 seconds. Another worker acquires and advances state. Original worker resumes and flushes buffered writes from stale context.
Stop-the-world pauses, runtime safepoints, and scheduler starvation are common in real clusters. The lock service cannot distinguish “slow” from “dead”; it only sees missed renewals and proceeds.
Mitigation summary: enforce fencing/version checks in storage for every write path.
Severity: efficiency: low/medium. correctness: critical.
4) Network partition
What goes wrong: node believes it still owns while quorum moved on.
Missing property: single authoritative ownership view at mutation time.
Concrete example: holder loses quorum connectivity but retains DB connectivity. It continues mutating state locally while quorum elects a new owner in the coordination system.
This is a split-visibility problem, not a rare edge case. Different dependencies fail independently, so “lost lock visibility” does not imply “lost write capability.”
Mitigation summary: ensure writes require monotonic authority evidence, not local lock belief.
Severity: efficiency: medium. correctness: critical.
5) Mistaken unlock
What goes wrong: actor releases a lock it no longer owns.
Missing property: owner-bound release semantics.
Concrete example: client times out, retries acquire, then later executes stale unlock call against the same key and clears current owner. Another actor now enters critical section concurrently.
This usually comes from release operations that are not compare-and-delete on owner token. Blind delete patterns are fragile under retries and reordered async callbacks.
Mitigation summary: make unlock atomic with owner-token comparison.
Severity: efficiency: medium. correctness: high.
6) Degraded quorum and assumption drift
What goes wrong: either no ownership can be acquired, or teams assume stronger guarantees than the system actually provides.
Missing property: explicit degraded-mode policy and precise guarantees.
Concrete example: during quorum degradation, operators force fallback path that keeps jobs running without strong ownership checks. Incidents then shift from availability to correctness corruption.
The technical root cause is contract drift: system guarantees changed under failure, but application behavior did not. Undefined degraded mode is where many lock designs fail operationally.
Mitigation summary: define fail-closed behavior for correctness workloads before incidents happen.
Severity: efficiency: high operational impact. correctness: catastrophic if fail-open assumptions leak in.
Common pattern: “I still think I own the lock”
Most incidents reduce to this sentence. The process belief is stale, but execution continues. The coordination plane already moved on, while the worker continues acting on old authority.
Lease expiry does not stop execution. It only invalidates authority from the system’s perspective.
Why this breaks in real systems
GC pauses, network delay, clock drift, and process stalls are normal conditions at scale. If your design assumes those are rare enough to ignore, stale-owner behavior will eventually become a production incident.
What to Read Next
→ Read next: Fencing Tokens Explained.