Every language-level sync.Mutex rests on a guarantee the operating system makes for you: if a thread dies holding the lock, the kernel releases it. The moment the lock lives in Redis, etcd, or any other process on the network, that guarantee is gone — you cannot distinguish a holder that crashed from one that is slow, paused, or partitioned away. This is why distributed locking keeps generating horror stories, and why teams that treat the lock as a lease instead of a mutex end up with systems that survive the weird cases.
This post walks through what a distributed lock actually is (a lease), the failure modes that break naive implementations, and the escalating fixes: compare-and-delete releases, fencing tokens, and consensus-anchored sessions. All examples are in Go against Redis and etcd.
A Lock With an Expiry Is a Lease, Not a Mutex
The standard Redis recipe is SET with NX and PX: set a key only if it does not exist, with a time-to-live so the lock cannot leak forever when a holder dies. That TTL changes the semantics. A real mutex excludes everyone until the holder releases it. A keyed value with an expiry excludes everyone until the holder releases it or the expiry fires — whichever comes first. You have acquired the right to proceed for up to N seconds, not ownership of the resource.
That splits the failure modes in two. The first — the holder crashes — is the one the TTL exists for, and it works: the key evaporates, the next worker takes over. The second is nastier: the holder stalls without dying. A garbage collection pause, a VM migration, a network partition that later heals — the process resumes seconds later, convinced it still holds the lock. Meanwhile the TTL expired, another worker acquired it, and now two processes are mutating the same resource under the honest belief that they are alone. The lock never malfunctioned; the assumption that “holding the key” and “still running” are the same thing did.
The Naive Version, Done Correctly — and Why It Is Still a Lease
Even the simple version hides a classic bug: releasing the lock with a plain DEL. Worker A’s lease expires while it is still working; worker B acquires; A finishes and deletes the key — which is now B’s lock — and worker C immediately acquires it. For a moment, B and C both hold the lock even though every individual operation looked correct. The fix is a compare-and-delete: only delete the key if the value still identifies you. The value must be unique per acquisition (a random ID per holder, not a constant), and the check-and-delete must be atomic, which in Redis means a small Lua script:
package main
import (
"context"
"fmt"
"time"
"github.com/redis/go-redis/v9"
)
const releaseScript = `
if redis.call("GET", KEYS[1]) == ARGV[1] then
return redis.call("DEL", KEYS[1])
end
return 0
`
func Acquire(ctx context.Context, rdb *redis.Client, resource, owner string, ttl time.Duration) (bool, error) {
return rdb.SetNX(ctx, resource, owner, ttl).Result()
}
func Release(ctx context.Context, rdb *redis.Client, resource, owner string) error {
return rdb.Eval(ctx, releaseScript, []string{resource}, owner).Err()
}
func main() {
rdb := redis.NewClient(&redis.Options{Addr: "localhost:6379"})
ctx := context.Background()
owner := "worker-7-acquisition-4f2a"
ok, err := Acquire(ctx, rdb, "report-builder", owner, 30*time.Second)
if err != nil {
panic(err)
}
if ok {
defer Release(ctx, rdb, "report-builder", owner)
fmt.Println("holding the lease for up to 30s")
// ... exclusive work here ...
}
}Compare-and-delete fixes the “deleted someone else’s lock” bug, and it belongs in even the simplest setups. What it cannot fix is the overlap window itself: during the stall, A and B were both inside the critical section. Whether that matters depends on what the lock protects.
Efficiency Locks vs Correctness Locks
Most locks in practice are efficiency locks: they stop duplicate cron jobs from double-sending a report, or twenty replicas from rebuilding the same cache simultaneously. If the protected work is idempotent, a rare double-execution is an annoyance, not an incident — and the Redis lease is the right tool: cheap, one round trip, self-healing on crash. The engineering goes into making the downstream operation idempotent so the overlap window stays harmless.
A correctness lock enforces an invariant that must never break — “at most one process appends to this file”, “only one instance may process withdrawals for this account”. Here “usually exclusive” is not good enough, because the stall scenario violates the invariant with no error surfaced anywhere. You need fencing tokens or a consensus-anchored lease, decided at design time, not after the first incident.
Fencing Tokens: Making Stale Holders Detectable
The fencing technique moves the check into the resource itself. Every time the lock is granted, the holder receives a monotonically increasing number — the fencing token. Every write to the protected resource carries that token, and the storage layer rejects any write whose token is lower than the highest it has already observed. A stalled holder that wakes up after its lease expired still holds its old, smaller token, and its writes bounce off the storage layer instead of corrupting state. The token does not need to be dense — failed acquirers can burn numbers freely — it only needs to be monotonic. Redis INCR on a dedicated key is enough:
package main
import (
"context"
"fmt"
"time"
"github.com/redis/go-redis/v9"
)
// AcquireWithFencing returns (granted, fencingToken, err). The token is
// monotonic, so a delayed previous holder always carries a smaller number
// than the current holder.
func AcquireWithFencing(ctx context.Context, rdb *redis.Client, resource string, ttl time.Duration) (bool, int64, error) {
token, err := rdb.Incr(ctx, resource+":fencing").Result()
if err != nil {
return false, 0, err
}
ok, err := rdb.SetNX(ctx, resource, token, ttl).Result()
if err != nil {
return false, 0, err
}
if !ok {
return false, 0, nil
}
return true, token, nil
}
// StoreIfCurrent is the write path the protected resource exposes. It
// refuses writes from holders whose token is older than one already seen.
func StoreIfCurrent(ctx context.Context, rdb *redis.Client, key string, token int64, value string) error {
seen, err := rdb.Get(ctx, key+":seen").Int64()
if err != nil && err != redis.Nil {
return err
}
if token < seen {
return fmt.Errorf("stale lock holder: token %d, highest observed %d", token, seen)
}
pipe := rdb.TxPipeline()
pipe.Set(ctx, key, value, 0)
pipe.Set(ctx, key+":seen", token, 0)
_, err = pipe.Exec(ctx)
return err
}The catch is that fencing requires the storage layer’s cooperation. If the protected resource is your own PostgreSQL table, adding a token check is straightforward. If it is a third-party API with no notion of tokens, you cannot fence — your only defense is correctness-preserving design on top: idempotency keys, version checks, or an append-only log where duplicates are detectable later.
Consensus-Anchored Leases: The etcd Session
The third approach changes who decides the holder is dead. With a single Redis node, one node’s clock and reachability decide when the lease expires. etcd anchors the lease in a Raft-replicated store: the holder’s session is a lease the client keeps alive with heartbeats, and expiry is a fact recorded by a quorum of servers. The concurrency package builds a mutex on top: every contender creates an ephemeral key under a prefix, the key with the lowest creation revision holds the lock, and the rest watch for their predecessors to disappear.
package main
import (
"context"
"fmt"
"time"
clientv3 "go.etcd.io/etcd/client/v3"
"go.etcd.io/etcd/client/v3/concurrency"
)
func main() {
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"localhost:2379"},
DialTimeout: 5 * time.Second,
})
if err != nil {
panic(err)
}
defer cli.Close()
// The session owns a lease with a 15-second TTL. etcd keeps the lease
// alive while the client heartbeats; if the holder dies, the lease and
// its keys vanish and the next waiter takes over.
session, err := concurrency.NewSession(cli, concurrency.WithTTL(15))
if err != nil {
panic(err)
}
defer session.Close()
mutex := concurrency.NewMutex(session, "/locks/report-builder")
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
if err := mutex.Lock(ctx); err != nil {
panic(err)
}
fmt.Println("acquired:", mutex.Key())
defer mutex.Unlock(context.Background())
// ... exclusive work here ...
}When you do not want to queue, TryLock returns concurrency.ErrLocked immediately instead of waiting. And note what the session does not solve: the client process itself can still stall for twenty seconds after Lock returns and watch its session expire mid-work. Consensus improves the liveness decision — a quorum agrees the holder is gone, and lock ordering is total — but a paused process is invisible to any external observer. Correctness-critical paths still want fencing tokens on top, and holders should watch session.Done() before doing irreversible work.
Practical Rules
- Always set a TTL or a lease — a lock without expiry turns one crashed worker into a permanently stuck resource.
- Release with compare-and-delete, using an owner value unique to the acquisition. A plain DEL deletes whoever holds the lock now.
- Size the lease against your worst pause, not your average runtime; for work longer than the lease, extend it periodically instead of betting you will finish in time.
- Classify the lock at design time: efficiency lock → make the work idempotent and accept rare double-runs; correctness lock → fencing tokens on every write.
- Keep scope per resource key, and log acquisitions, expirations, and fencing rejections — the overlap window is invisible unless token rejections tell you it happened.
Wrapping Up
A distributed lock is a lease: a bounded right to proceed, not an ownership guarantee. Redis’s key-with-TTL is the cheapest form and adequate when the work is idempotent; fencing tokens upgrade it to something the storage layer can enforce; etcd sessions anchor the liveness decision in a quorum. None of them eliminates the possibility that your process stalls after acquiring — that risk lives in your runtime, not the lock service. Fence the writes, make the duplicates harmless, and treat mutual exclusion as a property you verify at the storage layer rather than assume at the front door.