Most services read the same rows over and over. A user profile endpoint gets hammered by the same traffic pattern every minute; a product page renders the same category tree thousands of times per second. You already have a Redis cache in the stack, so you put the data there, set a TTL, and move on. Then a colleague asks why the dashboard shows 40,000 Redis GETs per second for data that changes twice a day, and nobody has a good answer.
The uncomfortable truth is that a network cache still costs a network round trip, a serialization step, and a pooler wait on every hit. Client-side caching eliminates that hop by keeping hot data in the application’s own memory, with an invalidation channel pushing updates when the source of truth changes. Redis has shipped first-class support for this since 6.0, and it works so well that several database clients now build it directly into their drivers. This post walks through how it works, how to implement it in Go from scratch, and the failure modes that turn a cache into a correctness bug.
Why a Redis Hit Still Costs Too Much
A cache hit against Redis looks cheap in your metrics and expensive in your p99. The request path for a single cached read is: borrow a connection from the pool, write the command, wait for the server to scan its own memory, read the response, and deserialize it into your structs. On a healthy network that is 0.3 to 1 ms. Multiply by the thousands of keys a single page render touches and the cache has become a second database — a faster one, but one with the same topology: a shared, networked, serialized bottleneck.
The numbers get worse under load. Connection pools saturate, and suddenly your “cached” endpoint is queueing behind writes. Redis becomes the service you have to shard, monitor, and capacity-plan for, even though the actual data is small and stable. The fix is not a bigger Redis box. It is moving the last mile of the read path into process memory, where a lookup is a map access measured in nanoseconds, and paying for coordination only when data actually changes.
Invalidation Is the Whole Design
The naive version of client-side caching is a local map plus a TTL. That works until two instances of your service hold different versions of the same profile, and a support ticket explains that a user saw their old avatar for fifteen minutes. TTL-only invalidation means every key has two clocks — the server’s and the client’s — and they will disagree.
Push-based invalidation closes that gap. The flow:
- The client tracks every key it reads, grouped by the logical table or prefix they came from.
- When a write happens anywhere in the fleet, Redis pushes an invalidation message on a dedicated connection to every subscribed client.
- Each client drops its stale entry immediately and re-reads from the source on the next request.
Redis 6.0 implemented exactly this with client-side caching: the server remembers which client fetched which key, and when that key changes it sends an invalidate message over the client’s own connection. Two modes exist. In default mode, the server tracks exact keys per client, accurate but memory-hungry at high cardinality. In broadcast mode, clients subscribe to a key-prefix pattern and the server pushes invalidations for any matching write, trading precision for bounded server memory. Most production fleets end up in broadcast mode, because with hundreds of app instances the per-client key tables stop scaling.
The design lesson generalizes beyond Redis. Whatever mechanism you choose, invalidation is the load-bearing wall: the local map is trivial, the correctness lives in how reliably and quickly you learn that your copy is stale.
One Connection Must Be a Dedicated One
Here is the detail that breaks real deployments: the invalidation stream is delivered on a push channel, and if you multiplex regular request/response traffic over that same connection, the protocol gets ambiguous — a reader expecting a GET reply may consume an invalidation push, or vice versa. The Redis docs are explicit that RESP3 push messages arrive out of band, and the clean pattern is one connection reserved solely for subscriptions and invalidations, with normal reads and writes going through the pooled connections.
In Go, that means a small supervisor goroutine owning the subscribe connection, reconstructing it on failure, and applying invalidations to a shared in-memory table under a mutex or via a single-writer channel. If that goroutine dies silently, clients keep serving stale data forever, so it needs to log loudly and restart with backoff. Stale data with no recovery path is worse than no cache at all, because nobody knows they are reading wrong values.
A Minimal Go Implementation
The sketch below uses go-redis v9, which exposes invalidation natively, or you can do it manually with a pub/sub connection in broadcast mode. The manual version makes every moving part visible:
package cache
import (
"context"
"strings"
"sync"
"time"
"github.com/redis/go-redis/v9"
)
type Entry struct {
Value []byte
ExpiresAt time.Time
}
// ClientCache is a process-local cache kept fresh by Redis
// broadcast invalidations on a key prefix.
type ClientCache struct {
mu sync.RWMutex
items map[string]Entry
rdb *redis.Client
prefix string
maxSize int
}
func NewClientCache(rdb *redis.Client, prefix string, maxSize int) *ClientCache {
return &ClientCache{
items: make(map[string]Entry),
rdb: rdb,
prefix: prefix,
maxSize: maxSize,
}
}
// Get returns a cached value, falling back to Redis on a miss.
func (c *ClientCache) Get(ctx context.Context, key string) ([]byte, bool, error) {
c.mu.RLock()
e, ok := c.items[key]
c.mu.RUnlock()
if ok && time.Now().Before(e.ExpiresAt) {
return e.Value, true, nil
}
if ok {
c.mu.Lock()
delete(c.items, key)
c.mu.Unlock()
}
// Miss: read through to Redis and populate the local copy.
val, err := c.rdb.Get(ctx, key).Bytes()
if err == redis.Nil {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
c.mu.Lock()
if len(c.items) >= c.maxSize {
c.evictOldestLocked()
}
c.items[key] = Entry{Value: val, ExpiresAt: time.Now().Add(5 * time.Minute)}
c.mu.Unlock()
return val, true, nil
}
// InvalidateLoop consumes broadcast invalidations forever.
// Run this in its own goroutine at startup; it must never share
// a connection with normal request traffic.
func (c *ClientCache) InvalidateLoop(ctx context.Context) error {
sub := c.rdb.PSubscribe(ctx, "__redis__:invalidate")
defer sub.Close()
for {
msg, err := sub.ReceiveMessage(ctx)
if err != nil {
return err
}
if msg.Channel != "__redis__:invalidate" || msg.Payload == "" {
continue
}
key := msg.Payload
if !strings.HasPrefix(key, c.prefix) {
continue
}
c.mu.Lock()
delete(c.items, key)
c.mu.Unlock()
}
}
Production code needs three additions. First, a real eviction policy — the evictOldestLocked placeholder should be LRU or a size-bounded random sample, and sync.RWMutex contention becomes visible past roughly 100K lookups per second, at which point a sharded map is the standard fix. Second, the fallback TTL of five minutes bounds staleness if the invalidation connection is down, which is why it must be short even though the underlying data changes rarely. Third, once a fleet uses broadcast mode, every writer must publish through the same Redis; a second write path that bypasses invalidation will poison every replica of the cache silently.
Also worth knowing: groupcache took a different route entirely — no invalidation, instead a single-flight fill plus peer-to-peer lookup so each key lives on exactly one node. That trade works for immutable or append-only data and shows how much complexity the invalidation channel carries: removing it removes most of the system.
Thundering Herd: The Classic Failure Mode
Two seconds after an invalidation for a hot key lands, a thousand concurrent requests miss and all of them hit Redis for the same value. That is a thundering herd, and on a popular key it can flatten the very backend the cache was protecting. The standard fix is single-flight: collapse concurrent fills for the same key into one underlying fetch.
package cache
import (
"context"
"sync"
"time"
"github.com/redis/go-redis/v9"
"golang.org/x/sync/singleflight"
)
type Entry struct {
Value []byte
ExpiresAt time.Time
}
type ClientCache struct {
mu sync.RWMutex
items map[string]Entry
rdb *redis.Client
maxSize int
sf singleflight.Group
}
// Get collapses concurrent fills for the same key into one
// Redis round trip per process.
func (c *ClientCache) Get(ctx context.Context, key string) ([]byte, bool, error) {
c.mu.RLock()
e, ok := c.items[key]
c.mu.RUnlock()
if ok && time.Now().Before(e.ExpiresAt) {
return e.Value, true, nil
}
v, err, _ := c.sf.Do(key, func() (interface{}, error) {
val, err := c.rdb.Get(ctx, key).Bytes()
if err == redis.Nil {
return nil, nil
}
if err != nil {
return nil, err
}
c.mu.Lock()
if len(c.items) >= c.maxSize {
c.evictOldestLocked()
}
c.items[key] = Entry{Value: val, ExpiresAt: time.Now().Add(5 * time.Minute)}
c.mu.Unlock()
return val, nil
})
if err != nil || v == nil {
return nil, false, err
}
return v.([]byte), true, nil
}
Single-flight deduplicates within one process. Across a hundred app instances, an invalidation still triggers a hundred simultaneous Redis reads — usually acceptable, but if a single fill is expensive (a complex SQL query behind the cache), layer a short negative-lead TTL or a lock-based fill so one instance fills while the others briefly serve the previous value or wait. Which trade is right depends on how expensive a fill is and how tolerant your product is to seconds-old data.
Deployment Topology and Pools
Two practical notes on how this fits a real deployment. First, go-redis handles the protocol plumbing for you: it enables client tracking via the CLIENT TRACKING command on RESP3 connections and routes invalidation pushes to an internal channel, so in most cases you subscribe to its events rather than parsing RESP3 yourself. The manual PSubscribe loop above is still worth understanding because it shows exactly what the library is doing — and library magic is exactly what fails silently when a proxy or pooler in the middle does not forward push messages.
Second, if Redis runs behind a proxy or in cluster mode, verify that invalidations survive the hop. Some connection poolers and older proxies buffer or drop push-type messages, which converts your carefully designed invalidation system into a TTL-only cache with no warning. The test is simple: write a key from one shell, and watch how fast a subscribed client in another process observes the invalidation.
When Client-Side Caching Is the Wrong Call
- Write-heavy keys. If the data changes every few seconds per key, invalidation traffic exceeds read savings. Keep those in Redis only.
- Huge value sizes. The whole point is nanosecond lookups; caching 5 MB blobs per key in process memory fragments the Go heap and stresses the GC.
- Fleets with bypassing writers. If some jobs write to Redis directly from scripts or other services and you cannot guarantee they go through the tracked path, the cache will serve stale values with no upper bound. Fix the write path first.
- Memory-constrained pods. A local cache competes with your service for the same memory limit. Size it deliberately and cap entries.
Wrapping Up
Client-side caching moves the last mile of your read path into process memory and pays for freshness with an invalidation channel instead of a round trip on every read. Redis’s built-in tracking — in default mode for exact keys, broadcast mode for prefix-based fleets — makes this a configuration problem rather than a distributed systems project. The implementation itself is small; the engineering lives in the invalidation path: dedicated connections, supervisor goroutines, a fallback TTL, single-flight fills, and a hard rule that every writer goes through the tracked path. Get those right and the 40,000 redundant GETs per second turn into a handful of invalidations — which is exactly the trade a cache was supposed to buy you.