Every so often a profiler hands you a paradox. Your server is running eight worker goroutines, the load is perfectly even, no lock shows contention, the race detector is clean, and yet throughput stops scaling somewhere around two cores. The logic is correct. The synchronization is correct. The bottleneck is not in your code at all — it is in how the CPU physically moves memory between cores. The problem has a name: false sharing.
This post explains what cache lines are, why a coherence protocol quietly serializes your threads, what false sharing looks like in Go, how to measure the damage, and how padding fixes it. It is one of the few performance bugs where the fix is a few bytes of dead space and the payoff can be an order of magnitude.
Memory Moves in 64-Byte Chunks
CPUs do not read individual variables. When your code touches a single byte, the processor fetches an entire cache line — 64 bytes on virtually every modern x86 and ARM core — and stores it in cache. This is a deliberate design choice: most programs exhibit spatial locality, so hauling 64 bytes at a time amortizes the enormous cost of reaching main memory. An L1 cache hit costs on the order of a nanosecond; a trip to DRAM costs on the order of a hundred.
The consequence is that cache hardware is blind to your types and your ownership boundaries. It tracks 64-byte blocks. If two logically unrelated variables happen to sit inside the same block, the hardware treats them as one shared thing — whether or not your code ever shares them.
The Coherence Protocol That Keeps Caches Honest
Each core in a multicore chip has its own private L1 and L2 caches, and they must stay consistent. Hardware enforces this with a cache coherence protocol, the classic variant being MESI — each cache line sits in one of four states: Modified, Exclusive, Shared, or Invalid. The rule that matters here is brutally simple: when one core writes to a cache line, every other core’s copy of that line is invalidated. The next access on any other core must fetch the line again, typically from the writing core’s cache.
For genuinely shared data, this is exactly what you want. For unrelated data that merely lives side by side in memory, it is a disaster. Every write from core A evicts the line from core B, and every write from core B evicts it back. The line ping-pongs between cores — each hop costing tens of nanoseconds, memory traffic spikes, and your “parallel” workload quietly degrades into something closer to taking turns. The formal name in the C++ standard is hardware destructive interference, which is as good a description as any.
What It Looks Like in Go
Here is a pattern that shows up constantly: per-worker counters in an array, each worker incrementing only its own slot with an atomic operation. No locks, no shared writes, no data race.
package main
import (
"fmt"
"sync"
"sync/atomic"
)
var counters [8]atomic.Int64
func main() {
var wg sync.WaitGroup
for id := 0; id < 8; id++ {
wg.Add(1)
go func(slot int) {
defer wg.Done()
for i := 0; i < 1_000_000; i++ {
counters[slot].Add(1)
}
}(id)
}
wg.Wait()
var total int64
for i := range counters {
total += counters[i].Load()
}
fmt.Println(total)
}Now do the arithmetic the cache does. An atomic.Int64 occupies 8 bytes. Eight of them in an array occupy exactly 64 bytes — one entire cache line. All eight workers are hammering the same line millions of times. Every Add on worker 3 invalidates the copies cached by workers 0, 1, 2, 4, 5, 6, and 7. They hold no data in common, but the coherence protocol cannot tell the difference. This is “false” sharing: the sharing is an artifact of memory layout, not of your program’s logic.
It gets subtler. The same trap fires when two mutexes sit next to each other in one struct, when a metrics struct and a config flag share a line, or when an array of small structs is processed in parallel slices whose boundaries do not respect 64-byte alignment. Separate locks reduce logical contention; they do nothing about physical contention on the same line.
Measuring the Damage
A Go benchmark makes the effect concrete. Each goroutine is pinned to a fixed slot, first with the counters packed together, then with each counter isolated on its own cache line:
package stats
import (
"sync/atomic"
"testing"
)
type paddedCounter struct {
v atomic.Int64
pad [56]byte // 8 bytes for v + 56 bytes of padding = 64
}
func BenchmarkShared(b *testing.B) {
var counters [8]atomic.Int64
var next atomic.Int32
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
id := int(next.Add(1)-1) % len(counters)
for pb.Next() {
counters[id].Add(1)
}
})
}
func BenchmarkPadded(b *testing.B) {
var counters [8]paddedCounter
var next atomic.Int32
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
id := int(next.Add(1)-1) % len(counters)
for pb.Next() {
counters[id].v.Add(1)
}
})
}On an 8-core machine the shared version typically runs several times slower than the padded one, and the gap widens with core count because more cores are invalidating each other per unit of time. The exact ratio depends on the CPU’s topology and how many of the eight workers land on cores sharing an L3 versus crossing sockets, which is precisely what makes this bug so environment-dependent. A number that looks acceptable on your laptop can fall apart on a 32-core server with two NUMA nodes.
The Fix: Padding to the Line
The fix is to guarantee that independently written data occupies different cache lines. In Go there is no standard alignment attribute, so the idiom is explicit padding fields:
package main
import (
"fmt"
"sync/atomic"
"unsafe"
)
type paddedCounter struct {
v atomic.Int64
pad [56]byte
}
func main() {
fmt.Println(unsafe.Sizeof(paddedCounter{})) // 64
fmt.Println(unsafe.Offsetof(paddedCounter{}.pad)) // 8
}A few practical rules keep this from becoming cargo cult. Pad only fields that are genuinely written from different threads; padding everything bloats memory and pushes other data apart for no benefit. Verify with unsafe.Sizeof that the padded type is exactly 64 bytes, because the Go compiler will happily add or withhold alignment padding on its own terms, and a struct you believe is 64 bytes may not be. And keep the padding fields documented — an unexplained [56]byte will be deleted by the next well-meaning cleanup pass.
Other ecosystems have first-class support for this. C++ exposes std::hardware_destructive_interference_size so alignment does not have to hardcode 64, and Java offers the @Contended annotation, which the JVM honors by isolating annotated fields onto separate lines.
Finding It in Production
False sharing is invisible to the tools you would normally reach for. The race detector stays silent because there is no data race — every goroutine touches distinct memory. Mutex profiles show no contention because no lock is contended. CPU profiles just show your hot function burning more cycles than the work should cost.
The dedicated tool on Linux is perf c2c (“cache-to-cache”), which samples where loads and stores hit lines that were modified by another core and groups the offenders by address. The output pinpoints the exact struct offset that is ping-ponging, which tells you which padding to add. The cheaper diagnostic is the scaling curve itself: if per-thread work is independent but aggregate throughput drops as you add threads, coherence traffic is a prime suspect. For a Go service, exporting the counter values and watching their increment rates flatten under load is often enough to confirm it.
Where It Hides in Real Code
Three hotspots account for most of the cases in the wild. First, per-worker or per-shard metric structs stored in a slice — the exact shape of the example above, common in custom instrumentation and connection pools. Second, structs containing a hot atomic flag next to a counter owned by a different subsystem, such as a cancellation flag beside a stats field. Third, object pools and free lists where parallel producers write to adjacent bookkeeping fields. The common thread is ownership boundaries that are invisible in memory layout.
Wrapping Up
False sharing is a hardware-level story: caches move 64-byte lines, coherence protocols invalidate on write, and unrelated neighbors pay for each other’s traffic. When parallel code that touches disjoint data scales badly, check whether the disjoint data actually shares a cache line. Pad the hot fields, verify the sizes, confirm with a benchmark, and use cache-to-cache profiling when the layout is too complex to reason about by hand. Few performance fixes cost less — or pay more.