Virtual Memory Is Deferred Work: Page Faults, RSS, and Why Allocation Numbers Lie

Every allocation your program makes is a promise the kernel keeps lazily. When you write make([]byte, 512<<20) in Go, you have not consumed 512 MB of physical memory — you have reserved a range of virtual address space and asked the kernel to materialize pages later, one fault at a time. The actual cost lands on the first write to each page, in the kernel’s page-fault handler, at a moment your profiler never attributes to anything. Understanding that deferred mechanism explains a whole family of mysteries: RSS numbers that disagree with your allocator, containers that get OOM-killed at numbers lower than their peak, and Go services whose memory graph behaves differently after a minor version bump.

This post walks through how allocations become physical pages, how to tell a cheap fault from an expensive one, why Go changed how it returns memory to the operating system, and where transparent huge pages help and hurt. Everything here applies to any Linux process; the examples use Go because its runtime makes the behavior unusually visible.

An allocation is not memory

When a userspace program needs memory, the runtime eventually calls mmap to reserve address space. That call is cheap: it edits page tables in the kernel’s bookkeeping and creates a virtual memory area, but it does not allocate physical pages. The pages exist as entries in the VMA’s backing description — anonymous memory mapped to the kernel’s zero page or marked as not-yet-present. The physical allocation happens the first time your code reads or writes a page within that range. The CPU takes an exception, the kernel’s fault handler finds the VMA, allocates a physical page, wires it into the page table, and returns. Your instruction retries and succeeds. That exception is a page fault, and for a freshly mmaped anonymous region it is a minor fault — no disk I/O, just kernel work and a page table entry.

You can watch the gap between reserving and touching with a small program:

package main

import (
	"fmt"
	"time"
)

func main() {
	big := make([]byte, 512<<20) // reserved; almost nothing faulted in
	time.Sleep(3 * time.Second)  // check RSS here: near zero

	for i := 0; i < len(big); i += 4096 {
		big[i] = 1 // one minor fault per 4 KiB page
	}

	time.Sleep(3 * time.Second) // check RSS again: roughly 512 MiB
	fmt.Println("touched", big[0])
}

Run it and poll RssAnon for the process during the two sleeps. During the first, the process has reserved half a gigabyte and resident memory stays small. During the second, RSS climbs toward 512 MB — one fault and one physical page at a time. The kernel did 131,072 units of deferred work on your behalf, each one triggered by an innocent-looking store instruction.

This is also why fork behaves the way it does: the child shares the parent’s physical pages through copy-on-write mappings, and the bill arrives when either side first writes to a shared page. A process with a large RSS that forks frequently pays that bill per page, which is exactly the mechanism behind the classic advice to keep Redis’ memory footprint bounded relative to available RAM.

Minor faults are cheap; major faults mean disk

Not all faults cost the same. A minor fault is resolved without disk I/O: the page is already in the page cache, the mapping can point at the shared zero page, or copy-on-write just needs one fresh page. A major fault requires reading from storage — the first access to a file-backed page nobody has touched yet, or a swap-in after the kernel reclaimed the page. Minor faults cost microseconds of kernel time; major faults cost whatever your storage costs, and during a latency-sensitive window they are indistinguishable from a slow disk.

The perf stat tool breaks these out directly:

perf stat -e page-faults,minor-faults,major-faults ./touchmem

The page-faults counter is the total of the other two. For a healthy long-running service, thousands of minor faults at startup are uninteresting — that is your binary and libraries being mapped in. Major faults in steady state deserve attention: they tell you the working set did not fit, that the kernel reclaimed something you needed, or that a lazily-mapped file section just got read from disk under request load.

For a point-in-time summary of one process, /proc exposes aggregated counters without attaching a profiler:

cat /proc/1234/smaps_rollup

The Rss, RssAnon, RssFile, and Swap lines answer the question “what is resident right now” — which is not the same question as “what has this process allocated,” and conflating the two is the root of most memory-monitoring confusion.

The Go runtime and the RSS that would not shrink

Deferred materialization has a mirror image: memory that looks resident long after your program stopped using it. When a garbage collector frees a heap region, the bytes are logically gone, but the physical pages still hold old contents and are still counted in RSS until the runtime explicitly tells the kernel it can reclaim them. That notification goes through madvise, and the flag you pass changes what the memory graph looks like from the outside.

With MADV_DONTNEED, the kernel drops the page mappings immediately, RSS falls right away, and the next touch of that address pays a fresh minor fault. With MADV_FREE, the pages are only marked as reclaimable: RSS stays high on your dashboards, but the kernel can silently drop them under memory pressure, and a read returns the (now zero) page without a disk hit. Both are correct; they look completely different in monitoring.

Go has shipped both. Release 1.12 switched the Linux runtime from MADV_DONTNEED to MADV_FREE, which made many containerized deployments appear to leak memory — RSS grew during traffic spikes and never fell, because the kernel had no reason to reclaim pages from a machine with free RAM. The Go 1.16 release notes reversed the default back to MADV_DONTNEED, precisely because the lazy variant was too confusing to operate. Services running Go 1.12 through 1.15 under Kubernetes were routinely over-provisioned based on metrics that measured reclaimability rather than actual pressure.

The operational takeaway is not “RSS is useless” but “RSS is a claim about the past, not a reservation.” Two better signals exist today: the runtime package’s MemStats exposes what the heap actually holds, and GOMEMLIMIT lets you declare a ceiling the GC actively targets, turning soft-memory accounting into something you can reason about in a container with a fixed limit.

Transparent huge pages: fewer faults, more spikes

Base pages on x86-64 are 4 KiB, which means half a gigabyte of memory is 131,072 page table entries and 131,072 potential faults. Huge pages collapse that: a 2 MiB page is one entry covering 512 base pages, which reduces TLB pressure and cuts fault counts by three orders of magnitude for large, densely-used regions. Transparent Huge Pages (THP) lets the kernel promote eligible regions to huge pages automatically, without application changes.

The trade-off shows up in tail latency. Coalescing 512 base pages into one 2 MiB page requires physically contiguous memory, and when fragmentation is high the kernel either spends time compacting or promotes lazily via the khugepaged thread. Worse, huge pages amplify copy-on-write: a fork-heavy process with 2 MiB pages now duplicates 2 MiB per touched region instead of 4 KiB per touched page. This is why several memory-intensive databases advise disabling THP in their deployment guides — the throughput win does not survive contact with fork-based persistence, and the compaction stalls land precisely on tail requests. Workloads that allocate large regions once and touch them densely tend to benefit; workloads that fork, or whose memory is sparse, tend to pay.

Like madvise, THP is also tunable per-advice: an application can hint MADV_HUGEPAGE or MADV_NOHUGEPAGE for specific regions when the system-wide mode is set to madvise, which is a reasonable middle ground for mixed workloads on shared hosts.

A practical measurement routine

Three habits cover most of this surface without heavy tooling:

• Measure faults, not just bytes. perf stat -e minor-faults,major-faults around a representative workload tells you whether memory costs are CPU-bound page table work or disk stalls.
• Read /proc/<pid>/smaps_rollup alongside allocator-level stats. If RSS and heap occupancy disagree, madvise policy, file-backed mappings, or lazily-touched regions are the usual suspects.
• Check THP mode on production hosts (cat /sys/kernel/mm/transparent_hugepage/enabled) and make it a deliberate choice rather than a host-image accident, especially for anything that forks.

The deeper point is that memory performance on Linux is not a single number. Allocation, residency, reclamation, and fault handling are four separate mechanisms with separate costs, and every monitoring dashboard that collapses them into one line is hiding at least one of them. The next time a service’s memory graph surprises you, the fault counters will usually tell you which mechanism moved.