Every Kubernetes operator eventually meets the same 3 a.m. page: a pod stuck in a restart loop, kubectl describe shows Reason: OOMKilled, and nobody knows whether the app has a leak, the limit is too low, or the node is starving. The frustrating part is that “OOMKilled” can mean two completely different things, and the fixes are opposite — raise the container’s limit in one case, and lower the pod’s appetite (or raise its QoS class) in the other.
This post walks through the machinery Kubernetes uses to decide which processes die: the kernel’s cgroup-level OOM kill, the kubelet’s node-pressure eviction, how QoS classes and oom_score_adj shape those decisions, and the newer cgroup v2 knobs (memory.high throttling and tiered memory reservation) that change the picture on current clusters.
Two Different Ways a Pod Dies of Memory
The first mechanism is per-container. When you set a memory limit in the pod spec, the kubelet translates it into the cgroup memory.max control for the container. If the processes inside that cgroup try to allocate more than memory.max allows, the kernel invokes the OOM killer against that cgroup and kills the offending process. The container exits, the kubelet records OOMKilled as the termination reason with exit code 137, and (assuming a Deployment manages the pod) restarts the container. Only the offending container is affected. This is the case documented under resource management for containers.
The second mechanism is node-wide. When the node itself runs low on reclaimable memory, the kubelet steps in and evicts entire pods to relieve the pressure. This is node-pressure eviction, and it behaves very differently: the kubelet selects victims based on QoS class and resource usage, terminates the whole pod (all containers), and does not respect your PodDisruptionBudget or terminationGracePeriodSeconds. With hard eviction thresholds the grace period is 0 seconds.
Distinguishing the two is the first diagnostic step. A container that got OOMKilled by its own limit shows Last State: Terminated, Reason: OOMKilled, Exit Code: 137 on a specific container. A node-pressure eviction shows the pod’s phase as Failed with reason Evicted. They look similar in a Slack alert and demand opposite remedies.
What the Kubelet Actually Measures
When the kubelet evaluates memory pressure, it does not run free -m. It computes memory.available from the cgroup filesystem as node capacity minus the node’s working set, which is the memory the kernel considers genuinely unreclaimable. Notably, the kubelet excludes inactive_file — file-backed pages on the inactive LRU list — from the working set, on the assumption that page cache can be reclaimed under pressure. This is why a node can look “full” in monitoring dashboards (because RSS plus page cache is high) while the kubelet considers it perfectly healthy, and vice versa.
The kubelet polls cAdvisor for these stats on a regular interval, which creates a race: memory can spike to exhaustion faster than the polling window catches it, and the kernel OOM killer fires anyway. The --kernel-memcg-notification kubelet flag makes the kernel notify the kubelet immediately when thresholds are crossed, closing that gap. There is also a subtler trap: active_file memory is treated as non-reclaimable, so workloads with heavy block-backed local I/O can accumulate page cache on the active LRU list and make the kubelet see phantom memory pressure. Setting the container’s memory request equal to its limit works around it.
QoS Classes: How Requests and Limits Set Your Fate
Kubernetes assigns every pod a QoS class at creation time, derived purely from the resource requests and limits on its containers. The class never changes for the lifetime of the pod — even an in-place resize that would alter it is rejected. The three classes map directly onto eviction behavior:
- Guaranteed — every container has CPU and memory requests that equal its limits (all values positive). These pods are killed only after BestEffort and Burstable candidates are gone, and only when they exceed their own limits.
- Burstable — at least one container sets a request or limit, but the Guaranteed criteria are not met. Evicted after BestEffort pods, and only when using more than their requests.
- BestEffort — no requests or limits at all. First to go under any node pressure.
A pod whose containers set only memory limits without requests still gets a request auto-assigned equal to each limit — that alone does not make it Guaranteed unless CPU is also fully specified. A “limits only” pod is Burstable, competing for the middle tier rather than the protected tier.
oom_score_adj: The Kernel-Level Pecking Order
QoS classes are not just a label for the kubelet’s own bookkeeping. The kubelet writes a per-container oom_score_adj value — the Linux kernel’s hint for its OOM killer — based on the pod’s class. The values come straight from the kubelet source. The kubelet and kube-proxy run at −999, Guaranteed pods at −997 (killed last), BestEffort pods at +1000 (killed first). Burstable pods get a computed value: 1000 minus the container’s memory request as a fraction of node memory capacity. A container requesting 10% of node memory gets an adjustment of 900; once it consumes more than its request in absolute terms, its OOM score hits the maximum and it becomes a prime target for the kernel OOM killer.
This is worth internalizing because it means the kernel OOM killer — not the kubelet — is usually the one that executes Burstable pods on a node that suddenly runs out of memory, and the request you set directly tunes how protected that container is. To put a critical service in the protected tier, make it fully specified:
apiVersion: v1
kind: Pod
metadata:
name: payments-api
spec:
containers:
- name: server
image: registry.example.com/payments-api:1.42.0
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "1Gi"
cpu: "500m"With requests equal to limits on both dimensions, this pod is Guaranteed: oom_score_adj −997, last in line for both the kernel OOM killer and kubelet eviction.
cgroup v2: Throttling Before Killing
On cgroup v2 nodes, recent Kubernetes versions add a middle option between “fine” and “dead”: the Memory QoS feature (beta since v1.37, enabled by default, controlled via the MemoryQoS feature gate). It uses two mechanisms.
First, memory throttling. When the kubelet configuration sets memoryThrottlingFactor (a value between 0 and 1), the kubelet computes a memory.high soft limit for Burstable containers:
memory.high = requests + memoryThrottlingFactor × (limits − requests)
With a factor of 0.9, a container with a 256 MiB request and a 1 GiB limit gets memory.high set to roughly 947 MiB. Past that point the kernel doesn’t kill the container — it slows allocations down by forcing synchronous reclaim, giving the process a chance to garbage-collect or shed load before it reaches memory.max. Guaranteed containers get no memory.high (their request equals the limit, so the formula degenerates), and BestEffort containers get one derived from node allocatable memory.
Second, tiered memory reservation. With memoryReservationPolicy: TieredReservation, the kubelet sets memory.min for Guaranteed pods (memory the kernel will never reclaim) and memory.low for Burstable pods (memory retained preferentially), and mirrors the protection up the cgroup hierarchy so it actually holds. Both features require Linux with cgroup v2; kernel 5.9 or newer is recommended because older kernels have a known throttling livelock.
There is a caveat for page-cache-heavy workloads: for a Guaranteed pod, memory.min equals memory.max, so the kernel has little reclaimable headroom inside the cgroup. If the workload relies on a large page cache, it can be OOM-killed before the cache is reclaimed — size the limit with headroom, as the QoS documentation warns.
Diagnosing an OOM Incident in Under Two Minutes
The fastest triage path uses three commands:
kubectl describe pod payments-api-7d9f6b-x2r4n | grep -A 5 "Last State"
kubectl get pod payments-api-7d9f6b-x2r4n -o jsonpath="{ .status.qosClass}"
kubectl top pod payments-api-7d9f6b-x2r4n# run on the node for the cgroup-level view of a container
cat /sys/fs/cgroup/kubepods.slice/.../memory.max
cat /sys/fs/cgroup/kubepods.slice/.../memory.high
cat /sys/fs/cgroup/kubepods.slice/.../memory.eventsIf Last State shows OOMKilled with exit code 137, the container exceeded its own memory.max — check memory.events for the oom_kill counter and compare steady-state usage from memory.current against the limit. If the pod’s phase is Evicted, the kubelet made a node-level decision — check which QoS class the pod had and whether it was using more than its requests. The qosClass from the jsonpath query tells you which tier the pod competed in.
A Short Checklist
- Set memory requests to observed steady-state usage (not peak), and limits with enough headroom for spikes — and page cache if you read many files.
- Make latency-critical services fully specified (requests equal limits, CPU included) so they land in Guaranteed.
- Never leave a production pod BestEffort; it is first in line for both eviction and the kernel OOM killer.
- On cgroup v2 clusters, consider
memoryThrottlingFactorso Burstable pods degrade instead of dying at the limit. - If you see evictions with no real memory shortage on I/O-heavy pods, suspect the
active_fileaccounting quirk and pin requests to limits. - For a memory-hungry runtime, pair the cgroup limit with the runtime’s own soft limit (for example GOMEMLIMIT for Go) so the process sheds load before the kernel acts.
OOMKilled pages stop being mysterious once you know which kill switch fired. The memory resource assignment guide covers the configuration side; the eviction and QoS references above cover the decision side. Between them, every restart loop has an explanation — usually a one-line fix.