Moving Beyond OOM Kills: Introducing Memory QoS in Kubernetes 1.37

For most of Kubernetes’ history, memory management has been a blunt instrument. Cross your limit, and the kernel kills your container. There has been no equivalent to CPU throttling, no graceful backpressure, just a hard stop. With Kubernetes 1.37, that changes: Memory QoS, built on cgroups v2, graduates to Beta and is enabled by default.

The problem with cgroups v1

Under cgroups v1, the container runtime could throttle CPU using shares, quotas, and periods, but it had no comparable lever for memory.

Worse, spec.containers[].resources.requests["memory"] was effectively ignored by the runtime. Memory requests informed scheduling decisions, but they did nothing to protect a running container from reclaim under pressure. Because memory cannot be compressed the way CPU time can, the only tool available when a container exceeded its limit was an OOM kill.

What Memory QoS changes

Memory QoS uses the cgroup v2 memory controller to give the kernel much richer guidance about how to treat a pod’s memory. Rather than a single hard ceiling, it maps pod resource requests and limits onto four cgroup v2 interfaces:

memory.max is the hard limit, and behaves the same as before: cross it, and the container is OOM killed.

memory.min provides hard protection from reclaim, guaranteeing a floor of memory that the kernel will not take away under pressure.

memory.low provides softer protection, similar in spirit to memory.min but more readily reclaimed under stronger system-wide pressure. Where memory.min is treated as a hard promise, memory.low is more of a preference: the kernel will generally leave this memory alone, but if the node comes under enough pressure and other, unprotected memory has already been reclaimed, the kernel can still reclaim into this range rather than let the whole node fail. This makes memory.low a reasonable default for workloads that would like some protection from noisy neighbours without permanently locking away memory the node might occasionally need.

memory.high is a throttling threshold. Exceeding it does not trigger a kill. Instead, the kernel applies reclaim pressure to slow the container down, buying it room to recover rather than terminating it outright.

In practice, this means pod memory requests now translate into real kernel-level protection through memory.min, while memory.high gives operators a way to throttle workloads that creep past their configured threshold instead of losing them entirely.

Configuring it

Two kubelet configuration fields control the behaviour.

memoryThrottlingFactor determines how memory.high is calculated relative to a container’s limit. Setting it to 1.0 effectively disables early throttling.

memoryReservationPolicy determines whether memory.min and memory.low protection are applied at all. The default, None, leaves them unset; operators who want tiered protection by QoS class need to opt in.

What is new in 1.37

The headline change in 1.37 is graduation to Beta, with the MemoryQoS feature gate now on by default. No explicit opt-in is required to get the throttling and protection behaviour. The defaults have also been deliberately chosen so that clusters upgrading to 1.37 do not suddenly see new throttling behaviour on existing workloads. Operators who want the additional protection can turn it on deliberately through memoryReservationPolicy.

A caveat worth knowing

Memory QoS requires cgroup v2, and a kernel version of 5.9 or later is recommended. On older kernels, memory.high throttling can trigger a known livelock bug, and the kubelet will log a warning at startup if the feature gate is enabled in that situation. Anyone planning to enable this in production should check kernel versions across their node pool first.

A sample manifest

Getting Memory QoS working takes two pieces: kubelet configuration on the node, and resource requests/limits on the pod. Memory QoS reads the pod’s requests and limits and translates them into the cgroup v2 interfaces described above, so a pod without a fairly typical requests and limits definition, there is nothing for the kubelet to translate.

Worth noting up front: none of the four cgroup interfaces are set directly in either manifest below. There is no field anywhere that says memory.min or memory.low or memory.high. Instead, the kubelet derives all three at runtime from ordinary requests and limits values, combined with the two KubeletConfiguration fields. memory.max comes straight from the container’s memory limit. memory.high is calculated by multiplying that limit by memoryThrottlingFactor. memory.min or memory.low (whichever applies, based on QoS class) comes from the container’s memory request, and is only written at all if memoryReservationPolicy is set to TieredReservation.

Kubelet configuration, applied on each node:

# kubelet-config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration

featureGates:
  # This is the feature gate that must be enabled for Memory QoS.
  MemoryQoS: true

# Default is 0.9, meaning emory.high is set at 90% of the limit. 
# Set to 1.0 to effectively disable early throttling.
memoryThrottlingFactor: 0.9

# Controls whether memory.min / memory.low protection is written at
# all. "None" (default) skips protection and only applies throttling
# via memory.high. "TieredReservation" applies protection based on
# the pod's QoS class (Guaranteed pods get memory.min, Burstable
# pods get memory.low).
memoryReservationPolicy: TieredReservation

Pod manifest, requesting and limiting memory so the kubelet has values to translate:

# memory-qos-demo.yaml
apiVersion: v1
kind: Pod
metadata:
  name: memory-qos-demo
spec:
 containers:
  - name: app
    image: registry.k8s.io/pause:3.9
    resources:
    requests:
     # Depending on QoS class, memory.min may be derived from the
     # pod's memory requests. It isn't explicitly specified.
     memory: "256Mi"
     cpu: "250m"
   limits:
    # Mapped to memory.max, the hard ceiling. Exceeding this
    # still results in an OOM kill. memory.high is likewise 
    # not set directly.
    memory: "512Mi"
    cpu: "500m"

Because requests equal limits for CPU but not for memory here, this pod lands in the Burstable QoS class. Under TieredReservation, that means the kubelet writes memory.low rather than memory.min. Setting memory requests equal to memory limits would move the pod into the Guaranteed class and would result in memory.min protection instead.

Two things worth checking before relying on this in production: confirm the nodes are running cgroup v2 with a 5.9+ kernel, and confirm the container runtime supports it, since containerd 1.6+ or CRI-O 1.22+ are required.

Conclusion

Memory QoS has been a long time coming: first introduced as alpha in 1.22, updated in 1.27, given tiered protection by QoS class in 1.36, and now reaching Beta in 1.37. The direction is clear. Kubernetes is steadily closing the gap between how it treats CPU and how it treats memory, giving operators a genuine middle ground between “fine” and “killed.”

Connect with me on Medium @david.b.chase