Make zram swap size proportional to node RAM (currently hardcoded to 1 GiB)

[alex-berger]

What I'd like:

That the zram swap device introduced in
core-kit#590 should be sized such that it cannot mask kubelet's memory.available eviction signal — i.e. sized relative to the node's RAM and to the configured eviction thresholds, rather than as a fixed constant.

What actually happened:

prepare-swap.service creates a fixed 1 GiB device on every node, regardless of effective RAM and regardless of the eviction thresholds in effect:

ExecStart=/usr/sbin/zramctl /dev/zram0 --size 1G

Verified unchanged on 1.63.0 (16 GiB r7g.large):

# cat /sys/block/zram0/disksize
1073741824
# cat /proc/sys/vm/swappiness /proc/sys/vm/page-cluster
200
0

Because swapped-out anonymous pages leave workingSet, and kubelet computes memory.available = capacity − workingSet, the device is a blind spot in the eviction signal of up to its own size. When it is large relative to the eviction threshold, neither eviction nor the OOM killer fires and the node thrashes indefinitely instead of shedding load.

The default configuration cannot trigger hard eviction

This is the part I'd most like maintainers to look at, because it is not a small-instance edge case — it affects every node running stock settings. Unless of course, my analysis is wrong ;-).

The kubelet config template emits eviction thresholds only if the user set
them (kubelet-config):

{{#if settings.kubernetes.eviction-hard}}
evictionHard:
  {{#each settings.kubernetes.eviction-hard}}
  {{@key}}: "{{this}}"
  {{/each}}
{{/if}}

Bottlerocket ships no default, so a stock node falls back to kubelet's own built-in default — memory.available<100Mi, absolute — and no soft thresholds at all.

Now consider a full zram device. Its pages are compressed into zsmalloc memory, which is unaccounted kernel memory: it occupies physical RAM but appears in nobody's workingSet. So with a device of size Z and compression ratio r, kubelet's reported memory.available has a hard floor of roughly Z / r, and cannot go below it no matter how much memory the workload demands:

compression ratio	floor on memory.available with Z = 1 GiB
r = 1 (incompressible; zram stores such pages raw)	1024 MiB
r = 3.0 (measured, low end)	341 MiB
r = 4.1 (measured, high end)	250 MiB

Every one of those is above the 100 MiB default threshold. So on a stock Bottlerocket node with a saturated zram device, memory.available never reaches evictionHard, at any instance size — 2 GiB or 768 GiB alike.

Two defaults that ship together are mutually incompatible: swap-on-by-default and kubelet's 100 MiB absolute hard-eviction default. Whichever way this issue is resolved, I think those two need to be reconciled.

Why a fixed size is the wrong shape of knob

For users who do configure percentage thresholds, the failure becomes size-dependent instead of universal. A threshold is only reachable if its band exceeds the device. With our thresholds (hard 5% / soft 15%):

Nominal RAM	hard band (5%)	soft band (15%)	hard reachable?	soft reachable?
4 GiB	205 MiB	614 MiB	❌	❌
8 GiB	410 MiB	1229 MiB	❌	⚠️ 205 MiB margin
16 GiB	819 MiB	2458 MiB	❌	✅
32 GiB	1638 MiB	4915 MiB	✅	✅

(Nominal RAM; kubelet's capacity is ~4–6% lower.)

The core point is scale invariance. A fixed 1 GiB device is safe only above some node size — ~20 GiB for a 5% hard threshold, ~6.7 GiB for a 15% soft one. A device sized as a fraction of the threshold is safe at every size, which is the property a node OS needs, since it cannot know what instance types or thresholds its users will run.

Note also that #4735's own validation was performed on a 7.6 GiB node — right at the boundary where the soft band stops exceeding the device.

What this looked like in production

A c5a.large (3.7 GiB) node went into permanent reclaim thrash. kubelet reported 46% memory available while only 91 MiB was actually free, at ~29,700 page ops/s. No eviction and no OOM kill ever occurred; the node simply degraded until we replaced it.

Measurements, including an honest accounting of what masked what

• memory PSI full avg300: 21.71%; io PSI full: 6.99%
• zram: 318 MiB of anon parked; compression ratio 3.0–4.1× (mm_stat)
• page cache: ~1.8 GiB, hot and constantly refaulting
• A healthy node of identical uptime and instance family: pswpin/pswpout = 0
• Collateral damage: our monitoring agent's probes timed out → exitCode 137,
  1524 restarts. That is what led us to investigate.

To be precise about causation: on this node the larger share of the
memory.available inflation was inactive page cache, not zram — that is the long-standing kubernetes#43916 blind spot, which is not Bottlerocket's doing. zram contributed the smaller share directly.

Its real contribution was to make the state stable. With vm.swappiness = 200 and a 1 GiB absorber, the kernel had somewhere to keep putting anon pages, so it ping-ponged indefinitely rather than being forced into reclaim that would have resolved the situation one way or the other. The two blind spots also compose — which is the main reason the proposal below sizes the device at a fraction of the threshold rather than up to it.

After moving the same workloads to 16 GiB nodes (no other change): memory PSI full 0.00 (total = 0), zram used 0, 0 restarts.

We worked around this by enforcing a ≥16 GiB floor on all node pools — a blunt and expensive workaround for what is really a sizing constant, and one that is unavailable to anyone whose workloads legitimately belong on small nodes.

Existing workarounds are all unpleasant

• There is no supported setting to resize or disable the device.
  settings.kubernetes.memory-swap-behavior (v1.42.0) governs pod swap only
  and has no effect on the host device.
• The workarounds circulating in Unresponsive/unreachable Bottlerocket EKS nodes #4075 are a bootstrap container running
  swapoff / zramctl -r / modprobe -r zram (with a retry loop for
  Device or resource busy), or vm.swappiness=60 via EC2NodeClass userData.
  Both fight the OS rather than configure it, and the latter does not address the
  accounting blind spot at all.
• Upstream Kubernetes will not solve this: making the eviction manager
  swap-aware is an explicit Non-Goal of KEP-2400,
  kubernetes#129578 was
  closed unmerged, and no PSI-based eviction signal is proposed.

Proposed change

Note

Maybe there are better solutions to this problem than what I propose here (I am really not a subject matter exper neither on ZRAM nor on kubelet's eviction feature). So, please do not hesitate to bring up more suitable alternatives.

The invariant

Size the device so that a full device still leaves memory.available able to cross the tightest threshold that is meant to fire:

Z  ≤  α · T        where  T = min over the configured memory.available
                              eviction thresholds (hard and soft),
                              resolved to bytes against MemTotal

α = 0.5 is the value I'd suggest. The floor on memory.available is Z / r, so even at the worst case r = 1 (incompressible data, which zram stores raw) α = 0.5 leaves a 2× margin — and that margin is what absorbs the page-cache blind spot the two effects compose into.

Taking min(hard, soft) rather than one or the other is what makes both thresholds keep working, which is the property we actually want.

Option A — derive the size from the thresholds

Z = min( α · T , Z_max )        Z_max = 4 GiB

With hard 5% / soft 15%, so T = 5% · MemTotal:

Nominal RAM	T	proposed Z	today's Z
4 GiB	205 MiB	102 MiB	1024 MiB (10.0× too large)
8 GiB	410 MiB	205 MiB	1024 MiB (5.0× too large)
16 GiB	819 MiB	410 MiB	1024 MiB (2.5× too large)
32 GiB	1638 MiB	819 MiB	1024 MiB (1.25× too large)
64 GiB	3277 MiB	1638 MiB	1024 MiB (too small)
128 GiB	6554 MiB	3277 MiB	1024 MiB (too small)

The constant is wrong in both directions — it defeats eviction below ~32 GiB and under-provisions the absorber above ~64 GiB. That asymmetry is, I think, the clearest argument that this should not be a constant.

If the computed size falls below a small floor (~64 MiB), I'd suggest simply not enabling swap: at that point the device cannot help, and silently clamping it upward would reintroduce the bug.

Implementation. The thresholds are already settings, so this needs no new
API surface:

1. Render the existing values into an env file via the normal template
   mechanism, e.g. /etc/zram.env:

   EVICTION_MEMORY_AVAILABLE_HARD={{#if settings.kubernetes.eviction-hard}}...{{/if}}
   EVICTION_MEMORY_AVAILABLE_SOFT={{#if settings.kubernetes.eviction-soft}}...{{/if}}
   

2. Replace the literal --size 1G with a ~25-line shell helper that reads that
   file plus MemTotal from /proc/meminfo, handles both N% and NMi/NGi
   forms, and computes Z. No new Handlebars helper needed — the arithmetic
   stays in shell, the template only passes the raw strings through.
3. Reorder the unit. This is the one real cost, and it is small:

   -DefaultDependencies=no
   -Wants=modprobe@zram.service
   -After=modprobe@zram.service dev-zram0.device
   +Wants=modprobe@zram.service
   +After=modprobe@zram.service dev-zram0.device settings-applier.service
   
    [Install]
   -WantedBy=sysinit.target
   +RequiredBy=preconfigured.target

   settings-applier.service (thar-be-settings --all) is where every other
   templated config is written, and it is RequiredBy=preconfigured.target,
   while kubelet.service waits on configured.target. So the device would
   still be online well before kubelet starts, in the same phase as the rest of
   the settings-derived configuration. Nothing in early boot needs swap — the
   non-swap variants have never had any at that point.

The default case still needs a decision

Deriving from thresholds does not by itself fix a stock node, because a stock node has T = 100 MiB absolute, which yields Z = 50 MiB — effectively disabling the feature. That is a real tension and I don't think it can be papered over: a meaningfully sized zram device is incompatible with kubelet's default 100 MiB absolute hard-eviction threshold.

So I'd suggest pairing the change with a default percentage threshold shipped alongside the swap feature, chosen so the invariant holds by construction — e.g. defaulting settings.kubernetes.eviction-hard to memory.available: 10% when release-swap is in the variant. Then Z = 5% · MemTotal satisfies Z ≤ 0.5·T on every node, at every size, with no user configuration.

If shipping a default threshold is unpalatable, the fallback is a plain RAM-proportional size — Z = min(5% · MemTotal, 4 GiB) — which is still strictly better than a constant, but leaves stock hard eviction unreachable.

For prior art, systemd's zram-generator defaults to min(RAM/2, 4096 MiB) and Fedora has sized zram proportionally since F33/F34. As far as I can tell, no other Kubernetes node OS (AL2023, Flatcar, Talos, COS) enables zram by default at all.

Option B — expose it as a setting

settings.swap.zram-size, accepting an absolute size or a percentage, with "auto" meaning the Option A derivation. More flexible, and now a smaller delta since Option A already introduces the template and the ordering change — but it needs a model type in sources/models and boot-only restart semantics. I'd suggest Option A as the default behaviour regardless, with B layered on later if there is demand.

One caveat to be explicit about

zramctl --size is uncompressed capacity. At the 3–4× ratios we measured, a device of size Z costs only Z/r physically — but it hides up to the full Z from workingSet. All the math above is driven by the uncompressed figure, which is the conservative and correct one for eviction reasoning; the physical cost of the proposed sizes is 3–4× smaller again.

Related

• Unresponsive/unreachable Bottlerocket EKS nodes #4075 — the de-facto thread for these symptoms; multiple reporters plus a
  maintainer confirm kswapd saturating CPU
• variants: add k8s-1.35 variants #4735 / core-kit#590, core-kit#792 — where zram was introduced and defaulted
• Configuration option to set up swap #4808 — requests swap files; does not cover zram sizing
• kubernetes#43916 — the page-cache half of the accounting blind spot