Surfaced by the 10 h release soak for v1.5.0 (run-v1.5.0-10h, seed 20260920).
What happens
One cgroup_squeeze application in 152 failed:
cgroup_squeeze(target=pg_random, max_bytes=33554432)
docker update --memory=33554432 --memory-swap=33554432 <container>
→ Error response from daemon: runc did not terminate successfully:
failed to write "33554432": write /sys/fs/cgroup/.../memory.max
--memory-swap equals --memory, so swap is disabled for the container. To
apply a 32 MiB cap the kernel must reclaim the cgroup down to 32 MiB with
nowhere to page anon memory out to, and PostgreSQL's shared_buffers alone
defaults to 128 MB. When the unreclaimable footprint happens to exceed the
target at that instant, the write is refused.
It is therefore timing-dependent, not deterministic:
| outcome |
count |
fault_apply |
152 |
fault_recovered |
146 |
fault_apply_failed |
1 (0.7%) |
Why it is worth fixing
Nothing broke. ubuntu-2204-pg15-arm carried on for 157 more events, 16
backups started and 16 completed, 14 further faults applied and recovered. No
backup failure, no verify failure, no cell abort.
The cost is triage. A soak report showing fault_apply_failed: 1 makes someone
stop and investigate, and the answer is "the kernel declined an impossible
limit" — which is the injector behaving correctly, not a product fault. That is
the same class the ErrTargetNotRunning work addressed: a benign, well-understood
refusal recorded as a generic apply failure, so it costs attention every time a
run is read.
Suggested fix
internal/testkit/inject already has the precedent — ErrCapSysResource types
the missing-CAP_SYS_RESOURCE case so the orchestrator can classify it rather
than count it as a failure. The same treatment fits here:
- detect the cgroup write refusal in
DockerTarget.SetMemoryLimit
(target_docker.go) and return a typed sentinel, e.g. ErrLimitUnreachable
- have the orchestrator record it as a skip/refusal rather than
fault_apply_failed, as it already does for ErrTargetNotRunning
- optionally, have
cgroupSqueezeFault pick a floor from the container's
current usage instead of a fixed max_bytes, so the squeeze is always
applicable
Not fixed in #63 deliberately
This was found during the release soak that gates #63. Changing the fault
injector mid-run would have put code on that branch which the 10 h run never
exercised, and a gate is only worth what it actually tested.
Surfaced by the 10 h release soak for v1.5.0 (
run-v1.5.0-10h, seed 20260920).What happens
One
cgroup_squeezeapplication in 152 failed:--memory-swapequals--memory, so swap is disabled for the container. Toapply a 32 MiB cap the kernel must reclaim the cgroup down to 32 MiB with
nowhere to page anon memory out to, and PostgreSQL's
shared_buffersalonedefaults to 128 MB. When the unreclaimable footprint happens to exceed the
target at that instant, the write is refused.
It is therefore timing-dependent, not deterministic:
fault_applyfault_recoveredfault_apply_failedWhy it is worth fixing
Nothing broke.
ubuntu-2204-pg15-armcarried on for 157 more events, 16backups started and 16 completed, 14 further faults applied and recovered. No
backup failure, no verify failure, no cell abort.
The cost is triage. A soak report showing
fault_apply_failed: 1makes someonestop and investigate, and the answer is "the kernel declined an impossible
limit" — which is the injector behaving correctly, not a product fault. That is
the same class the
ErrTargetNotRunningwork addressed: a benign, well-understoodrefusal recorded as a generic apply failure, so it costs attention every time a
run is read.
Suggested fix
internal/testkit/injectalready has the precedent —ErrCapSysResourcetypesthe missing-
CAP_SYS_RESOURCEcase so the orchestrator can classify it ratherthan count it as a failure. The same treatment fits here:
DockerTarget.SetMemoryLimit(
target_docker.go) and return a typed sentinel, e.g.ErrLimitUnreachablefault_apply_failed, as it already does forErrTargetNotRunningcgroupSqueezeFaultpick a floor from the container'scurrent usage instead of a fixed
max_bytes, so the squeeze is alwaysapplicable
Not fixed in #63 deliberately
This was found during the release soak that gates #63. Changing the fault
injector mid-run would have put code on that branch which the 10 h run never
exercised, and a gate is only worth what it actually tested.