Skip to content

testkit: cgroup_squeeze records a kernel limit refusal as fault_apply_failed #64

Description

@postgresql007

Surfaced by the 10 h release soak for v1.5.0 (run-v1.5.0-10h, seed 20260920).

What happens

One cgroup_squeeze application in 152 failed:

cgroup_squeeze(target=pg_random, max_bytes=33554432)
docker update --memory=33554432 --memory-swap=33554432 <container>
  → Error response from daemon: runc did not terminate successfully:
    failed to write "33554432": write /sys/fs/cgroup/.../memory.max

--memory-swap equals --memory, so swap is disabled for the container. To
apply a 32 MiB cap the kernel must reclaim the cgroup down to 32 MiB with
nowhere to page anon memory out to, and PostgreSQL's shared_buffers alone
defaults to 128 MB. When the unreclaimable footprint happens to exceed the
target at that instant, the write is refused.

It is therefore timing-dependent, not deterministic:

outcome count
fault_apply 152
fault_recovered 146
fault_apply_failed 1 (0.7%)

Why it is worth fixing

Nothing broke. ubuntu-2204-pg15-arm carried on for 157 more events, 16
backups started and 16 completed, 14 further faults applied and recovered. No
backup failure, no verify failure, no cell abort.

The cost is triage. A soak report showing fault_apply_failed: 1 makes someone
stop and investigate, and the answer is "the kernel declined an impossible
limit" — which is the injector behaving correctly, not a product fault. That is
the same class the ErrTargetNotRunning work addressed: a benign, well-understood
refusal recorded as a generic apply failure, so it costs attention every time a
run is read.

Suggested fix

internal/testkit/inject already has the precedent — ErrCapSysResource types
the missing-CAP_SYS_RESOURCE case so the orchestrator can classify it rather
than count it as a failure. The same treatment fits here:

  • detect the cgroup write refusal in DockerTarget.SetMemoryLimit
    (target_docker.go) and return a typed sentinel, e.g. ErrLimitUnreachable
  • have the orchestrator record it as a skip/refusal rather than
    fault_apply_failed, as it already does for ErrTargetNotRunning
  • optionally, have cgroupSqueezeFault pick a floor from the container's
    current usage instead of a fixed max_bytes, so the squeeze is always
    applicable

Not fixed in #63 deliberately

This was found during the release soak that gates #63. Changing the fault
injector mid-run would have put code on that branch which the 10 h run never
exercised, and a gate is only worth what it actually tested.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions