Repository navigation
Conversation
The ZTS test fault/zed_retire_leak runs the ZED under valgrind and is skipped when valgrind is not installed. Install it on the Linux test VMs so that the test runs in CI. Signed-off-by: Will Rouesnel <wrouesnel@wrouesnel.com>
Add a test that runs the ZED under valgrind, faults vdevs, waits for the retire agent to handle them, stops the ZED, and fails if valgrind reports a definitely lost block allocated by the retire agent. It first faults a vdev of a mirror with "zpool offline -f". Then it creates a dRAID with two failure groups, faults one vdev with "zpool offline -f", and has the ZED diagnose an I/O fault for the vdev at the same index in the other group, so the retire agent handles a failure domain failure, both for the fault and for the state change. The retire agent currently leaks the two pool handles that is_draid_fdomain_failure() opens for every faulted vdev, for any pool type, and on a failure domain failure it returns without closing the pool handle and freeing the vdev name, so this test fails. Only leak records with a zfs_retire.c function in the stack count, so unrelated ZED leaks do not fail the test. If the ZED has no symbols, pool handles leaked from a zpool_iter() call made by the ZED count instead. The test is skipped when valgrind is not installed. Add valgrind to the ZTS command list so the test can find it. Signed-off-by: Will Rouesnel <wrouesnel@wrouesnel.com>
is_draid_fdomain_failure() looks up the pool twice per try, up to four tries, via find_by_guid_impl(), but never closes the returned zpool_handle_t. Since it is called for every removed/faulted vdev, regardless of the pool type (the "not dRAID" check happens after the handles are opened), zed leaks two pool handles, including their config nvlists, on every such event, and up to eight when a dRAID with failure domains is not (yet) in a domain failure state. The nfaults_map buffer was also leaked on the early returns. Rework it to open the pool only once per try, look up the top-level vdev in the same config snapshot, and close the handle at the end of each try. Also guard against a zero top-level guid, which previously left nvtop uninitialized. zfs_retire_recv() had similar leaks: - In the resource.fs.zfs.removed/statechange path, the early returns for an offline vdev, a dRAID domain failure and the loopback removal event leaked both zhp and devname. Route them through the common cleanup. - Commit c21c4ff added a "return" inside the fault suspect loop when a dRAID domain failure is detected, which leaked zhp. The intent is only to not kick in a hot spare for this vdev, so close the handle and continue with the next suspect instead. Finally, remove_spares() leaked the vdev name returned by zpool_vdev_name(). This makes the new fault/zed_retire_leak test pass. Signed-off-by: Will Rouesnel <wrouesnel@wrouesnel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
The ZED's retire agent leaks memory for every faulted vdev. With the ZED under valgrind, faulting one vdev of a mirror with
zpool offline -fleaves two pool handles definitely lost (352 bytes each, plus the pool config they hold, about 107 KB each, counted as indirectly lost). Both come fromis_draid_fdomain_failure(), which looks the pool up twice throughfind_by_guid_impl()and never closes the handles. It does this for every faulted vdev, before it checks whether the vdev is in a dRAID at all.On a dRAID with failure groups there is more:
zfs_retire_recv()returns without closing its own pool handle: in the state change path it also leaks the vdev name, and in the fault (suspect list) path thereturnthat c21c4ff added leaks the handle.valgrind shows each of these on master in the test below.
Found during a bug hunt I'm running with Claude Opus 5.5; the analysis, fix, test and testing were AI-generated, and reviewed and tested in VMs. Any "Extra Notes" section is a human-written addendum.
Description
This PR has three commits. The first one is not a test-only commit, so zfs-precheck takes the second as the failing test and the third as its fix:
CI: install valgrind in the qemu test VMs: the new test runs the ZED under valgrind, and is skipped when valgrind is not installed. This adds valgrind to the package lists inqemu-3-deps-vm.sh(Alpine, Arch, Debian/Ubuntu, RHEL/Fedora), so the test runs in the Linux test VMs. It comes first so that every commit after it is tested with valgrind.ZTS: add fault/zed_retire_leak(Linux only): starts the ZED under valgrind, faults a vdev of a mirror withzpool offline -f, then creates adraid1:2d:4c:8wpool (two failure groups), faults its first vdev withzpool offline -fand makes the ZED diagnose an I/O fault for the vdev at the same index in the other group (zinject -e io -T read), so the retire agent sees a failure domain failure for both the fault and the state change. It then stops the ZED and fails if valgrind reports a definitely lost block allocated by the retire agent (azfs_retire.cfunction in the stack). Leaks elsewhere in the ZED don't count. When the ZED has no symbols (as in the packages the CI installs), it counts pool handles leaked from azpool_iter()call made by the ZED, which is how the retire agent looks up pools.valgrindis added to the ZTS command list.zed: fix zpool handle leaks in zfs_retire agent:is_draid_fdomain_failure()opens the pool once per try, finds the top-level vdev in the same config, and closes the handle at the end of each try.zfs_retire_recv()go through its common cleanup, which closes the handle and frees the vdev name.Reading the code, the fix also frees the vdev name in
remove_spares(), freesnfaults_mapon the early exits ofis_draid_fdomain_failure(), and checks for a zero top-level vdev guid there. The test does not reach these paths.How Has This Been Tested?
Ubuntu 24.04 VM, Linux 6.8, debug build, valgrind 3.22, on master 4542a31:
fault/zed_retire_leakfailed 3 times in 3 runs, with 19 to 21 blocks definitely lost (5,841 to 6,609 bytes). In the full valgrind report of one such run, the leak records withzfs_retire.cframes were the pool handles fromis_draid_fdomain_failure()(called from both the state change and the suspect list paths), the pool handles opened in both paths ofzfs_retire_recv(), and the vdev name fromzpool_vdev_name()in the state change path.zfs_diagnosisstill has open when the ZED exits, not in the retire agent.zedand the libraries stripped asdh_stripdoes for the Debian packages, the test failed without the fix, where it found the leaks through thezpool_iter()fallback, and passed with it (one run each).With this change, in the same VM: the
eventsgroup passed (16 passed), and thefaultgroup had 20 passed and 3 failed:auto_replace_001_pos,auto_replace_002_posandauto_spare_multiple, which fail the same way without this change in this VM (master plus the test commit).Types of Changes
Checklist
Signed-off-by.