RAID-Z expansion: a crash right after completion leaves the added capacity unavailable #19420

System information

Everything below was run in userspace, with libzpool and zdb, not with the kernel module.

Describe the problem you're observing

When a RAID-Z expansion completes, spa_raidz_expand_thread() (module/zfs/vdev_raidz.c) does two things in order:

1. The raidz_reflow_complete_sync() sync task, in completion txg T, marks the expansion finished and adds the raidz_expand_txgs entry. vdev_asize keeps its pre-expansion value.
2. vdev_online(spa, <raidz vdev guid>, ZFS_ONLINE_EXPAND, NULL). Its reopen is the only place where vdev_asize grows to the new width, since vdev_open() only adopts a larger asize when vdev_expanding or autoexpand is set. The txg it syncs writes the new asize, and the config update it requests adds the metaslabs through vdev_expand().

If the system stops after txg T has synced and before step 2 has synced, the pool comes back with the expansion finished but with the old asize and metaslab count, and nothing adds the capacity later:

• At load, the asize comes from ZPOOL_CONFIG_ASIZE, the old value, and vdev_expand() finds no metaslab to add. This holds for an import after a device scan and for an open from the cachefile with spa: Take the RAID-Z layout from labels newer than the config #19405 applied (without it, the cachefile open fails, RAID-Z expansion: a crash just after completion leaves a cachefile that cannot open the pool #19404).
• Nothing at load acts on a finished expansion, so step 2 is never retried.
• autoexpand=on followed by export and import does not bring it back.
• zpool online -e <pool> <leaf> does, and so does autoexpand=on followed by a plain zpool online of a leaf. A user has little reason to try either, since the expansion is reported as completed.

An export during the window does not lose the capacity (zthr_cancel() waits for the thread, and the export's last sync writes the new asize), and neither does a crash after step 2 has synced (the next import's vdev_expand() adds the metaslabs). The window is one reopen and one txg long, and no data is lost.

Possible directions, which I would rather leave to someone who knows this code:

• Grow vdev_asize to the full width in raidz_reflow_complete_sync(), so that txg T already carries it; vdev_expand() then adds the metaslabs at the next config update or import, which already works for a pool with a grown asize and the old metaslab count. This changes vdev_asize in syncing context, outside vdev_open() and its HEALTHY check.
• Redo the ZFS_ONLINE_EXPAND step at load for a raidz vdev whose expansion is finished. Without an on-disk marker this cannot tell the missing capacity from disks that grew under autoexpand=off; with one, it adds on-disk state.

I can send a pull request for whichever is preferred.

Describe how to reproduce the problem

I found no deterministic way to hit the window with the kernel module: it takes a crash between txg T and the next one. In userspace it can be stopped exactly. A libzpool program defines its own vdev_online() and is linked -rdynamic. libzpool reaches vdev_online through its PLT, so the expansion thread's call lands in the program, which calls _exit(0); every other call is forwarded with dlsym(RTLD_NEXT, "vdev_online").

Pool: raidz1 of four 128 MiB files, 24 MiB written, a fifth file attached, zfs_scrub_after_expand=0. After the _exit(0), the pool is found by a device scan (zpool_find_config(), as zdb -e does) and imported read-write with spa_import(). In these terms zpool online -e is vdev_online(spa, <leaf guid>, ZFS_ONLINE_EXPAND, &state).

517996544 is 4 x 129499136, the pre-expansion size, and 647495680 is 5 x 129499136. The stopped run was repeated three times with the same numbers. Without interposing anything, a program that polls the cachefile and calls _exit(0) as soon as txg T has synced (raidz1 3 -> 4) also left the 3-wide asize.

Include any warning/errors/backtraces from the system logs

Nothing is logged. Stopped before step 2, after the expansion completed 4 -> 5:

$ zdb -e -p <dir> -C ecx | grep -A30 "type: 'raidz'"
                nparity: 1
                raidz_expand_txgs[0]: 45
                metaslab_shift: 24
                asize: 517996544
$ zdb -e -p <dir> -m ecx | grep metaslabs
	metaslabs    30   offset                spacemap          free

Uninterrupted:

                raidz_expand_txgs[0]: 45
                metaslab_shift: 24
                asize: 647495680
	metaslabs    38   offset                spacemap          free