zfs destroy <filesystem> fails with dataset busy

[x3x7]

I am trying to destroy a ZFS filesystem but even if zfs umount succeeds I still get cannot destroy <filesystem>: dataset is busy

Can somebody with deeper knowledge of ZFS clarify what is going on?

[wrouesnel]

Do you have snapshots which have created clones from that filesystem? If you do you need to promote the clone (zfs promote)

[x3x7]

There are no snapshots.

To destroy a filesystem I have to reboot the machine. After importing a pool there are no problems with removing filesystems on it. After using the pool for a while the filesystems can no longer be destroyed. Unmount works but not destroy which is very strange.

I am using zfs-2.2.7-2_gff7d05147. Next week I will upgrade to 2.4.

[wrouesnel]

I tasked this to Claude and it got a pretty strong hit for what you're seeing. Short answer is: the upgrade will fix it.

For a filesystem with no snapshots, zfs destroy reports "dataset is busy" for one reason only: something still holds a long hold on the dataset. In 2.2.7 the check is dsl_destroy_head_check_impl(), which returns EBUSY when ds_longholds isn't zero. A mounted filesystem holds one for as long as its superblock is alive, and zfs umount only removes the mount from your own mount namespace.

The likely cause is a bug that every 2.2.x release has.

• zfsvfs_hold() in module/zfs/zfs_ioctl.c takes a superblock reference through getzfsvfs(). If it then finds the filesystem marked z_unmounted, it returns EBUSY without dropping that reference.
• The superblock then never goes away. zfs umount succeeds, but the objset stays owned, so destroy fails with "dataset is busy". zpool export of that pool fails too, and only a reboot or re-import clears it. That matches "fine after import, busy after using the pool for a while".
• The leak needs a quota or space call to run while a mounted filesystem is in that state: zfs userspace, zfs groupspace, or zfs get/zfs set of userquota@, userused@ and similar properties.
• A mounted filesystem gets into that state when an online rollback, or a zfs receive into it, fails to resume it.
• It was fixed by Fix s_active leak in zfsvfs_hold() when z_unmounted is true #18310 (fd06764), which is in 2.3.7 and later and 2.4.2 and later. It was never backported to 2.2.

You can check the alternative before upgrading. The same error also happens if another mount namespace still has the filesystem mounted, for example a container or a systemd service with PrivateTmp, ProtectSystem or PrivateMounts. The upgrade won't change that case. Run:

grep -l ' <pool>/<fs> ' /proc/*/mountinfo

If that finds nothing and zpool export of the pool also says busy, it's the leaked reference, and 2.4 has the fix.

Other, temporary holders that also give EBUSY in 2.2.7:

• a zfs send or zfs receive running on the dataset, including a resumable receive into it;
• a zfs wait on it;
• the background objset upgrade, which runs the first time the filesystem is opened after a feature upgrade.

This comes from reading the 2.2.7-2-gff7d05147 source. I haven't reproduced the leak.

[x3x7]

Thank you for the detailed analysis. Much appreciated!

I did check the grep after unmounting a ZFS filesystem and it was still showing in the mountinfo for all processes.

I am using Linux kernel 6.12.45 (if that has any impact). Once my upgrade is done I will update this discussion.