Skip to content

Snapshots copy from Linstor to NFS fail #14011

Open
Bug
#14066
Open
Snapshots copy from Linstor to NFS fail#14011
Bug
#14066
Assignees
rp-
Labels
component:kvmtype:bug
Milestone
4.22.2

Description

@scornet256
scornet256
opened on Aug 31, 2026
ISSUE TYPE
  • Bug Report
COMPONENT NAME
Storage - Linstor snapshot
CLOUDSTACK VERSION
4.22.1.0
CONFIGURATION

N/A

OS / ENVIRONMENT
  • Cloudstack 4.22.1.0
  • Hypervisor KVM
  • AlmaLinux 9.8
  • Primary storage: Linstor (LVM-thin)
  • Secondary storage: NFS
SUMMARY

createSnapshot on any volume backed by a Linstor primary storage pool fails 100% of the time with: Failed to create Snapshot due to an internal error creating Snapshot for volume <volume-uuid>.

The failure is deterministic and reproducible on every attempt, regardless of volume, VM, or snapshot name.

First the snapshot is created succesfully on the Linstor backend, but then the copy to NFS (secundairy storage) fails as CloudStack's KVM agent invokes qemu-img convert against a /dev/mapper/... device path that does not match the path the kernel actually created.

Actual device location: /dev/mapper/linstor_pool-lv--thin-cs--12fc4055--3985--4025--8eb5--d6fd53effe37_00000
Path CloudStack looks for: /dev/mapper/linstor_pool-lvm-thin-cs--12fc4055--3985--4025--8eb5--d6fd53effe37_00000_cs--d7aea646--5f40--46a2--b9dc--77e41ea29336

Notice the missing dashes in the linstor_pool-lv--thin-cs-name.

STEPS TO REPRODUCE

Create disk snapshot on a Linstor volume with NFS as secundairy storage and observe the error message.

EXPECTED RESULTS

Snapshot is created on the Linstor backend and successfully copied to secondary (NFS) storage.

ACTUAL RESULTS

The error logged in management-server.log:

ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] Failed to create Snapshot due to an internal error creating Snapshot for volume 12fc4055-3985-4025-8eb5-d6fd53effe37 java.lang.RuntimeException: Unexpected exception
	at com.cloud.storage.VolumeApiServiceImpl.takeSnapshotInternal(VolumeApiServiceImpl.java:3999)
	at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:3910)
	...
Caused by: com.cloud.utils.exception.CloudRuntimeException: Exception: org.apache.cloudstack.utils.qemu.QemuImgException
Message: qemu-img: Could not open '/dev/mapper/linstor_pool-lvm-thin-cs--12fc4055--3985--4025--8eb5--d6fd53effe37_00000_cs--d7aea646--5f40--46a2--b9dc--77e41ea29336': Could not open '/dev/mapper/linstor_pool-lvm-thin-cs--12fc4055--3985--4025--8eb5--d6fd53effe37_00000_cs--d7aea646--5f40--46a2--b9dc--77e41ea29336': No such file or directory
Stack: org.apache.cloudstack.utils.qemu.QemuImgException: ...
	at org.apache.cloudstack.utils.qemu.QemuImg.convert(QemuImg.java:495)
	...
	at com.cloud.hypervisor.kvm.resource.wrapper.LinstorBackupSnapshotCommandWrapper.convertImageToQCow2(LinstorBackupSnapshotCommandWrapper.java:99)
	at com.cloud.hypervisor.kvm.resource.wrapper.LinstorBackupSnapshotCommandWrapper.execute(LinstorBackupSnapshotCommandWrapper.java:156)
	...

	at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:475)
	at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:204)
	at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1787)
	at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1660)
	...

Suggested fix: apply LVM's standard dash-doubling escape to the volume-group name the same way it's already correctly applied to the cs--<uuid> logical-volume/snapshot name portion.

Activity

  1. scornet256
    added
    bug
    on Aug 31, 2026

  2. boring-cyborg commented on Aug 31, 2026

    @boring-cyborg
    boring-cyborgbot
    on Aug 31, 2026 – with boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template!

  3. DaanHoogland
    assigned
    rp-
    on Aug 31, 2026
  4. DaanHoogland
    removed
    bug
    on Sep 2, 2026
  5. DaanHoogland
    added the
    Bug
    issue type on Sep 2, 2026
  6. DaanHoogland
    added this to the 4.22.2 milestone on Sep 2, 2026
  7. DaanHoogland
    changed the title [-][BUG][4.22.1.0] Snapshots copy from Linstor to NFS fail[/-] [+]Snapshots copy from Linstor to NFS fail[/+] on Sep 2, 2026

  8. DaanHoogland commented on Sep 2, 2026

    @DaanHoogland
    DaanHoogland
    on Sep 2, 2026
    Contributor

    @rp- could you have a look?

  9. github-actions
    added
    component:kvm
    type:bug
    on Sep 2, 2026

  10. github-actions commented on Sep 2, 2026

    @github-actions
    github-actionsbot
    on Sep 2, 2026 – with GitHub Actions

    🎯 Triage report

    Deterministic failure when backing up a snapshot from Linstor primary storage to NFS secondary storage on KVM. The KVM agent constructs a /dev/mapper/... device-mapper path for the LVM-thin volume that does not match the actual path the kernel creates, because the volume-group name's dashes are not escaped using LVM's standard dash-doubling convention (only the cs--<uuid> portion is escaped). The reporter included full stack traces and a concrete suggested fix.

    📊 Assessment

    Dimension Value Reasoning
    Type type:bug Deterministic, reproducible failure with clear stack trace and root cause identified
    Component component:kvm Failure occurs in LinstorBackupSnapshotCommandWrapper (KVM agent) during qemu-img convert
    Severity Severity:Major Snapshot backup to secondary storage fails 100% of the time for Linstor+NFS setups, but is scoped to a specific storage combination
    Labels type:bug, component:kvm
    Coding agent Suitable Root cause and exact escaping bug are clearly identified with file/line references and a concrete suggested fix (apply LVM dash-doubling to the volume-group name)

    🔗 Similar issues

    No similar open issues found; this appears to be a distinct, Linstor-specific path-escaping bug.

    💡 Notes and suggestions
    • Root cause: LVM device-mapper names double every literal dash in both the VG name and LV name (e.g. vg-name → vg--name). The current code appears to correctly escape the cs--<uuid> LV/snapshot suffix but not the VG name segment (linstor_pool-lvm-thin) which should become linstor_pool-lvm--thin.
    • Suggested fix location: LinstorBackupSnapshotCommandWrapper.convertImageToQCow2 (and any shared path-building helper used to construct the /dev/mapper/... path for Linstor-backed volumes) — apply the same dash-doubling escape uniformly to the whole device-mapper name, not just the LV suffix.
    • A quick regression test could construct a Linstor VG name containing dashes and assert the resulting /dev/mapper/... path matches dmsetup/kernel-created naming.

    Generated by Daily Issue Triage · sonnet50 135.6K · ◷

    Add this agentic workflows to your repo

    To install this agentic workflow, run

    gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9
  11. rp-
    linked a pull request that will close this issuelinstor: escape dashes in the LVM volume group name of snapshot paths #14066on Sep 7, 2026

  12. rp- commented on Sep 7, 2026

    @rp-
    rp-
    on Sep 7, 2026
    Contributor

    Yes, valid report and I opened a PR to fix this.

    A workaround for now is to use VG names that don't use any -

  13. rp-
    added a commit that references this issue on Sep 29, 2026

    linstor: escape dashes in the LVM volume group name of snapshot paths (…

    31b6855
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

component:kvmtype:bug

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions