CloudStack KVM + LINSTOR primary storage: VM stuck in Starting although LINSTOR volume is successfully created

[anirudh09041]

CloudStack 4.22.1 + KVM + LINSTOR: VM stuck in Starting

Hi,

I'm troubleshooting a VM deployment issue with CloudStack 4.22.1.0 using LINSTOR as primary storage.

Environment

• CloudStack: 4.22.1.0
• Hypervisor: KVM / RHEL 9.x
• LINSTOR: 1.33.1
• DRBD: 9.3.3 kernel / 9.34.0 userspace
• Zone: zone-sv2
• KVM host: sj-sv2-devop-80 (10.96.32.31)
• LINSTOR controllers: 10.96.32.193/195/197

CloudStack configuration

Primary storage:

Name: linstor-primary
Type: Linstor
Scope: ZONE
Storage tag: linstor
Path: cs-rg
Hypervisor: KVM
Status: Up


Compute offering:

Name: acos-dev-compute-offering
CPU: 16
Memory: 24576 MB
Provisioning: thin
Storage type: shared
Storage tag: linstor
Root disk size: 0 GB

Template:

base_image-0.0.7-dd_disabled_2
KVM / x86_64
QCOW2
Size: 301.20 GiB
Ready: true
Problem

VM test-vm-ak (i-2-33-VM) is created but remains stuck in Starting.

The ROOT volume is successfully created:

ROOT-33
Size: 301.20 GiB
Status: Allocated

LINSTOR also successfully creates the resource:

cs-edf8f4ea-6463-42b1-979f-4c4b49402dc0

with two replicas:

sj-sv2-devop-80   UpToDate
sj-sv2-devop-82   UpToDate

DRBD is healthy:

sj-sv2-devop-80:
  role: Secondary
  disk: UpToDate
  peer: sj-sv2-devop-82
  peer-disk: UpToDate

sj-sv2-devop-82:
  role: Secondary
  disk: UpToDate
  peer: sj-sv2-devop-80
  peer-disk: UpToDate

The DRBD device exists on the KVM host:

/dev/drbd1000
301.2G

The LINSTOR-backed libvirt storage pool is also active and reports:

Capacity:   1.75 TiB
Available:  1.70 TiB
Important observation

Despite the successful LINSTOR/DRBD provisioning, the libvirt domain is never created:

virsh list --all --name

does not show:

i-2-33-VM

and:

virsh domblklist i-2-33-VM

returns:

error: failed to get domain 'i-2-33-VM'

Interestingly, the same VM creation works successfully when NFS is used as primary storage.

I also noticed this libvirt message:

no connection driver available for lxc:///
End of file while reading data: Input/output error

although I'm not sure if it is related.

My question

Since CloudStack successfully creates the LINSTOR resource and /dev/drbd1000, but never creates the libvirt domain:

What should the expected CloudStack → LINSTOR → DRBD → libvirt workflow be at this stage, and where should I look next?

In particular:

Should CloudStack/LINSTOR promote the DRBD resource to Primary before VM creation?
How should CloudStack attach /dev/drbd1000 to libvirt/QEMU?
Is the LINSTOR libvirt storage pool required, and what should its configuration look like?
Are there any known CloudStack 4.22.1 LINSTOR/KVM issues?
Which CloudStack agent/libvirt logs would be most useful for debugging this stage?

Thanks!

[nickaccuweb]

@anirudh09041

Thanks for sharing the details.

The LINSTOR resource and DRBD device are created, but the ROOT volume is still showing Allocated instead of Ready. This means CloudStack may not have completed the volume preparation.

For now, please do not manually promote the DRBD resource to Primary or make any changes to the libvirt storage pool.

CloudStack should handle the volume attachment and use the appropriate /dev/drbd/by-res/ path automatically.

Please also confirm that the KVM hosts are running CloudStack 4.22.1.0 and that their hostnames match the LINSTOR node names.

For one failed VM deployment, please share:

1. Deployment time, timezone, and asynchronous job ID.
2. Relevant entries from management-server.log.
3. Relevant entries from agent.log, including CopyCommand and StartCommand.
4. LINSTOR and libvirt errors from the same time.
5. /var/log/libvirt/qemu/i-2-33-VM.log, if available.
6. Confirm that the /dev/drbd/by-res/ path exists and works correctly.

These details will help us identify whether the issue is happening during template preparation, disk access, or VM startup.

Regards,
Nick R
AccuWeb.Cloud