Repository navigation
VM console cannot access after VNC certificate expired #9718
Description
Activity
@havengit , I think this is more a lack of feature than a bug, which admittedly comes down to the same to you. So I will mark it as an improvement. Dynamically setting the certificate will not be easy. Did you try rebooting just the console proxy? I would guess that is the actually needed action.
there are some global settings ("ca.framework.*"), including
- ca.framework.background.task.delay
- ca.framework.cert.automatic.renewal
- ca.framework.cert.expiry.alert.period
if auto renewal is enabled, the agent cert should be auto-renewed.
otherwise you can do it manually . refer to #9562 (comment)
Yes , I have restarted cpvm and libvirt on host. I confirm that the certificate has been successfully updated on the host. May be same bug in qemu. After stop vm and start vm console works fine .
Yes , I have restarted cpvm and libvirt on host. I confirm that the certificate has been successfully updated on the host. May be same bug in qemu. After stop vm and start vm console works fine .
@havengit
is it necessary to stop/start vm if you have restarted libvirtd and cloudstack-agent ?
if so, it has big impact@weizhouapache Restart libvirtd and cloudstack-agent has no effect ，VM must be rebooted or live migration to other host .
I don't know if anyone else has encountered this, but this feature should have been added in 4.18, so maybe not many people are using it.Using virt-viewer also fails to connect, so it shouldn't be an ACS issue.
this may be related to #7015
can you check if
vnc_tlsis set to 1 in /etc/libvirt/qemu.conf ?- console works after live migration to another host ?
Yes , the vnc tls were enabled in all host. It seems that the qemu process that is running, does not recognize the certificate change and still uses the old certificate. Stop the startup or migrate and the new qemu process will use the new certificate.
vnc_tls=1
vnc_tls_x509_verify=1
vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc"Yes , the vnc tls were enabled in all host. It seems that the qemu process that is running, does not recognize the certificate change and still uses the old certificate. Stop the startup or migrate and the new qemu process will use the new certificate. vnc_tls=1 vnc_tls_x509_verify=1 vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc"
thanks @havengit
good to know that migration fixes the issue. stopping/starting all vms is not possible for large production environments.To summarize,
- vm console becomes unavailable if cloudstack-agent certificates expires and is renewed.
- it is a qemu issue. ACS cannot do anything to fix it.
- users have to migrate or stop/start the VMs so that new qemu process will use the new certificates
Thanks ,weizhou and community , I have change Ca framework cert validity period to a very long time . I won't run into this problem in the future.
Hi all, by default I think this should work. The ca.framework.cert.automatic.renewal needs to be enabled (true), and there's also ca.framework.cert.expiry.alert.period and ca.framework.background.task.delay. For agents that are expired certs but are connected it's not an issue, but such agents risk failing to join when restarted - for them an explicit API can be called:
(homecloud) 🐵 > provision certificate hostid= -h provisionCertificate: Issues and propagates client certificate on a connected host/agent using configured CA plugin This API is asynchronous. Required params: hostid, API Params Type Description ========== ==== =========== hostid uuid The host/agent uuid to which the certific ate has to be provisioned (issued and pr opagated) provider string Name of the CA service provider, otherwis e the default configured provider plugin will be used reconnect boolean Whether to attempt reconnection with host /agent after successful deployment of ce rtificate. When option is not provided, configured global setting is usedHowever, VNC console to users browser uses admin uploaded certificate - when they expire, admin needs to upload new end-user TLS/SSL certs. Lastly, I don't remember if we'd restart libvirtd on automatic cert renewal - the same API (or UI) button can be used to restart agent+libvirt I think - worth testing.
@DaanHoogland @weizhouapache is there any movement on the certificate management, is really a problem when the certificates expire for a critical environment have to restart or migrate the VMs - Making a certificate update on a major issue - mostly because CA authorities are not providing certificates for a period longer than a year, and it does not matter if you have the certificate to auto-renew.
rebooting the Systemproxy VM i think that is perfectly sustainable, however have to restart VMs and on routed VPCs we have had to restart the VPC itself then the vrouters and then the VMs, even when whe have HA VPC - obviously the backup vrouter took over, but have to restart the whole VPC as restarting just the vrouters did not work.
Certificate rules are changing.
@DaanHoogland I have to add something interesting, after updating the CA (not selfsigned certificate) and after rebooting the System proxy, windows machines with virtio drivers installed reconnect to the vnc console, however linux fails have to restart the linux VMs including the VRs. Just a note: we do everything using the UI only not CMK.
ok, this does not ring a bell at all. Please start a new issue or discussion/qa item?
It should be possible to do this without restarting the VM using
virsh domdisplay-reload <domain> --type vnc.
This command is available from libvirt 10.2.0
https://www.libvirt.org/manpages/virsh.html#domdisplay-reloadFor older versions,
virsh qemu-monitor-command i-2-4-VM '{"execute":"display-reload","arguments":{"type":"vnc","tls-certs":true}}'should work. Please note that"tls-certs": trueneeds to be added, otherwise it doesn't work. I tested this with libvirt 9.0.0 on oracle linux 9.A small script for doing it for all VMs on a host.
for vm in $(virsh list --name); do echo "Reloading VNC certs for: $vm" virsh qemu-monitor-command "$vm" '{"execute":"display-reload","arguments":{"type":"vnc","tls-certs":true}}'' done
Reacted by Suresh Kumar Anaparti, Wei Zhou and ABHISHEK PANDEYI have reopened this ticket and added to 4.22.2 milestone
we can apply the fix on all running vms automatically when the certificate is renewed
Reacted by Vishesh, ABHISHEK PANDEY and dahn- linked a pull request that will close this issuekvm: reload VNC TLS certificate on running VMs after cert renewal #14151
on Sep 30, 2026
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
advanced networking
OS / ENVIRONMENT
Hypervisro: KVM virtualization with ubuntu 22.04
SUMMARY
VM console cannot access after VNC certificate expired。
Default certificate valid for 1 year ，then automatically renews certificates when they expire by cloudstack。
I made sure the certificate was updated on the host，but Th running VM instances need stop and start to apply the new certificate or migraton to other host. For production environments, this is hard to do
I don't know how to get a new certificate to take effect without rebooting the vm
#7015
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS