PDA

View Full Version : [SOLVED] 20.04 Server w/ KVM + libvirt Error - No VMs can be created


TheFu
May 31st, 2023, 03:10 PM
So, I've been looking at this for about a week.

If I run virt-manager and try to create a new VM (there aren't any there now), the error is:

Error: No hypervisor options were found for this connection.
This usually means that QEMU or KVM is not installed on your machine, or the KVM kernel modulus are not loaded.


Check KVM kernel modules:

$ lsmod |grep kvm
kvm_amd 155648 0
kvm 1015808 1 kvm_amd
ccp 102400 1 kvm_amd
They are there, loaded.

Next, check the packages:

$ dpkg -l qemu* |egrep '^ii'
ii qemu 1:4.2-3ubuntu6.26 amd64 fast processor emulator, dummy package
ii qemu-block-extra:amd64 1:4.2-3ubuntu6.26 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:4.2-3ubuntu6.26 amd64 QEMU Full virtualization on x86 hardware
ii qemu-system-common 1:4.2-3ubuntu6.26 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-data 1:4.2-3ubuntu6.26 all QEMU full system emulation (data files)
ii qemu-system-gui:amd64 1:4.2-3ubuntu6.26 amd64 QEMU full system emulation binaries (user interface and audio support)
ii qemu-system-x86 1:4.2-3ubuntu6.26 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:4.2-3ubuntu6.26 amd64 QEMU utilities

So, those are all that should be needed. Checking libvirt packages:

$ dpkg -l libvirt* |egrep '^ii'
ii libvirt-clients 6.0.0-0ubuntu8.16 amd64 Programs for the libvirt library
ii libvirt-daemon 6.0.0-0ubuntu8.16 amd64 Virtualization daemon
ii libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.16 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.16 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 6.0.0-0ubuntu8.16 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 6.0.0-0ubuntu8.16 amd64 Libvirt daemon configuration files (systemd)
ii libvirt-glib-1.0-0:amd64 3.0.0-1 amd64 libvirt GLib and GObject mapping library
ii libvirt0:amd64 6.0.0-0ubuntu8.16 amd64 library for interfacing with different virtualization systems

Let's check the daemons,

$ sudo systemctl status libvirt*
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
└─override.conf
Active: active (running) since Wed 2023-05-31 08:59:40 EDT; 43min ago
TriggeredBy: ● libvirtd-ro.socket
● libvirtd.socket
● libvirtd-admin.socket
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 5238 (libvirtd)
Tasks: 19 (limit: 32768)
Memory: 77.9M
CGroup: /system.slice/libvirtd.service
├─2908 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
├─2909 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
└─5238 /usr/sbin/libvirtd

May 31 09:38:11 hadar libvirtd[5238]: internal error: Cannot find suitable emulator for x86_64
May 31 09:38:11 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-i386 for probing: qemu-system-i386: cannot create PID file>
May 31 09:38:11 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-i386: internal error: Failed to start QEMU binary /usr/bin/qemu-syste>
May 31 09:38:11 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-x86_64 for probing: qemu-system-x86_64: cannot create PID >
May 31 09:38:11 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-x86_64: internal error: Failed to start QEMU binary /usr/bin/qemu-sys>
May 31 09:38:11 hadar libvirtd[5238]: unknown procedure: 436
May 31 09:38:15 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-i386 for probing: qemu-system-i386: cannot create PID file>
May 31 09:38:15 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-i386: internal error: Failed to start QEMU binary /usr/bin/qemu-syste>
May 31 09:38:15 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-x86_64 for probing: qemu-system-x86_64: cannot create PID >
May 31 09:38:15 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-x86_64: internal error: Failed to start QEMU binary /usr/bin/qemu-sys>

If I directly run /usr/bin/qemu-system-x86_64 (no options), I'm shown a new window with a VM trying to boot. Something between qemu and libvirt is screwed?

Now, I'm desperate. Time to check the system logs ...

$ egrep -i 'err|warn' /var/log/syslog
...
May 31 09:38:15 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-i386: internal error: Failed to start QEMU binary /usr/bin/qemu-system-i386 for probing: qemu-system-i386: cannot create PID file: Failed to write pid file
May 31 09:38:15 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-x86_64 for probing: qemu-system-x86_64: cannot create PID file: Failed to write pid file
...
Time to google those errors. Check for issues with the PID file first. The only result was that /var was full. Checking:

$ dft
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg01-root01 ext4 35G 5.8G 27G 18% /
/dev/nvme0n1p3 ext4 672M 145M 479M 24% /boot
/dev/nvme0n1p2 vfat 50M 6.1M 44M 13% /boot/efi
/dev/mapper/vg01-tmp01 ext4 3.9G 88K 3.7G 1% /tmp
/dev/mapper/vg01-var01 ext4 20G 1.7G 17G 9% /var
/dev/mapper/vg01-home01 ext4 9.8G 5.0G 4.3G 54% /home
/dev/mapper/vg01-libvirt--01 ext4 134G 131G 0 100% /var/lib/libvirt
/dev/mapper/vg--hadar-lv--backups ext4 459G 23G 436G 5% /Backups
Plenty of room in /var/.

Next error ...

Try stopping AppArmor and then restart libvirtd and relaunch virt-manager
sudo systemctl stop apparmor
Did that. Got a little farther, but still getting error.

Switched to attempting to use virsh to add an exist VM:

$ virsh define /tmp/Win7Ult.xml
error: Failed to define domain from /tmp/Win7Ult.xml
error: internal error: Failed to start QEMU binary /usr/bin/kvm for probing: qemu-system-x86_64: cannot create PID file: Failed to write pid file

Fresh 20.04 server install, not an upgrade.

Some system basic information:

$ inxi -bz
System:
Kernel: 5.15.0-72-generic x86_64 bits: 64 Desktop: FVWM2 2.6.8
Distro: Ubuntu 20.04.6 LTS (Focal Fossa)
Machine:
Type: Desktop Mobo: ASUSTeK model: ROG STRIX B450-F GAMING v: Rev 1.xx
serial: <filter> UEFI: American Megatrends v: 5003 date: 02/03/2023
CPU:
6-Core: AMD Ryzen 5 5600G with Radeon Graphics type: MT MCP
speed: 4198 MHz min/max: 1400/4200 MHz
Graphics:
Device-1: AMD driver: amdgpu v: kernel
Display: x11 server: X.Org 1.20.13 driver: amdgpu,ati
unloaded: fbdev,modesetting,vesa resolution: 1920x1200~60Hz
OpenGL: renderer: AMD RENOIR (DRM 3.42.0 5.15.0-72-generic LLVM 12.0.0)
v: 4.6 Mesa 21.2.6
Network:
Device-1: Intel I211 Gigabit Network driver: igb
Info:
Processes: 359 Uptime: 1h 13m Memory: 30.72 GiB used: 2.25 GiB (7.3%) Shell: bash
inxi: 3.0.38
This is a Server install with a minimal WM-only GUI. No DE. It has been stable the last 2 weeks now. The hardware was running 18.l04 with KVM for the last 3+ yrs, upgraded to a newer Ryzen a few months ago and has been very stable. I rebooted it today to manually check the BIOS SVR was set. Why can't they just call it Virtualization?

BIOS has SVR enabled.

$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used


I've reinstalled all the KVM/QEMU/Libvirt packages ... but I didn't purge them first. Also, this system doesn't have IPv6 enabled. It is disabled by grub.

I'm not stuck, but would appreciate some help. Think I've checked all the bonehead stuff, but I can do some really bonehead things. Discovered some permission problems in /var/lib/libvirt/. Fortunately, I have another 20.04 system running lots of VMs that has the correct permissions, which I've setup. libvirt-qemu:kvm

Ideas?
#&thj^%
May 31st, 2023, 06:19 PM
My God it's nice to help someone that puts great information to start from. :)

First thing:


sudo usermod -aG kvm $USER
sudo usermod -aG libvirt $USER
@TheFu I'm not second guessing you just habbit's in helping.

Beware that if the host has SELinux or AppArmor active that may also cause permission problems when using QEMU binaries in "unusual" locations.
You seem to have checked that one.


All I can think of off the top of my head and *AMD is the following,
Try these steps:


sudo setfacl -m user:$USER:rw /var/run/libvirt/libvirt-sock

log out of the session and again login then run,


sudo systemctl enable libvirtd
sudo systemctl start libvirtd
EDIT: Forgot one:

groups
sys nordvpn rfkill wheel me

My Arch may look different than yours.
SeijiSensei
May 31st, 2023, 08:16 PM
$ dft
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg01-root01 ext4 35G 5.8G 27G 18% /
/dev/nvme0n1p3 ext4 672M 145M 479M 24% /boot
/dev/nvme0n1p2 vfat 50M 6.1M 44M 13% /boot/efi
/dev/mapper/vg01-tmp01 ext4 3.9G 88K 3.7G 1% /tmp
/dev/mapper/vg01-var01 ext4 20G 1.7G 17G 9% /var
/dev/mapper/vg01-home01 ext4 9.8G 5.0G 4.3G 54% /home
/dev/mapper/vg01-libvirt--01 ext4 134G 131G 0 100% /var/lib/libvirt
/dev/mapper/vg--hadar-lv--backups ext4 459G 23G 436G 5% /Backups
Plenty of room in /var/.

But not in /var/lib/libvirt which is what matters for KVM.
#&thj^%
May 31st, 2023, 08:23 PM
+1 Good catch SeijiSensei
MAFoElffen
May 31st, 2023, 09:23 PM
Let's check the daemons,

$ sudo systemctl status libvirt*
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
└─override.conf
Active: active (running) since Wed 2023-05-31 08:59:40 EDT; 43min ago
TriggeredBy: ● libvirtd-ro.socket
● libvirtd.socket
● libvirtd-admin.socket
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 5238 (libvirtd)
Tasks: 19 (limit: 32768)
Memory: 77.9M
CGroup: /system.slice/libvirtd.service
├─2908 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
├─2909 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
└─5238 /usr/sbin/libvirtd

May 31 09:38:11 hadar libvirtd[5238]: internal error: Cannot find suitable emulator for x86_64
May 31 09:38:11 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-i386 for probing: qemu-system-i386: cannot create PID file>
May 31 09:38:11 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-i386: internal error: Failed to start QEMU binary /usr/bin/qemu-syste>
May 31 09:38:11 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-x86_64 for probing: qemu-system-x86_64: cannot create PID >
May 31 09:38:11 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-x86_64: internal error: Failed to start QEMU binary /usr/bin/qemu-sys>
May 31 09:38:11 hadar libvirtd[5238]: unknown procedure: 436
May 31 09:38:15 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-i386 for probing: qemu-system-i386: cannot create PID file>
May 31 09:38:15 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-i386: internal error: Failed to start QEMU binary /usr/bin/qemu-syste>
May 31 09:38:15 hadar libvirtd[5238]: internal error: Failed to start QEMU binary /usr/bin/qemu-system-x86_64 for probing: qemu-system-x86_64: cannot create PID >
May 31 09:38:15 hadar libvirtd[5238]: Failed to probe capabilities for /usr/bin/qemu-system-x86_64: internal error: Failed to start QEMU binary /usr/bin/qemu-sys>

If what they suggested (on no space left / 100% used) doesn't work, the next thing I would look at is indicated above in red...

Compare that with output from mine:


mafoelffen@Mikes-B460M:~$ sudo systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-05-31 12:35:51 PDT; 32min ago
TriggeredBy: ● libvirtd-ro.socket
● libvirtd-admin.socket
● libvirtd.socket
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 5341 (libvirtd)
Tasks: 24 (limit: 32768)
Memory: 30.0M
CGroup: /system.slice/libvirtd.service
├─5341 /usr/sbin/libvirtd
├─5511 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/Isolated-196-168-1-0.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
├─5546 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/Isolated-172-16-10-0.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
├─5547 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/Isolated-172-16-10-0.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
├─5597 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
└─5598 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

May 31 12:35:51 Mikes-B460M dnsmasq-dhcp[5546]: read /var/lib/libvirt/dnsmasq/Isolated-172-16-10-0.hostsfile
May 31 12:35:51 Mikes-B460M dnsmasq[5597]: started, version 2.86 cachesize 150
May 31 12:35:51 Mikes-B460M dnsmasq[5597]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC loop-detect inotify dumpfile
May 31 12:35:51 Mikes-B460M dnsmasq-dhcp[5597]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h
May 31 12:35:51 Mikes-B460M dnsmasq-dhcp[5597]: DHCP, sockets bound exclusively to interface virbr0
May 31 12:35:51 Mikes-B460M dnsmasq[5597]: reading /etc/resolv.conf
May 31 12:35:51 Mikes-B460M dnsmasq[5597]: using nameserver 127.0.0.53#53
May 31 12:35:51 Mikes-B460M dnsmasq[5597]: read /etc/hosts - 7 addresses
May 31 12:35:51 Mikes-B460M dnsmasq[5597]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
May 31 12:35:51 Mikes-B460M dnsmasq-dhcp[5597]: read /var/lib/libvirt/dnsmasq/default.hostsfile

If I do


mafoelffen@Mikes-B460M:~$ ls /lib/systemd/system/libvirtd.service*
/lib/systemd/system/libvirtd.service

It seems from your output, I think you will find a directory of /lib/systemd/system/libvirtd.service.d/ with an override conf file inside of it...

When I see that there in the SystemD Service directories, then that service_name.d directory usually has an override.conf file inside that directory, either turning that service off or limiting it somehow... That is where I would look further.

Just an observation...

EDIT: I think I remember the way to change that back in SystemD is to do:


sudo systemctl revert <ServiceName.service>
sudo systemctl restart

So


sudo systemctl revert libvirtd.service
sudo systemctl restart

...should delete all overrides to that and get you back to vendor default on that service.
TheFu
May 31st, 2023, 10:22 PM
But not in /var/lib/libvirt which is what matters for KVM.

There's 3 GB free there. I use LVM for new VMs and plan to migrate a bunch back. Evening selecting the "File --> New VM" causes the error. I need to import an existing VM. The definition was from 18.04 and I've ensured the storage and bridge are the same. I can add 5G more (or 400G more) and that won't matter, I'm certain.


$ virsh list
Id Name State
--------------------

$ virsh define /tmp/Win7Ult.xml
error: Failed to define domain from /tmp/Win7Ult.xml
error: internal error: Failed to start QEMU binary /usr/bin/kvm for probing: qemu-system-x86_64: cannot create PID file: Failed to write pid file


Good idea. Let me review all the other messages. May come back to this space issue. Thanks. Heck, let me drop the 5% reserved space for the ext4 file system to 0%. Done.

Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg01-var01 ext4 20G 1.7G 17G 9% /var
/dev/mapper/vg01-libvirt--01 ext4 134G 131G 2.8G 98% /var/lib/libvirt

Holy crap batman! That was it. Non-root users couldn't add anything to the libvirt area.


$ virsh define /tmp/Win7Ult.xml
Domain Win7Ult defined from /tmp/Win7Ult.xml

The domain I restored is working. Need to migrate a few of the LVs with VMs from my other Ryzen to this one.

Seems it needed to be able to create and write these files:

# ls -l /var/lib/libvirt/qemu/domain-1-Win7Ult/
total 8
-rw------- 1 libvirt-qemu kvm 32 May 31 17:19 master-key.aes
srwxrwxr-x 1 root root 0 May 31 17:19 monitor.sock

Like I thought. Something bonehead on my side.
TheFu
May 31st, 2023, 10:27 PM
It seems from your output, I think you will find a directory of /lib/systemd/system/libvirtd.service.d/ with an override conf file inside of it...

When I see that there in the SystemD Service directories, then that service_name.d directory usually has an override.conf file inside that directory, either turning that service off or limiting it somehow... That is where I would look further.

Just an observation...

Good idea. No directory there:

# ls -F /lib/systemd/system/libvirt*
/lib/systemd/system/libvirtd-admin.socket /lib/systemd/system/libvirtd-tcp.socket
/lib/systemd/system/libvirtd-ro.socket /lib/systemd/system/libvirtd-tls.socket
/lib/systemd/system/libvirtd.service /lib/systemd/system/libvirt-guests.service
/lib/systemd/system/libvirtd.socket


Problem was that non-root users couldn't add anything to my /var/lib/libvirt .... area.
TheFu
May 31st, 2023, 10:40 PM
Replying because you took the time to post some great ideas that I easily could have discounted or missed completely. Solved. It was a 100% space issue because file systems don't allow non-root users to write anything once the reserved blocks have been hit. I just modified the default 5% reserved blocks to 0% and everything started working. I could have added more storage, but since I use LVM for nearly all my VMs backend storage, that would have been a waste.

# vgs
WARNING: PV /dev/sda1 in VG vg-hadar is using an old PV header, modify the VG to update.
VG #PV #LV #SN Attr VSize VFree
hadar-vg 1 11 0 wz--n- 476.22g 71.00g
vg-hadar 1 1 0 wz--n- <465.76g 36.00m
vg01 1 7 0 wz--n- <930.78g <690.68g
BTW, the new storage allocations will come from vg01, which has plenty of available storage.


My God it's nice to help someone that puts great information to start from. :)

I worried about providing too much information. Asking questions is like a seduction. Need to provide just enough to get others engaged and helping, but not so much that they aren't willing to try suggesting ideas.



First thing:


sudo usermod -aG kvm $USER
sudo usermod -aG libvirt $USER
@TheFu I'm not second guessing you just habbit's in helping.

Good idea. I'd checked that, but could easily have forgotten.

$ id
uid=1000(tf) gid=1000(tf) groups=1000(tf),4(adm),24(cdrom),27(sudo),29(audio ),30(dip),46(plugdev),108(kvm),117(lxd),125(libvir t),128(pulse),129(pulse-access)




Beware that if the host has SELinux or AppArmor active that may also cause permission problems when using QEMU binaries in "unusual" locations.
You seem to have checked that one.

I went down the AppArmor rabbit hole before posting. The issue remained even after apparmor was disabled. Some of the bug reports for 20.04 and qemu with similar error messages were determined to be apparmor related, but I'm not as sure today that those really were apparmor issues.



All I can think of off the top of my head and *AMD is the following,
Try these steps:


sudo setfacl -m user:$USER:rw /var/run/libvirt/libvirt-sock

log out of the session and again login then run,


sudo systemctl enable libvirtd
sudo systemctl start libvirtd
EDIT: Forgot one:

groups
sys nordvpn rfkill wheel me

My Arch may look different than yours.
There's no /var/run/libvirt/libvirt-sock file. Seems those exist below /var/run/libvirt/qemu/domain* // They appear to be per domain in 20.04.
I'd attempted to restart libvirtd a number of times and looked at the errors generated. They were the apparmor errors I initially posted.

Thanks for replying. All good ideas.