downgrading qemu on Ubuntu 18.04

[monkeybrain20122]

There is no downgrading. AFAIK the only Linux distros that let you downgrade are Arch and its derivatives (there is actually a utility called downgrade from the AUR)

Maybe upgrading your qemu-kvm and libvirt packages will solve your problem. Upgrading the host to 20.04 because of this seems to be massive overkill. I would try this ppa first instead.

[TheFu]

I'm on

$ dpkg -l *kvm* |egrep '^ii'
ii  qemu-kvm       1:2.11+dfsg-1ubuntu7.37 amd64        QEMU Full virtualization on x86 hardware

Haven't had any issues on either Intel or Ryzen CPUs with any VMs.

The KVM host kernels on 3 systems are:

$ uname -r
5.4.0-80-generic

that's the HWE kernel.

The guest VMs are almost every supported Ubuntu version and a few EoL Windows systems.

What do the log files show related to warnings and errors on the VM host? Always check the logs.

Update:
I run apt-cacher-ng, but the APT cache doesn't keep packages forever. They automatically expire. Only have

-rw-r--r-- 1 apt-cacher-ng apt-cacher-ng 13176 Jul 17 09:44 /var/cache/apt-cacher-ng/uburep/pool/main/q/qemu/qemu-kvm_2.11+dfsg-1ubuntu7.37_amd64.deb

Let me check my system backups for that machine. I don't usually backup /var/ stuff. Nope. Nothing from /var/ in the backups. Sorry.

[MAFoElffen]

As an example, the KVM version of KVM in 18.04 is 2.11. This is what TheFu is currently on. He knows KVM inside and out.

In 20.04.2, KVM is version 4.2.1. 21.10 Dev, KVM is version 6.0. These are what I am currently on and testing.

KVM upstream now in dev is version 6.1-beta, which monkeybrains compiled and is testing... KVM releases 6.1 at the end of August. I see that as the version for 22.04.

You now have 3 people here to help you... If you can explain what is going on.

[nataraj88]

Interesting, I may try the HWE kernel, though my system is a Dell XPS from around 2017. The problem Iis that the VM's hang and ultimately the whole system hangs, if not during operation of the VM, during system shutdown. I will gather more info the next time it occurs.

[TheFu]

Interesting, I may try the HWE kernel, though my system is a Dell XPS from around 2017. The problem Iis that the VM's hang and ultimately the whole system hangs, if not during operation of the VM, during system shutdown. I will gather more info the next time it occurs.

Did you check the logs for issues? There should be something in the logs which will provide a hint. I was able to track down a RAM issue by reviewing the logs last week and about a month before that.

[MAFoElffen]

The logs he is referring to are located at /var/log/libvirt/qemu/<domain_name.>.log ... There should be one log file for each VM Guest you have...

So a soft lockup? Interesting read on this qemu-kvm bug report at launchpad, between response #3 and #15...

LaunchPad > Bug # 1730717 > Some VMs fail to reboot with "watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]"
= https://bugs.launchpad.net/ubuntu/+s.../+bug/1730717\

At first you are going to see the title and ask me what that has to do with you... Well, it's tied and intertwined with another Bug Report

LaunchPad > Bug # 1713751 > soft lockup / stall on CPU when shutting down with hwe 4.10 kernel
= https://bugs.launchpad.net/ubuntu/+s...x/+bug/1713751

Is not just that kernel, but a range of Kernels... Not saying that is anything like you... but something you said reminded me of these interrelated bugs. And you are on 18.04 ,and I was thinking maybe something setup a perfect storm(?) or not.. Just me always thinking.

You haven't even looked at "your logs" yet... But if you do see messages in those logs saying "soft lockup - CPU#0 stuck for 22s!"... Well...

[nataraj88]

The Errors that I'm finding so far in my logs are these (Note this is just a few of each error, edited down )

Leaked cluster 253193 refcount=3 reference=2
Leaked cluster 262568 refcount=5 reference=4

ERROR cluster 267915 refcount=0 reference=1
ERROR cluster 267916 refcount=0 reference=1

Leaked cluster 275793 refcount=5 reference=4
Leaked cluster 412410 refcount=5 reference=4

Rebuilding refcount structure
Repairing cluster 250896 refcount=1 reference=0
Repairing cluster 250897 refcount=1 reference=0

I took the VM down and ran this:

# qemu-img info ubuntu18.04.qemu
image: ubuntu18.04.qemu
file format: qcow2
virtual size: 15G (16106127360 bytes)
disk size: 33G
cluster_size: 65536
Snapshot list:
ID        TAG                 VM SIZE                DATE       VM CLOCK
1         Ubuntu 18.04 initial install      0 2018-04-24 01:16:36   00:00:00.00\
0
153       21-01-22-01               0 2021-01-22 08:21:28   00:00:00.000
156       21-03-22-01               0 2021-03-22 14:11:23   00:00:00.000
159       21-06-04-02               0 2021-06-04 01:52:25   00:00:00.000
160       21-07-16-01               0 2021-07-16 20:11:04   00:00:00.000
Format specific information:
    compat: 1.1
    lazy refcounts: true
    refcount bits: 16
    corrupt: false
# qemu-img check ubuntu18.04.qemu
No errors were found on the image.
245760/245760 = 100.00% allocated, 15.12% fragmented, 0.00% compressed clusters
Image end offset: 60129673216

So if the image was corrupt it has been repaired now.

[nataraj88]

So at this point, I will see if the problem recurs and try to track down more information. I will also run memtest on my system later tonight.

[TheFu]

So at this point, I will see if the problem recurs and try to track down more information. I will also run memtest on my system later tonight.

I use LVM LVs for my VM storage. The LV is passed as a block device into the VM. libvirt understands how to use an LVM storage pool. Creating a 50G VM storage area is nearly instant - another reason to use LVM for storage. Being able to resize the storage while the VM keeps running in 5 seconds is handy too. Too hard to get a perfectly clean backup from a running VM without using LVM snapshots.

File-based VM storage is so ... so ... 2010. Though I do admit to using it for a few needs, but nothing used in production.

[nataraj88]

Thank you. That is a good point. I do have LVM storage on a few of my VM's that do heavier IO.