Kernel upgrade - server keeps booting from old kernel

[ajgreeny]

Unfortunately I can not help at all with LVM installs as I have never used it and do not understand it at all.

However, as mentioned above by MAFoElffen it could be possible that you have more than one grub on your computer but I can see no mention of that in your various output posts shown here.

I think it might be worth using the Boot-Repair in my signature below. Follow the instructions there to run the Boot-Info-Script.
Do not run the default repair just yet but simply copy back here the pastebin link you get which will show us a lot more about your system and all the boot system currently in use.
It can certainly do no harm as long as you do not just run the Default repair, and it may just help figure out why grub is not doing what it should.

[TheFu]

I use LVM and have upgraded a 16.04 --> 18.04 install. I don't remember anything funny related to storage, though I did have some issues with my perlbrew management of perl versions getting in the way of the upgrade. Lots of things were broke due to that - 100% my fault for putting a very new perl in my PATH before the system perl version and having lots of very new CPAN modules to go along with that perl version rather than using the Canonical Repo versions of those perl modules.

I have had multiple disks with identical UUIDs and that definitely causes some odd behaviors until I finally figured out what was happening. Thought I was going mad for a few weeks, since about 50% of the time a different OS would be booted and programs I'd just installed and was using weren't there. Crazy, right? Once I figured out that UUID-based mounts were the issue and were being found randomly, I immediately forced UUID changes for any partitions/LVs that had a duplicate. After that, all those issues stopped. The same sort of thing can happen if VGs and LVs are named the same, so be certain that isn't possible when LVM is used. There are lots of different techniques that people use for this. vg00, vg01, vg02 ... come to mind.

Also, I've found this df alias NOT to show junk we don't want to see:

alias dft='df -hT -x squashfs -x tmpfs -x devtmpfs'

Not sure if that would remove all the docker crap or not. I've only played with docker.

[MAFoElffen]

@TheFu

If you look at the commands both of us posted, they are exactly the same commands, with mine actually filtering out more (trying to exclude Snaps), so yours would actually not filter out those as you thought...

If it might be a problem of duplicate UUID's like TheFu brought up, then additionally:

lsblk -o NAME,HOTPLUG,PARTUUID,UUID | grep -v 'loop'

[TheFu]

@TheFu

If you look at the commands both of us posted, they are exactly the same commands, with mine actually filtering out more (trying to exclude Snaps), so yours would actually not filter out those as you thought...

If it might be a problem of duplicate UUID's like TheFu brought up, then additionally:

lsblk -o NAME,HOTPLUG,PARTUUID,UUID | grep -v 'loop'

Mine also excludes squashfs ... which includes snaps and some other things ... perhaps docker?

[kinser06]

Unfortunately I can not help at all with LVM installs as I have never used it and do not understand it at all.

However, as mentioned above by MAFoElffen it could be possible that you have more than one grub on your computer but I can see no mention of that in your various output posts shown here.

I think it might be worth using the Boot-Repair in my signature below. Follow the instructions there to run the Boot-Info-Script.
Do not run the default repair just yet but simply copy back here the pastebin link you get which will show us a lot more about your system and all the boot system currently in use.
It can certainly do no harm as long as you do not just run the Default repair, and it may just help figure out why grub is not doing what it should.

Thanks for this, i will work on it tomorrow and post the output.

[kinser06]

@TheFu

If you look at the commands both of us posted, they are exactly the same commands, with mine actually filtering out more (trying to exclude Snaps), so yours would actually not filter out those as you thought...

If it might be a problem of duplicate UUID's like TheFu brought up, then additionally:

lsblk -o NAME,HOTPLUG,PARTUUID,UUID | grep -v 'loop'

lsblk -o NAME,HOTPLUG,PARTUUID,UUID | grep -v 'loop' showes

root@dellserver:/home/kinser06# lsblk -o NAME,HOTPLUG,PARTUUID,UUID | grep -v 'loop'
NAME                      HOTPLUG PARTUUID                             UUID
sda                             0
├─sda1                          0 e54dd991-32b8-4b9c-8e6c-76dd9addf18b 1A4D-ED70
├─sda2                          0 4f8d2be0-d57f-4e07-b5a3-be0b23cc06f3 85498d87-6d8b-11eb-bfff-b083feda1f05
└─sda3                          0 2e5574d4-3470-4c5c-b2fa-1d84cc161547 U4k6HX-WVr0-Y2KI-MzsK-IMug-vPw1-n8tjOB
  └─ubuntu--vg-ubuntu--lv       0                                      85498d88-6d8b-11eb-bfff-b083feda1f05
sdb                             0
└─sdb1                          0 207bcd68-a916-be49-8bc8-0e5a88e4d58c c8ab2a4b-79ab-4b9b-865d-f2e97eed6309

[MAFoElffen]

I don't see any duplicates. So no conflicts there... Go to
https://help.ubuntu.com/community/Boot-Repair


Do Option #2 on Yann's PPA.That is the most current scripts. Yann asked me and some others to help him test this, so I know it's up to date and in active development right now. The ISO will be update soon, once we are through... run the boot-info report and paste it to a pasebin. Do not apply anything until we can read that report and interpret it.

[kinser06]

I don't see any duplicates. So no conflicts there... Go to
https://help.ubuntu.com/community/Boot-Repair


Do Option #2 on Yann's PPA.That is the most current scripts. Yann asked me and some others to help him test this, so I know it's up to date and in active development right now. The ISO will be update soon, once we are through... run the boot-info report and paste it to a pasebin. Do not apply anything until we can read that report and interpret it.

https://paste.ubuntu.com/p/WGnQQYC88M/

If you'd prefer the contents be uploaded let me know. I didn't want to fill a entire page with its output. sdc1 is the live-boot flash drive.

[ajgreeny]

I am not knowledgable enough to know what is going on here as the LVM info is pretty meaningless to me.

However, I note that you have both a grub.cfg and a menu.list file on the system both of which are loaded by grub according to the report.

I have not seen a menu.list file in Ubuntu for a very long time as it was used by legacy grub, ie grub 1 long ago overtaken by grub 2.
I will leave this to others to help further but I wonder if this is at least partly behind your problem.

[kinser06]

Doesn't it seems odd its loading grub from sda2 but once booted sda2 isn't mounted?