20.04: Kernel Panic - not syncing: No working init found

[carpy]

I'm stuck with an unbootable system after my latest update required me to reboot.

Kernel 5.8.0.59-generic
I tried providing "boot=/bin/bash" with the same result.
I've tried break=top,modules,premount and not gotten a shell (not sure what I'm doing wrong there)
I've tried debug=vc but got nothing really helpful.

I'm guessing this is a initrd problem. I've had a similar problem on a different machine and using an older initrd was a workaround.

I'm not sure the role of these microcode segments at the start of the initrd.

This is not a home-grown initrd, it's just plain generated using update-initramfs.

Only thing special is that this machine boots from a LVM volume.

HELP???

[MAFoElffen]

At BIOS POST, can you press the Left shift key and bring up the Grub Boot Menu, form there select an earlier Kernel? And does it boot okay from that?

[carpy]

Unfortunately, none of the older kernels work either, nor the "Recovery" options.
I happened to find backed up copies of the initrd's which still kinda work. Although my NVIDIA Persistence Daemon is failing which is causing my GUI to hang... but that's a separate issue.

I'm still baffled by the smattering of 5.4 and 5.8 kernels around, as well. I haven't seen an LTS change kernel major/minor versions mid-cycle (though I may have just not paid enough attention), is that common?

[carpy]

Can anyone point me to to any guidance which could inform me what I'm doing wrong with the "break" kernel cli arg that would make it not work? I believe, based on https://wiki.debian.org/InitramfsDebug that I should be able to get into the initramfs shell. I don't believe that's only Debian proper, based on this: https://wiki.ubuntu.com/DebuggingKernelBoot

I would like to see what the environment is like pre-boot, so I can better diagnose what's different between the two initrd environments (yes, I know I can unpack them, etc... but that's not the same as live/interactive session in them)

[carpy]

Furthermore, since this has hit me on two different laptops (both with NVidia cards) that something changed in the conversion to 5.8 that breaks certain builds???

[carpy]

I realized that I'm a few kernel revisions out of date from 20.04.3, so I installed the latest (using apt install linux-generic-hwe-20.04), but to no avail. Same issues generating a working initrd for that kernel as well.

[kevdog]

Sorry -- I know how to potentially rescue a system like this with Arch but not with Ubuntu -- although I'm sure behind the scenes its the same process. On Arch I would make an install disk or media, and boot to the media. I would then mount whatever drives (assuming I don't have any encryption) and then perform a chroot. I would then try installing whatever kernel which should then call mkinitcpio to generate the ramdisk. I believe the process is going to be the same here, however it's just that I've never done it with Ubuntu/Debian although I'm aware there is a debian-chroot utility.

[MAFoElffen]

Look at post #3 of the sticky in my signature line. There are instructions there on how to mount and chroot into an Ubuntu System from an Ubuntu LiveCD...

Even before getting it mounted or chrooting in, if you could first, while booted on a LiveCD, post the results of

lsblk -o NAME,SIZE,FSTYPE,LABEL,MOUNTPOINT,MODEL | grep -v '/snap/'

That will help you a lot on what you will need to mount that...

Then it the file manager, looking at your /boot directory and noting which kernel images are there...

[carpy]

Awesome, I know how to mount and chroot in. What then? I have a problem where my initrd generation process appears to be broken

I'm very interested in your suggestions. What next?

I was able to boot in using the 5.11.0-27 kernel/initrd I swiped off the 20.04.3 bootable image. The problem is in the update-initramfs process. I've swapped out for the stock (provided) kernel and initrd images and I boot ok (I run into NVIDIA problems later in the boot because the latest NVIDIA driver expects the updated (5.11.0-34) kernel, but I can boot the system).

Here's the output you requested:

NAME               SIZE FSTYPE      LABEL                     MOUNTPOINT MODEL
nvme0n1            1.9T                                                  PM981a NVMe SAMSUNG 2048GB
├─nvme0n1p1        190M vfat        ESP                       /boot/efi  
├─nvme0n1p2        128M                                                  
├─nvme0n1p3       97.7G ntfs        OS                                   
├─nvme0n1p4        990M ntfs        WINRETOOLS                           
├─nvme0n1p5       13.8G ntfs        Image                                
├─nvme0n1p6        1.3G ntfs        DELLSUPPORT                          
└─nvme0n1p7        1.8T LVM2_member                                      
  └─vg_elf-root       1.8T ext4                                  /

total 753726
drwxr-xr-x  9 root root      4096 Sep 10 14:02 .
drwxr-xr-x 22 root root      4096 Sep 10 15:00 ..
drwxr-xr-x  2 root root      4096 Jul 28 12:36 backup-210728
drwxr-xr-x  2 root root      4096 Jul 28 13:00 backup-210728-2
drwxr-xr-x  2 root root      4096 Sep  6 01:17 backup-210904
drwxr-xr-x  2 root root      4096 Sep 10 14:02 backup-210910
-rw-r--r--  1 root root    253599 Aug 11 10:53 config-5.11.0-27-generic
-rw-r--r--  1 root root    253507 Aug 27 03:43 config-5.11.0-34-generic
-rw-r--r--  1 root root    237807 Nov  5  2020 config-5.4.0-54-generic
-rw-r--r--  1 root root    237850 Feb 19  2021 config-5.4.0-67-generic
-rw-r--r--  1 root root    237883 Aug 26 14:48 config-5.4.0-84-generic
-rw-r--r--  1 root root    248277 Feb 23  2021 config-5.8.0-45-generic
drwx------  4 root root      2048 Dec 31  1969 efi
drwxr-xr-x  4 root root      4096 Sep  6 23:50 from_nomad256_20.04.3
drwxr-xr-x  4 root root      4096 Sep 10 15:00 grub
lrwxrwxrwx  1 root root        27 Sep  8 06:44 initrd.img -> initrd.img-5.4.0-84-generic
-rw-r--r--  1 root root  57294335 Sep 10 06:28 initrd.img-5.11.0-27-generic
-rw-r--r--  1 root root  57303852 Sep 10 06:29 initrd.img-5.11.0-34-generic
-rw-r--r--  1 root root  87063852 Sep  6 12:28 initrd.img-5.4.0-54-generic
-rw-r--r--  1 root root  88070926 Sep  6 12:28 initrd.img-5.4.0-67-generic
-rwxr-x---  1 root root 115956285 Sep  6 12:28 initrd.img-5.4.0-77-generic
-rw-r--r--  1 root root  56287179 Sep  8 06:45 initrd.img-5.4.0-84-generic
-rw-r--r--  1 root root  91555308 Sep  6 12:28 initrd.img-5.8.0-45-generic
-rwxr-x---  1 root root 119456182 Sep  6 12:28 initrd.img-5.8.0-55-generic
lrwxrwxrwx  1 root root        28 Sep  8 06:44 initrd.img.old -> initrd.img-5.11.0-34-generic
-rw-r--r--  1 root root    182704 Aug 18  2020 memtest86+.bin
-rw-r--r--  1 root root    184380 Aug 18  2020 memtest86+.elf
-rw-r--r--  1 root root    184884 Aug 18  2020 memtest86+_multiboot.bin
-rw-------  1 root root   5833109 Aug 11 10:53 System.map-5.11.0-27-generic
-rw-------  1 root root   5836236 Aug 27 03:43 System.map-5.11.0-34-generic
-rw-------  1 root root   4743490 Nov  5  2020 System.map-5.4.0-54-generic
-rw-------  1 root root   4749481 Feb 19  2021 System.map-5.4.0-67-generic
-rw-------  1 root root   4753288 Aug 26 14:48 System.map-5.4.0-84-generic
-rw-------  1 root root   5520433 Feb 23  2021 System.map-5.8.0-45-generic
lrwxrwxrwx  1 root root        24 Sep  8 06:44 vmlinuz -> vmlinuz-5.4.0-84-generic
-rw-------  1 root root  10125664 Aug 11 11:39 vmlinuz-5.11.0-27-generic
-rw-------  1 root root  10132256 Aug 27 03:48 vmlinuz-5.11.0-34-generic
-rw-------  1 root root  11678464 Nov  5  2020 vmlinuz-5.4.0-54-generic
-rw-------  1 root root  11756288 Feb 19  2021 vmlinuz-5.4.0-67-generic
-rw-------  1 root root  11776256 Aug 26 14:54 vmlinuz-5.4.0-84-generic
-rw-------  1 root root   9781120 Feb 24  2021 vmlinuz-5.8.0-45-generic
lrwxrwxrwx  1 root root        25 Sep  8 06:44 vmlinuz.old -> vmlinuz-5.11.0-34-generic

[carpy]

Look at post #3 of the sticky in my signature line. There are instructions there on how to mount and chroot into an Ubuntu System from an Ubuntu LiveCD...

That is quite a treasure trove of information, btw. Thanks for putting it together and actively maintaining it. Lots of nuggets I often have to google about wrt video drivers, and the whole thing is quite well written