After kernel update no zfs

[mattiash]

I had an updates for my Ubuntu server 20.04.3 which is my storage server, with an zfs array. It said that it also was a new kernel (5.15.5-76051505-generic). After the required reboot, the zfs disks are not not mounted. When I try to activate the dkms it says that zfs is not loaded.

# systemctl status zfs-import-cache
● zfs-import-cache.service - Import ZFS pools by cache file
Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2021-12-01 22:21:43 CET; 3min 40s ago
Docs: man:zpool(8)
Process: 1343 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=1/FAILURE)
Main PID: 1343 (code=exited, status=1/FAILURE)

# systemctl status zfs-import-scan
● zfs-import-scan.service - Import ZFS pools by device scanning
Loaded: loaded (/lib/systemd/system/zfs-import-scan.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:zpool(8)

# /sbin/modprobe zfs
modprobe: FATAL: Module zfs not found in directory /lib/modules/5.15.5-76051505-generic

If I check /lib/modules/5.15.5-76051505-generic/kernel/
There is no zfs modules found.
It seems like kernel 5.15 doesn't come with zfs modules installed?!?!
How do I solve this?

[LHammonds]

Reboot and tell it to load the prior kernel. It will likely be corrected in the next kernel update.

LHammonds

[mattiash]

Reboot and tell it to load the prior kernel. It will likely be corrected in the next kernel update.

LHammonds

I wish it was that easy. For some reason the datasets was not mounted, I had to manually mount them. I havn't dared to reboot the server just yet. I am copying all the data to another system that luckily had space enough to host this panic moving. Only 6 TB of data...

Also sure this a home system but still this is not ok. Would this happen in a production environment I would have ditched Ubuntu the next day.

[darkod]

Out of curiocity, are these the standard kernels we are talking about? Because the number you stated is unlike the kernels I have usually seen. Usually they don't have that long number in the middle of the name. I have to admit though that due to various reasons I still haven't installed 20.04 on my home server, so I don't know if that is how kernels are "marked" now.

My point being if you added kernels manually, the more advanced ones still not in the official repos, then maybe other manual steps are needed after the new kernel is added.

[kevdog]

Did you try to regenerate the initramfs?

[Doug S]

Out of curiocity, are these the standard kernels we are talking about? Because the number you stated is unlike the kernels I have usually seen.

Good catch. It looks as though it might be a System76 kernel, but not sure. I am not aware of any falvour of Ubuntu using a 5.15 series kernel by default yet, even 22.04 development.

[kevdog]

Do ubuntu kernels utilize dkms or is the zfs module built into the kernel?

[mattiash]

Well all I can say about the kernel was that I logged in to check something else and I saw that the OS wanted to me restart it. So the kernel was automatically installed.
I did not reinstall zfs or rebuilt the initramfs, panic took hold and I booted using 5.13, manually mounted the datasets. I am in the process of moving 6 TB of off the zfs array.
After that I will try to experiment a bit more. I say that one of the datasets had a folder inside the pool folder.
I had a plan of switching to Debian to be able to run ZoL 2.x instead, but I think I'll wait and get this back to the rock solid system it has been since April 2020 (before that 18.04 and before that 16.04). With 14.04 I tried to use BTRFS that bit me hard, scared me for ever using BTRFS on a server again. Oh well.

[darkod]

We all get the part where the system asked you to reboot. No doubt about it.

My point was that depending how kernels were installed (and as Doug said the 5.15 is not standard kernel for 20.04) then some manual actions might be needed for zfs or other stuff once new kernel is installed.

Things are rather automatic when using the "standard" kernels. But once you go away from that, maybe because you need a newer kernel or what ever, then what happens during kernel update is not so clear any more.

[LHammonds]

I don't use ZFS so I am not familiar with it. But I do know that ZFS is not installed by default on Ubuntu Server 20.04 LTS. That has to be added with "apt install zfsutils-linux"

The kernel versions I've had on a long-running 20.04 LTS server are as follows:

5.4.0-26-generic  5.4.0-48-generic  5.4.0-64-generic  5.4.0-80-generic
5.4.0-29-generic  5.4.0-51-generic  5.4.0-65-generic  5.4.0-81-generic
5.4.0-31-generic  5.4.0-52-generic  5.4.0-66-generic  5.4.0-84-generic
5.4.0-33-generic  5.4.0-53-generic  5.4.0-67-generic  5.4.0-86-generic
5.4.0-37-generic  5.4.0-54-generic  5.4.0-70-generic  5.4.0-88-generic
5.4.0-39-generic  5.4.0-56-generic  5.4.0-71-generic  5.4.0-89-generic
5.4.0-40-generic  5.4.0-58-generic  5.4.0-72-generic  5.4.0-90-generic
5.4.0-42-generic  5.4.0-59-generic  5.4.0-73-generic  5.4.0-91-generic
5.4.0-45-generic  5.4.0-60-generic  5.4.0-74-generic
5.4.0-47-generic  5.4.0-62-generic  5.4.0-77-generic

None of them are the 5.15 flavor. 5.15 is bleeding edge and thus, not recommended for a production server.

LHammonds