22.04 can no longer boot from zfs due to grub bug

[tkubnt]

All of a sudden I was not able to boot 22.04 with root on ZFS. I would get dropped at the grub prompt which could not recognize zfs on the root pool. I use a single `rpool`, no separate `bpool`. Of course I panicked, I thought the drives had failed. I booted from a live USB, was able to import the rpool, scrubbed it, everything checked out OK. Phew, at least the hardware is fine and the data is still there! But I still didn't know how to fix. I noticed this error though:

/usr/sbin/grub-probe: error: compression algorithm inherit not supported

Which lead to this bug and potential future solution to get grub 2.12 into 22.04 or backport the fix to grub 2.06:
https://github.com/openzfs/zfs/issue...ent-1889885090

I eventually managed to boot my system using ZfsBootMenu running off a flash drive but that also was not trivial because it could not import the rpool because I had imported it from the live USB with a different hostid. From the ZBM recovery shell I had to force import it: `zpool import -f rpool` and then I could finally boot it! Phew again!

Now the question is, how did this break all of a sudden and how can we fix grub? Or should we just move to ZfsBootMenu and get it over with?

[MAFoElffen]

Wait, the link you gave was for 'snapshotting top-level "bpool" filesystem causes grub to fail #13873' That doesn't describe what you said you had, nor how you said it happened. How does that relate again?

The top return I get on that same is ' grub-probe ("suddenly") fails with "algorithm inherit not supported" #15261' and 'bug #64297: snapshotting a zfs-based boot pool renders it unreadable by grub'.

When you created the zpool rpool, while not using a bpool (which the Ubuntu Grub scripts are hardwired into), did you set the property capatibilty=grub?

From what methods did you create it as ZFS-On-Root?

While chrooted into it, or booted, what does this return?

sudo grub-probe /boot
sudo zpool get compatibility rpool

[tkubnt]

It looks like it happens if you have any boot pool that you snapshot, otherwise I cannot explain it.

This is a server that started 5 years ago on 18.04 based on these instructions (or similar, looks like the instructions got updated, there was no bpool mention at the time) and then got upgraded to 20.04, 22.04 and hopefully this year to 24.04

> sudo grub-probe /boot
grub-probe: error: compression algorithm inherit not supported
.

> sudo zpool get compatibility rpool
NAME   PROPERTY       VALUE          SOURCE
rpool  compatibility  off            default

This has worked fine for years, not sure what recent update broke it. Last reboot was on Nov 29, then today and I ended up at the grub prompt.

Do you think I should move /boot to its own zpool or better yet just plain old ext4 and never worry about grub issues again?

[MAFoElffen]

I hate to say this, but... In all three of those threads, they never resolved it, except by recreating the zpool.

This is what I would do. chroot into the installed system from an installer LiveUSB. Do you have another drive with enough free storage to backup the pool with rsync? That is what I would do first. Then test to see if it was successful. Then run the 'system-info' script from my signature line, but use this to start it, instead of just normally

./system-info --details

That will document all the ZFS details. Choose to upload the report, and keep track of the URL.

I would boot from an instaler LiveUSB. Deestroy rpool. I would change what you used from that link to create rpool to this

zpool create \
    -o ashift=13 \
    -o autotrim=on \
    -o cachefile=/etc/zfs/zpool.cache \
    -o compatibility=grub2 \
    -o feature@livelist=enabled \
    -o feature@zpool_checkpoint=enabled \
    -O devices=off \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off -O mountpoint=/ -R /mnt \
    rpool $DISK-part3

If you use a HDD change the ashift to 12. If SSD or NVMe, leave it as 13. That includes compatibility options for Grub2.

Then re-install & reconfigure grub.

[tkubnt]

Can I shrink the existing rpool? I'd rather do that and create a new bpool or just ext4 for /boot.
rpool is a mirror of 2 x 1TB SSDs with about 250GB free.
Not sure whether there are any tricks to detach one side, shrink it, attach it again and let it resilver, detach the other side, shrink it, attach and resilver. Or send/receive I guess ...

[MAFoElffen]

Well, if you removed one drive, it would run degraded. Then you could zero the removed drived, and rsync to it. Thinking out loud...

[tkubnt]

Well, if you removed one drive, it would run degraded. Then you could zero the removed drived, and rsync to it. Thinking out loud...

zfs send|receive is probably a better choice over rsync.

[MAFoElffen]

Well, the problem is with the ZFS Zpool right? When I am dumping a pool to destroy it, create new and restore what was on it...

I have done it both ways. Both ways work, though with send/receive you can use higher compression, so it tends to take less space and time.

You could do a full snapshot of the zpool, send/receive it somewhere, destroy original, send/recieve back to restore it... But we don't know where this problem is from yet right?

I also just use rsync to dump the filesystem that was there. Destroy the Main Zpool. Recreate it from my install ZFS Recipe Script, that I created it with, then rsync the filesystem contents back into it. If there is something wonky about the filesystem within a pool, and I'm not sure where tha came with, or I want to change some of the immutable properties that can only be set during pool creation time (and cannot be reset), then I don't want a copy of the pool or it's datasets. Then I want a copy of the filesystem within it.

It's like you do for defragmenting a pool, rebalancing what is written to the members of a RAIDZ, changing compression or encryption, or (now) if adding a disk to a RAIDZ array.
When something is fragmented within a pool, that only way to defragment it it to move the date off it, and right it fresh.
If you run RAIDZ, if you add a disk and grow it, or run with a disk degraded/failed, then it keeps writing data to the other disks, and only writes the new data to that new or re-added disk. To rebalance that data between all the disks, you use the same method as above...

If you change the compression or encrytpted, the existing data is still the old way, the new data written used the changed way, So to get it all the new, changed method, the same method used as above...

If everything is okay, then you make a snapshot of a zpool or dataset, then you send/receive it to somewhere else. So you can roll them back (differential) or restore (full). If you are destroying what was there, then a differential snapshot won't work. You use a full snapshot so it can be restored.

If I want to change something what is in the foundation of the pool, then I rsync the contents and restore it to new. The capatibility=grub2. I think that only applyss during creation. I can see where you can change it after creation, but people are unsure what that really does, and if it doesn anything... Becaseu you can see it in the zpool history, but it doesn't show up right when you do a get property on it.
RE: https://github.com/openzfs/zfs/issues/12261

See the differences with that, and how that might make more sense with that explanation of the why we are trying to be careful with that? Or do you have a bit different perspective of that?

[tkubnt]

I think my plan is to keep the rpool the way it is and just switch to booting using ZBM. If Grub gets fixed some day, then great, another way to boot the same thing. Grub+ZFS feels very messy, I'm surprised I haven't had any problems with it the past 5 years.

[MAFoElffen]

You told us you don't ave a bpool...

Do you have datasets for /boot? Something similar to rpool/BOOT/ubuntu_$UUID? That way, it would be a filesystem type dataset that resides there as if it was a bpool... This is what I usually do if I am not using a bpool.

You could rsync everything under that directory (_-excluding the esp under /boot/efi) and then remove it... create datasets with a mountpoint of /boot... rsync everything back into that dataset, then do a remount to the mount tags there...

I think that would be a lot faster, and make things down the road easier, if it ever happens again. And you can make snapshots of that easily.