Page 1 of 2 12 LastLast
Results 1 to 10 of 17

Thread: 22.04 can no longer boot from zfs due to grub bug

Hybrid View

  1. #1
    Join Date
    Jun 2023
    Beans
    8

    Exclamation 22.04 can no longer boot from zfs due to grub bug

    All of a sudden I was not able to boot 22.04 with root on ZFS. I would get dropped at the grub prompt which could not recognize zfs on the root pool. I use a single `rpool`, no separate `bpool`. Of course I panicked, I thought the drives had failed. I booted from a live USB, was able to import the rpool, scrubbed it, everything checked out OK. Phew, at least the hardware is fine and the data is still there! But I still didn't know how to fix. I noticed this error though:

    Code:
    /usr/sbin/grub-probe: error: compression algorithm inherit not supported
    Which lead to this bug and potential future solution to get grub 2.12 into 22.04 or backport the fix to grub 2.06:
    https://github.com/openzfs/zfs/issue...ent-1889885090

    I eventually managed to boot my system using ZfsBootMenu running off a flash drive but that also was not trivial because it could not import the rpool because I had imported it from the live USB with a different hostid. From the ZBM recovery shell I had to force import it: `zpool import -f rpool` and then I could finally boot it! Phew again!

    Now the question is, how did this break all of a sudden and how can we fix grub? Or should we just move to ZfsBootMenu and get it over with?
    Last edited by tkubnt; January 16th, 2024 at 03:13 AM.

  2. #2
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 22.04 can no longer boot from zfs due to grub bug

    Wait, the link you gave was for 'snapshotting top-level "bpool" filesystem causes grub to fail #13873' That doesn't describe what you said you had, nor how you said it happened. How does that relate again?

    The top return I get on that same is ' grub-probe ("suddenly") fails with "algorithm inherit not supported" #15261' and 'bug #64297: snapshotting a zfs-based boot pool renders it unreadable by grub'.

    When you created the zpool rpool, while not using a bpool (which the Ubuntu Grub scripts are hardwired into), did you set the property capatibilty=grub?

    From what methods did you create it as ZFS-On-Root?

    While chrooted into it, or booted, what does this return?
    Code:
    sudo grub-probe /boot
    sudo zpool get compatibility rpool
    Last edited by MAFoElffen; January 16th, 2024 at 03:57 AM.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  3. #3
    Join Date
    Jun 2023
    Beans
    8

    Re: 22.04 can no longer boot from zfs due to grub bug

    It looks like it happens if you have any boot pool that you snapshot, otherwise I cannot explain it.

    This is a server that started 5 years ago on 18.04 based on these instructions (or similar, looks like the instructions got updated, there was no bpool mention at the time) and then got upgraded to 20.04, 22.04 and hopefully this year to 24.04

    Code:
    > sudo grub-probe /boot
    grub-probe: error: compression algorithm inherit not supported
    .
    Code:
    > sudo zpool get compatibility rpool
    NAME   PROPERTY       VALUE          SOURCE
    rpool  compatibility  off            default
    This has worked fine for years, not sure what recent update broke it. Last reboot was on Nov 29, then today and I ended up at the grub prompt.

    Do you think I should move /boot to its own zpool or better yet just plain old ext4 and never worry about grub issues again?
    Last edited by tkubnt; January 16th, 2024 at 04:28 AM.

  4. #4
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 22.04 can no longer boot from zfs due to grub bug

    I hate to say this, but... In all three of those threads, they never resolved it, except by recreating the zpool.

    This is what I would do. chroot into the installed system from an installer LiveUSB. Do you have another drive with enough free storage to backup the pool with rsync? That is what I would do first. Then test to see if it was successful. Then run the 'system-info' script from my signature line, but use this to start it, instead of just normally
    Code:
    ./system-info --details
    That will document all the ZFS details. Choose to upload the report, and keep track of the URL.

    I would boot from an instaler LiveUSB. Deestroy rpool. I would change what you used from that link to create rpool to this
    Code:
    zpool create \
        -o ashift=13 \
        -o autotrim=on \
        -o cachefile=/etc/zfs/zpool.cache \
        -o compatibility=grub2 \
        -o feature@livelist=enabled \
        -o feature@zpool_checkpoint=enabled \
        -O devices=off \
        -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
        -O compression=lz4 \
        -O normalization=formD \
        -O relatime=on \
        -O canmount=off -O mountpoint=/ -R /mnt \
        rpool $DISK-part3
    If you use a HDD change the ashift to 12. If SSD or NVMe, leave it as 13. That includes compatibility options for Grub2.

    Then re-install & reconfigure grub.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  5. #5
    Join Date
    Jun 2023
    Beans
    8

    Re: 22.04 can no longer boot from zfs due to grub bug

    Can I shrink the existing rpool? I'd rather do that and create a new bpool or just ext4 for /boot.
    rpool is a mirror of 2 x 1TB SSDs with about 250GB free.
    Not sure whether there are any tricks to detach one side, shrink it, attach it again and let it resilver, detach the other side, shrink it, attach and resilver. Or send/receive I guess ...
    Last edited by tkubnt; January 16th, 2024 at 07:06 AM.

  6. #6
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 22.04 can no longer boot from zfs due to grub bug

    Well, if you removed one drive, it would run degraded. Then you could zero the removed drived, and rsync to it. Thinking out loud...

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  7. #7
    Join Date
    Jun 2023
    Beans
    8

    Re: 22.04 can no longer boot from zfs due to grub bug

    Quote Originally Posted by MAFoElffen View Post
    Well, if you removed one drive, it would run degraded. Then you could zero the removed drived, and rsync to it. Thinking out loud...
    zfs send|receive is probably a better choice over rsync.

  8. #8
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 22.04 can no longer boot from zfs due to grub bug

    Well, the problem is with the ZFS Zpool right? When I am dumping a pool to destroy it, create new and restore what was on it...

    I have done it both ways. Both ways work, though with send/receive you can use higher compression, so it tends to take less space and time.

    You could do a full snapshot of the zpool, send/receive it somewhere, destroy original, send/recieve back to restore it... But we don't know where this problem is from yet right?

    I also just use rsync to dump the filesystem that was there. Destroy the Main Zpool. Recreate it from my install ZFS Recipe Script, that I created it with, then rsync the filesystem contents back into it. If there is something wonky about the filesystem within a pool, and I'm not sure where tha came with, or I want to change some of the immutable properties that can only be set during pool creation time (and cannot be reset), then I don't want a copy of the pool or it's datasets. Then I want a copy of the filesystem within it.

    It's like you do for defragmenting a pool, rebalancing what is written to the members of a RAIDZ, changing compression or encryption, or (now) if adding a disk to a RAIDZ array.
    When something is fragmented within a pool, that only way to defragment it it to move the date off it, and right it fresh.
    If you run RAIDZ, if you add a disk and grow it, or run with a disk degraded/failed, then it keeps writing data to the other disks, and only writes the new data to that new or re-added disk. To rebalance that data between all the disks, you use the same method as above...

    If you change the compression or encrytpted, the existing data is still the old way, the new data written used the changed way, So to get it all the new, changed method, the same method used as above...

    If everything is okay, then you make a snapshot of a zpool or dataset, then you send/receive it to somewhere else. So you can roll them back (differential) or restore (full). If you are destroying what was there, then a differential snapshot won't work. You use a full snapshot so it can be restored.

    If I want to change something what is in the foundation of the pool, then I rsync the contents and restore it to new. The capatibility=grub2. I think that only applyss during creation. I can see where you can change it after creation, but people are unsure what that really does, and if it doesn anything... Becaseu you can see it in the zpool history, but it doesn't show up right when you do a get property on it.
    RE: https://github.com/openzfs/zfs/issues/12261

    See the differences with that, and how that might make more sense with that explanation of the why we are trying to be careful with that? Or do you have a bit different perspective of that?

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  9. #9
    Join Date
    Jun 2023
    Beans
    8

    Re: 22.04 can no longer boot from zfs due to grub bug

    I think my plan is to keep the rpool the way it is and just switch to booting using ZBM. If Grub gets fixed some day, then great, another way to boot the same thing. Grub+ZFS feels very messy, I'm surprised I haven't had any problems with it the past 5 years.
    Last edited by tkubnt; January 16th, 2024 at 09:48 PM.

  10. #10
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 22.04 can no longer boot from zfs due to grub bug

    You told us you don't ave a bpool...

    Do you have datasets for /boot? Something similar to rpool/BOOT/ubuntu_$UUID? That way, it would be a filesystem type dataset that resides there as if it was a bpool... This is what I usually do if I am not using a bpool.

    You could rsync everything under that directory (_-excluding the esp under /boot/efi) and then remove it... create datasets with a mountpoint of /boot... rsync everything back into that dataset, then do a remount to the mount tags there...

    I think that would be a lot faster, and make things down the road easier, if it ever happens again. And you can make snapshots of that easily.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •