booting in emergency mode... what is wrong [read-only file system?]

[pqwoerituytrueiwoq]

Power when out, UPS battery ran out of power, power was lost

BIOS powered up when power was restored, but the system was waiting on user input (Ctrl+D to continue) before it started my VM (pfsense; my router)

it said see journalctl -xb ^(attached) for details (had to connect my portable monitor and grab a keyboard)

---md Status---
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda2[1] sdb2[0]
      124965888 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md1 : active raid1 sdh1[1] sdg1[0]
      124966912 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

unused devices: <none>
---zfs Status---
  pool: mypool
 state: ONLINE
  scan: scrub repaired 0B in 00:01:57 with 0 errors on Sun Sep 10 00:25:59 2023
config:

        NAME                        STATE     READ WRITE CKSUM
        mypool                      ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            wwn-0x50014ee263e2b396  ONLINE       0     0     0
            wwn-0x50014ee263ce2c81  ONLINE       0     0     0
          mirror-1                  ONLINE       0     0     0
            wwn-0x50014ee263e27e85  ONLINE       0     0     0
            wwn-0x50014ee265d71fad  ONLINE       0     0     0

errors: No known data errors
---Disk Usage---
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0p1      117G   13G   99G  12% /
/dev/md1p1      117G   19G   93G  17% /mnt/Data
mypool          7.2T   31G  7.1T   1% /mnt/HDD
chad@niceserver:~$ cat /etc/fstab 
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/md0p1 during curtin installation
/dev/disk/by-id/md-uuid-ede07d21:4114781a:b6762645:db22bf0e-part1 /         ext4 defaults,noatime 0 1
# /mnt/Data was on /dev/md1p1 during curtin installation
/dev/disk/by-id/md-uuid-28843301:920a306f:47f8cd80:d4d35fcd-part1 /mnt/Data ext4 defaults,noatime 0 1
#/dev/disk/by-id/md-uuid-a1a41d92:421eda05:59f6da66:3d3f0f6e       /mnt/HDD  ext4 defaults         0 1
/mnt/Data/kvm-images    /home/chad/kvm/images    bind x-systemd.requires=/mnt/Data,defaults,bind  0 0
/mnt/HDD/www            /var/www                 bind x-systemd.requires=/mnt/HDD,defaults,bind   0 0
/mnt/HDD/apt-cacher-ng  /var/cache/apt-cacher-ng bind x-systemd.requires=/mnt/HDD,defaults,bind   0 0
/swap.img               none                     swap sw                                          0 0

my guess is some mount order stuff with zfs and fsatab, but i do not know
edit found this: https://www.reddit.com/r/zfs/comment...etcfstab_bind/
just edited /etc/fstab with x-system.after=zfs-mount.service maybe that will solve this issue?

[MAFoElffen]

It's a ZFS cache thing. It lost power and corrupted the cache, which was keeping track of the mounts when it goes to boot next. Both 1fallen and I have had to do this a few times over time for various reason. (We tend to break things while pushing them to the limits during testing.) LOL

This is only going to be a summary, as I have to go do something right now:
Boot from a LiveUSB, install intiramfs-zfs, export the pools, import the pools, mount the pools and datasets, chroot into the system, update-initramfs image. Exit the chroot. Exit the filesystem gracefully. Unmount, Reboot.

If you need more details, mayb 1fallen can jump in... or I will be back later. It's not a biggee. Everything is still there.

[MAFoElffen]

Back. But see no replies from you yet.

Booted from an Installer LiveUSB. Try. Open a terminal session.

sudo su
apt update
apt install -y zfs-initramfs zfsutils-linux
zpool export -a
zpool import -N -R /mnt rpool
zpool import -N -R /mnt bpool
UUID=$(zfs list | grep -m 1 'rpool/ROOT/ubuntu_' | cut -b 19-24 )
zfs mount rpool/ROOT/ubuntu_$UUID
zfs mount bpool/BOOT/ubuntu_$UUID
zfs mount -a
mount --make-private --rbind /dev  /mnt/dev
mount --make-private --rbind /proc /mnt/proc
mount --make-private --rbind /sys  /mnt/sys
mount --make-private --rbind /run /mnt/run
chroot /mnt /bin/bash --login
mount -a

apt update
update-intramfs -c -k all
exit

mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
    xargs -i{} umount -lf {}
zpool export -a
reboot

[pqwoerituytrueiwoq]

so a power outage will cause this to happen? can i automate a repair attempt? (having to use a live GUI session on a headless server is no fun)

right now the server is booted and running, i should be able to do it without using a live session, guessing there is a faster way since the server is up and running

chad@niceserver:~$ apt-cache policy zfs-initramfs zfsutils-linux 
zfs-initramfs: 
  Installed: (none) 
  Candidate: 2.1.5-1ubuntu6~22.04.1 
  Version table: 
     2.1.5-1ubuntu6~22.04.1 500 
        500 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages 
        500 http://us.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages 
     2.1.2-1ubuntu3 500 
        500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages 
zfsutils-linux: 
  Installed: 2.1.5-1ubuntu6~22.04.1 
  Candidate: 2.1.5-1ubuntu6~22.04.1 
  Version table: 
 *** 2.1.5-1ubuntu6~22.04.1 500 
        500 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages 
        500 http://us.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages 
        100 /var/lib/dpkg/status 
     2.1.2-1ubuntu3 500 
        500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
$ zfs list 
NAME     USED  AVAIL     REFER  MOUNTPOINT 
mypool  31.0G  7.09T     30.9G  /mnt/HDD

the power did go out again after the original post, before the UPS battery died i used a laptop to ssh in and run poweroff, when it booted up again (assuming it did not go out again then boot again) that would have been a clean shutdown and it still happened
at this time i have this modification to my /etc/fstab, no idea if it will help or not (have not rebooted since saving it)

$ cat /etc/fstab
# /etc/fstab: static file system information. 
# 
# Use 'blkid' to print the universally unique identifier for a 
# device; this may be used with UUID= as a more robust way to name devices 
# that works even if disks are added and removed. See fstab(5). 
# 
# <file system> <mount point>   <type>  <options>       <dump>  <pass> 
# / was on /dev/md0p1 during curtin installation 
/dev/disk/by-id/md-uuid-ede07d21:4114781a:b6762645:db22bf0e-part1 /         ext4 defaults,noatime 0 1 
# /mnt/Data was on /dev/md1p1 during curtin installation 
/dev/disk/by-id/md-uuid-28843301:920a306f:47f8cd80:d4d35fcd-part1 /mnt/Data ext4 defaults,noatime 0 1 
#/dev/disk/by-id/md-uuid-a1a41d92:421eda05:59f6da66:3d3f0f6e       /mnt/HDD  ext4 defaults         0 1 
/mnt/Data/kvm-images    /home/chad/kvm/images    bind x-systemd.requires=/mnt/Data,defaults,bind  0 0 
/mnt/HDD/www            /var/www                 bind x-systemd.after=zfs-mount.service,x-systemd.requires=/mnt/HDD,defaults,bind   0 0 
/mnt/HDD/apt-cacher-ng  /var/cache/apt-cacher-ng bind x-systemd.after=zfs-mount.service,x-systemd.requires=/mnt/HDD,defaults,bind   0 0 
/swap.img               none                     swap sw                                          0 0

when i tried to use ctrl+d the next time it was not enough to get it working, i had to use the reset button and press enter then reboot or something like that


and my pool is degraded now... one scrub and clear and it looks fine

[1fallen]

So is everything back to norm?? you keep getting asked to reply, but nothing from you yet?
Also did you try the suggestion from post #3

apt-cache policy zfs-initramfs zfsutils-linux 
zfs-initramfs:
  Installed: 2.2.0~rc3-0ubuntu4
  Candidate: 2.2.0~rc3-0ubuntu4
  Version table:
 *** 2.2.0~rc3-0ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu mantic/main amd64 Packages
        100 /var/lib/dpkg/status
zfsutils-linux:
  Installed: 2.2.0~rc3-0ubuntu4
  Candidate: 2.2.0~rc3-0ubuntu4
  Version table:
 *** 2.2.0~rc3-0ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu mantic/main amd64 Packages
        100 /var/lib/dpkg/status

[MAFoElffen]

He has no internet, because his Local LAN's router was virtually hosted on that machine, right? You are going to have a challenge with that...

He might want to try to create a persistent Bootable Rescue USB that already has packages initramfs-zfs and zfsutils-linux installed so it so it can work with ZFS. With that portable bootable recovery device, then he wouldn't need a working network connection to recover his server...

***
At the OP: Isn't this the second time the power has gone off and this system has been done from a power surge? You say you have a UPS, but this has happened to your multitple times now, very recently. I think it it where me, I would look into adding more surge suppression and adding some batteries to your your UPS so that you have a longer power loss time.

--AND/OR--

The UPS'es I had connected to my servers via USB and Network ports and used UPS Management software, such as apcupsd or Power Panel for Linux, so that when the power fails, you have an alarm and the server(s), after a certain time, that if the power does not come back up when that timer hits, that the server(s) can be told (by the management software) shutdown gracefully. automatically. You should really think about adding these to your disaster recovery plan.

Fortunately, working with you last time on ZFS, I know you keep good backups. When you get this resolved this time, I think you have a reoccurring problem that you need to plan for to save you some work and heartbreak. Luckily, you recently converted this server to ZFS, where it has better recovery options, and is safer in it's filesystem integrity than what you had previously.

In that recovery plan, I think you need to add a portable Persistent LiveUSB that can work with ZFS (noted above), or add a basic recovery partition, that also has copies of those two packages stored locally.

[1fallen]

He has no internet, because his Local LAN's router was virtually hosted on that machine, right? You are going to have a challenge with that...

Good to know Thanks for that

He might want to try to create a persistent Bootable Rescue USB that has initramfs-zfs and zfsutils-linux installed o it so it can work with ZFS.

Yep i have one already, although I've not had to use it yet but it's good to have for such cases.

[MAFoElffen]

To import and export the pool(s) if ZFS-On-Root, then you need to boot from a different device... so it can update the caches.

But if "booted and running"... then the caches will update on their own, and doing all that is unneeded. Right? If booted and running, why isn't your virtual router up?

I guess I am missing something in that equation...

[1fallen]

why isn't your virtual router up?

I guess I am missing something in that equation...

That makes two of us...

[pqwoerituytrueiwoq]

the instructions in post #3 are for use on a live session, I am not sure what needs to be omitted to run them on a normal session

yes if i can't boot my server i have no internet access (well unless i pull out my WTR54GL) which i have not resorted to


When the power came back up the system booted and my VM had not started as it was at the prompt to press Ctrl+D in emergency mode, upon pressing CTRL+D my VM was started


Order of events:

1. Power went out
2. UPS battery died
3. I got home from work (I'd guess about a hour and a half into the power outage as one of my UPS units was still on and died when i got in the door)
4. went to my mom's so i could get a shower
5. power came back
6. i got back home
7. began trying to figure out why my DHCP server was down
8. upon getting my portable monitor and the 1st USB keyboard i could find connected i read the messages and pressed Ctrl+D and everything was working
9. I made the 1st post (no photos attached)
10. fell asleep in my chair
11. Power went out again and i woke up
12. safely shutdown the server over SSH with a laptop and went to bed (it was running on UPS power)
13. When i got up power was back and my server was down again...
14. plugged a USB keyboard in and pressed Ctrl+D
15. nothing happened and i got the portable monitor out again (see attached photo, the short one)
16. ended up pressing the chassis reset button
17. Pressed crtl+D again (see attached photo, the tall one)
18. ended up pressing the chassis reset button
19. pressed enter instead of Ctrl+D
20. started trying to troubleshoot stuff
21. suddenly stuff was working (my beep speaker must not have sounded or i missed it, it should have beeped when the VM was started and again when it gave my server it's IP address)
22. at this time there were no replies here
23. made a attempt to fix this issue with the /etc/fstab thing as the sometimes it works sometimes not seems like a mount order issue
24. i have not rebooted the system just in case the issue is not resolved, hoping to get a reply here
25. found out i did not have apcupsd installed, got it installed and configured (guess i forgot to do that months ago)
26. now we are here and everyone is confused as to the state of things $ uptime -p up 1 day, 15 hours, 3 minutes
    

We had a tropical storm here Friday night, this is the 1st time my UPS batter has been insufficient to outlast a power outage since it was deployed, though in my EXP it is always around 30-60 min after the battery dies power comes back