booting in emergency mode... what is wrong [read-only file system?]

**MAFoElffen** · September 25th, 2023

First you should back off any changes you made to the fstab file. You should really keep a copy of that in your User Home directory... So you have a copy of what was working.

With it running, it should have refreshed the zpool cache files in /etc/zfs/zfs-list.cache/ already. Just to make sure, and that all is loaded early in the ramdisk image during boot, as a failsafe to that, all you should have to do is

Code:

sudo update-initramfs -c -k all

Then reboot to test.

It came up by itself? (Hiting <Enter> instead of <Cntrl><D>) Good deal.

**pqwoerituytrueiwoq** · September 25th, 2023

this morning i noted some errors in my array from, but i was able to pass a scrub so i cleared it and the drive shows good in SMART, if i have ANOTHER bad sata cable... anyone know where i can get some high quality sata cables? preferablly shielded case this is getting anying, maybe that was just related to the power issue, but i will be keeping a eye on it

i will try the fstab changes to see if it gives a error, as long is it does not hurt i'd rather keep that change, maybe the option is redundant, maybe it is not eigther way it needs to behave in that way

that cache dir does not exist...

Code:

chad@niceserver:~$ ls -l /etc/zfs/zpool.cache  
-rw-r--r-- 1 root root 3724 Sep 25 09:32 /etc/zfs/zpool.cache 
chad@niceserver:~$ tree /etc/zfs/  
/etc/zfs/ 
├── zed.d 
│   ├── all-syslog.sh -> /usr/lib/zfs-linux/zed.d/all-syslog.sh 
│   ├── data-notify.sh -> /usr/lib/zfs-linux/zed.d/data-notify.sh 
│   ├── history_event-zfs-list-cacher.sh -> /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh 
│   ├── pool_import-led.sh -> /usr/lib/zfs-linux/zed.d/pool_import-led.sh 
│   ├── resilver_finish-notify.sh -> /usr/lib/zfs-linux/zed.d/resilver_finish-notify.sh 
│   ├── resilver_finish-start-scrub.sh -> /usr/lib/zfs-linux/zed.d/resilver_finish-start-scrub.sh 
│   ├── scrub_finish-notify.sh -> /usr/lib/zfs-linux/zed.d/scrub_finish-notify.sh 
│   ├── statechange-led.sh -> /usr/lib/zfs-linux/zed.d/statechange-led.sh 
│   ├── statechange-notify.sh -> /usr/lib/zfs-linux/zed.d/statechange-notify.sh 
│   ├── vdev_attach-led.sh -> /usr/lib/zfs-linux/zed.d/vdev_attach-led.sh 
│   ├── vdev_clear-led.sh -> /usr/lib/zfs-linux/zed.d/vdev_clear-led.sh 
│   ├── zed-functions.sh 
│   └── zed.rc 
├── zfs-functions 
├── zpool.cache 
└── zpool.d 
    ├── ata_err -> /usr/lib/zfs-linux/zpool.d/ata_err 
    ├── cmd_to -> /usr/lib/zfs-linux/zpool.d/cmd_to 
    ├── defect -> /usr/lib/zfs-linux/zpool.d/defect 
    ├── dm-deps -> /usr/lib/zfs-linux/zpool.d/dm-deps 
    ├── enc -> /usr/lib/zfs-linux/zpool.d/enc 
    ├── encdev -> /usr/lib/zfs-linux/zpool.d/encdev 
    ├── fault_led -> /usr/lib/zfs-linux/zpool.d/fault_led 
    ├── health -> /usr/lib/zfs-linux/zpool.d/health 
    ├── hours_on -> /usr/lib/zfs-linux/zpool.d/hours_on 
    ├── iostat -> /usr/lib/zfs-linux/zpool.d/iostat 
    ├── iostat-10s -> /usr/lib/zfs-linux/zpool.d/iostat-10s 
    ├── iostat-1s -> /usr/lib/zfs-linux/zpool.d/iostat-1s 
    ├── label -> /usr/lib/zfs-linux/zpool.d/label 
    ├── locate_led -> /usr/lib/zfs-linux/zpool.d/locate_led 
    ├── lsblk -> /usr/lib/zfs-linux/zpool.d/lsblk 
    ├── media -> /usr/lib/zfs-linux/zpool.d/media 
    ├── model -> /usr/lib/zfs-linux/zpool.d/model 
    ├── nonmed -> /usr/lib/zfs-linux/zpool.d/nonmed 
    ├── nvme_err -> /usr/lib/zfs-linux/zpool.d/nvme_err 
    ├── off_ucor -> /usr/lib/zfs-linux/zpool.d/off_ucor 
    ├── pend_sec -> /usr/lib/zfs-linux/zpool.d/pend_sec 
    ├── pwr_cyc -> /usr/lib/zfs-linux/zpool.d/pwr_cyc 
    ├── realloc -> /usr/lib/zfs-linux/zpool.d/realloc 
    ├── rep_ucor -> /usr/lib/zfs-linux/zpool.d/rep_ucor 
    ├── r_proc -> /usr/lib/zfs-linux/zpool.d/r_proc 
    ├── r_ucor -> /usr/lib/zfs-linux/zpool.d/r_ucor 
    ├── serial -> /usr/lib/zfs-linux/zpool.d/serial 
    ├── ses -> /usr/lib/zfs-linux/zpool.d/ses 
    ├── size -> /usr/lib/zfs-linux/zpool.d/size 
    ├── slot -> /usr/lib/zfs-linux/zpool.d/slot 
    ├── smart -> /usr/lib/zfs-linux/zpool.d/smart 
    ├── smart_test -> /usr/lib/zfs-linux/zpool.d/smart_test 
    ├── smartx -> /usr/lib/zfs-linux/zpool.d/smartx 
    ├── temp -> /usr/lib/zfs-linux/zpool.d/temp 
    ├── test_ended -> /usr/lib/zfs-linux/zpool.d/test_ended 
    ├── test_progress -> /usr/lib/zfs-linux/zpool.d/test_progress 
    ├── test_status -> /usr/lib/zfs-linux/zpool.d/test_status 
    ├── test_type -> /usr/lib/zfs-linux/zpool.d/test_type 
    ├── upath -> /usr/lib/zfs-linux/zpool.d/upath 
    ├── vendor -> /usr/lib/zfs-linux/zpool.d/vendor 
    ├── w_proc -> /usr/lib/zfs-linux/zpool.d/w_proc 
    └── w_ucor -> /usr/lib/zfs-linux/zpool.d/w_ucor 

2 directories, 57 files

**pqwoerituytrueiwoq** · September 25th, 2023

now the issue is worse and i have my WTR54GL out

my md array for / is read-only as a result nothing is working

Code:

xubuntu@xubuntu:~$ cat /proc/mdstat 
Personalities : [raid1]  
md1 : active raid1 sdc1[0] sdd1[1] 
      124966912 blocks super 1.2 [2/2] [UU] 
      bitmap: 0/1 pages [0KB], 65536KB chunk 

md0 : active raid1 sdb2[0] sda2[1] 
      124965888 blocks super 1.2 [2/2] [UU] 
      bitmap: 0/1 pages [0KB], 65536KB chunk 

unused devices: <none> 
xubuntu@xubuntu:~$ sudo mdadm --detail /dev/md0 
/dev/md0: 
           Version : 1.2 
     Creation Time : Tue Sep 20 15:23:38 2022 
        Raid Level : raid1 
        Array Size : 124965888 (119.18 GiB 127.97 GB) 
     Used Dev Size : 124965888 (119.18 GiB 127.97 GB) 
      Raid Devices : 2 
     Total Devices : 2 
       Persistence : Superblock is persistent 

     Intent Bitmap : Internal 

       Update Time : Mon Sep 25 15:16:55 2023 
             State : clean  
    Active Devices : 2 
   Working Devices : 2 
    Failed Devices : 0 
     Spare Devices : 0 

Consistency Policy : bitmap 

              Name : ubuntu-server:0 
              UUID : ede07d21:4114781a:b6762645:db22bf0e 
            Events : 2154 

    Number   Major   Minor   RaidDevice State 
       0       8       18        0      active sync   /dev/sdb2 
       1       8        2        1      active sync   /dev/sda2 
xubuntu@xubuntu:~$ sudo mdadm --detail /dev/md1 
/dev/md1: 
           Version : 1.2 
     Creation Time : Tue Sep 20 15:23:25 2022 
        Raid Level : raid1 
        Array Size : 124966912 (119.18 GiB 127.97 GB) 
     Used Dev Size : 124966912 (119.18 GiB 127.97 GB) 
      Raid Devices : 2 
     Total Devices : 2 
       Persistence : Superblock is persistent 

     Intent Bitmap : Internal 

       Update Time : Mon Sep 25 15:17:00 2023 
             State : clean  
    Active Devices : 2 
   Working Devices : 2 
    Failed Devices : 0 
     Spare Devices : 0 

Consistency Policy : bitmap 

              Name : ubuntu-server:1 
              UUID : 28843301:920a306f:47f8cd80:d4d35fcd 
            Events : 3426 

    Number   Major   Minor   RaidDevice State 
       0       8       33        0      active sync   /dev/sdc1 
       1       8       49        1      active sync   /dev/sdd1

**pqwoerituytrueiwoq** · September 25th, 2023

found the cause in my fstab file

Code:

#/mnt/HDD/www            /var/www                 bind x-systemd.after=zfs-mount.service,x-systemd.requires=/mnt/HDD,defaults,bind   0 0 
#/mnt/HDD/www            /var/www                 bind x-systemd.requires=/mnt/HDD,defaults,bind   0 0 
#/mnt/HDD/apt-cacher-ng  /var/cache/apt-cacher-ng bind x-systemd.after=zfs-mount.service,x-systemd.requires=/mnt/HDD,defaults,bind   0 0 
#/mnt/HDD/apt-cacher-ng  /var/cache/apt-cacher-ng bind x-systemd.requires=/mnt/HDD,defaults,bind   0 0

how do i make it work with ZFS?
preferably something better than a cron job to run a script

Code:

@reboot /usr/local/sbin/zfs_binds

Code:

$ cat /usr/local/sbin/zfs_binds    
#/bin/sh 
while [ ! -d /mnt/HDD/www ] || [ ! -d /mnt/HDD/apt-cacher-ng ];do 
        sleep 1 
done 
if mountpoint -q /var/www;then 
        echo "/var/www is already mounted" 
else 
        mount --bind /mnt/HDD/www /var/www 
fi 
if mountpoint -q /var/cache/apt-cacher-ng;then 
        echo "/var/cache/apt-cacher-ng is already mounted" 
else 
        mount --bind /mnt/HDD/apt-cacher-ng /var/cache/apt-cacher-ng 
fi 
exit 0

**1fallen** · September 25th, 2023

May we see this:

Code:

sudo zfs get all | grep mountpoint

**pqwoerituytrueiwoq** · September 25th, 2023

Code:

mypool  mountpoint            /mnt/HDD               local

Code:

$ cat /proc/mounts 
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 
udev /dev devtmpfs rw,nosuid,relatime,size=3943536k,nr_inodes=985884,mode=755,inode64 0 0 
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 
tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=800304k,mode=755,inode64 0 0 
/dev/md0p1 / ext4 rw,noatime 0 0 
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 
tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiv
eprot 0 0 
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,m
axproto=5,direct,pipe_ino=25034 0 0 
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 
none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode
=700 0 0 
tmpfs /run/qemu tmpfs rw,nosuid,nodev,relatime,mode=755,inode64 0 0 
/dev/md1p1 /mnt/Data ext4 rw,noatime 0 0 
/dev/md1p1 /home/chad/kvm/images ext4 rw,noatime 0 0 
mypool /mnt/HDD zfs rw,xattr,noacl 0 0 
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=800304k,nr_inodes=200076,mode=70
0,uid=1000,gid=1000,inode64 0 0 
mypool /var/www zfs rw,xattr,noacl 0 0 
mypool /var/cache/apt-cacher-ng zfs rw,xattr,noacl 0 0

**1fallen** · September 25th, 2023

Sorry I'm playing catch up here, I was under the impression this was a full ZFS-Root, is that not right?

**pqwoerituytrueiwoq** · September 25th, 2023

note the issue seems to be my binds in /etc/fstab, disabling them made the system happy, these binds are linked to my zfs array

No i have 3 arrays
2 are in md using raid1 (/ and /mnt/Data)
1 is in zfs using raid10: /mnt/HDD

md0 is / and md1 is /mnt/Data

Code:

$ cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]  
md1 : active raid1 sdc1[0] sdd1[1] 
      124966912 blocks super 1.2 [2/2] [UU] 
      bitmap: 1/1 pages [4KB], 65536KB chunk 

md0 : active raid1 sdb2[0] sda2[1] 
      124965888 blocks super 1.2 [2/2] [UU] 
      bitmap: 1/1 pages [4KB], 65536KB chunk 

unused devices: <none>
$ zpool status 
  pool: mypool 
 state: ONLINE 
  scan: scrub repaired 0B in 00:01:55 with 0 errors on Mon Sep 25 09:32:34 2023 
config: 

        NAME                        STATE     READ WRITE CKSUM 
        mypool                      ONLINE       0     0     0 
          mirror-0                  ONLINE       0     0     0 
            wwn-0x50014ee263e2b396  ONLINE       0     0     0 
            wwn-0x50014ee263ce2c81  ONLINE       0     0     0 
          mirror-1                  ONLINE       0     0     0 
            wwn-0x50014ee263e27e85  ONLINE       0     0     0 
            wwn-0x50014ee265d71fad  ONLINE       0     0     0 

errors: No known data errors

**1fallen** · September 25th, 2023

Originally Posted by pqwoerituytrueiwoq

note the issue seems to be my binds in /etc/fstab, disabling them made the system happy, these binds are linked to my zfs array

Yep I'm caught up to that point

Originally Posted by pqwoerituytrueiwoq

No i have 3 arrays
2 are in md using raid1 (/ and /mnt/Data)
1 is in zfs using raid10: /mnt/HDD

Ok I'll watch on the sidelines, I just do different is all.
Not to confuse you just how I do for my usage.

Code:

zfs mount poolname
zfs mount poolname/datasetname

Yes, zfs instead of zpool with the poolname and then poolname/datasetname.
MAFoElffen is the right person for this task.

**MAFoElffen** · September 25th, 2023

I know your problem. (You have multiple conflicting mounts.) It's just going to take a while on my part, sorry.

Please stop until I can type this all out to explain what is going o, and what to do to change it... and in the process explain how ZFS mounts work and what 'should be' in your fstab, and what should not.

It's just a learning curve thing with ZFS. 1fallen will see it once I point it out.

@1fallen -- do <Cntrl><F> on the thread page, and search on "/mnt/HDD", then notice that he has a ZFS Pool (mypool), with no underlying ZFS DataSets... Hint. See the problem now? I'm going to type out a plan for him that should explain everything about that. That script of his is going to blow it up further, even though he thinks that will work. The solution to this is much simpler.

@ pqwoerituytrueiwoq's -- In the meanwhile, please post the results of this

Code:

sudo zfs list mypool

That output will make this all obvious. LOL

EDIT -- And backup anything in these two directories: /var/www and /var/cache/apt-cacher-ng... That's going to change, so if you want that data, you will have to restore it after the mounts are fixed.

In fact, backup anything you want to keep that is in zpool mypool...

Thread: booting in emergency mode... what is wrong [read-only file system?]

Thread Tools

Display

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Re: booting in emergency mode... what is wrong [read-only file system?]

Bookmarks

Bookmarks

Posting Permissions