NFS not mounted at boot

[kjur]

Hi,

I have a file server on Raspberry Pi (Raspberry OS + OpenMediaVault)
My Xubuntu 20.10 laptop cannot connect automatically to RPi's NFS share at boot (entry in fstab). The laptop uses wifi, no LAN port if that makes a difference.

I can mount it manually using a command:

sudo mount -t nfs 192.168.1.140:/export/shared/ /mnt/shared/

My fstab entry looks like that:

192.168.1.140:/export/shared/ /mnt/shared/ nfs auto,defaults,nofail,noatime,nolock,tcp,rsize=8192 ,wsize=8192,actimeo=1800,timeo=14 0 0
I tried with or without various options in the line above: auto, defaults, _netdev, nfsvers=3 and retry=1 (so after 1 minute).
No luck.

Removed tailing slashes from paths in fstab (so ...:/export/shared /mnt/shared etc...) - also no luck.

Searching for NFS in dmesg givs the result below:

[ 1.839073] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.656154] FS-Cache: Netfs 'nfs' registered for caching
[ 2.775216] NFS: Registering the id_resolver key type
[ 2.801509] NFS4: Couldn't follow remote path
[ 440.374225] NFS4: Couldn't follow remote path
The last line at 440 comes up at the time of manual mounting using the command from the top of this post.

showmount -e 192.168.1.140 give a result:

Export list for 192.168.1.140:
/export 192.168.1.0/24
/export/shared 192.168.1.0/24

Tried autofs which was doing the job on mounting at boot, but unmounting at shutting down or restart took a few minutes, so uninstalled it.

My 2nd Raspberry Pi (Raspberry OS + moode) can connect to NFS at boot with no problem out of the box.

Any idea how to fix it?

[TheFu]

Try:

sudo mkdir /mnt/shared
fstab:

192.168.1.140:/export/shared /mnt/shared nfs proto=tcp,async 0 0

Bogus options cause lines to be ignored. Only use valid options, please. NFS servers have negotiated the correct rsize and wsize parameters for years. Wifi that is a poor connection will always have issues. Avoid wifi on any NFS server and as much as possible on clients unless you 100% KNOW the client wifi connection is better than good.

I only use aufofs for NFS and USB devices. That's because they may not always be available. autofs should automatically remove any unused mount within about 5 minutes. The only reason it would delay a shutdown would be if there are files still open on the NFS share from the client. I.e. the NFS file system is busy still. Besides that, autofs performance is just like any fstab or manual mount.

[kjur]

/mnt/shared already exists. forgot to mention that in the original post.

Your fstab line also doesn't work.

In meantime I've noticed that /mnt/share ownership was nouser:nogroup. Changed it to root:root (same as /mnt/win10 where I mount win10 partition).
It didn't change anything unfortunately.

RPi server is connected to router via 1Gbps LAN. A laptop is via wifi.

No files were opened when I used autofs. Everything was closed. Just bare system.


Any ideas?

[TheFu]

What do you mean "my fstab line also doesn't work?" Be explicitly clear, please.

Start by showing the /etc/exports, exactly.
Then the /etc/fstab, exactly.
Finally, which file system is on the server-side shared directories?
Then the commands you run when you modify each - on each system. Please be clear. Just changing the file is NOT sufficient.

Having nested NFS shares is a bad idea. When you do that, I think the fsid= option is needed to keep the share from becoming confused.

[kjur]

Hi again.

By saying 'your fstab line also doesn't work' I meant that changing fstab line to the one from your post didn;t help. Still no automount.

/etc/exports doesn't exist on a client side.
/etc/exports from a server (Raspberry Pi OS based on Debian Buster + OpenMediaVault NAS software):

# This file is auto-generated by openmediavault (https://www.openmediavault.org)
# WARNING: Do not edit this file, your changes will get lost.

# /etc/exports: the access control list for filesystems which may be exported
# to NFS clients. See exports(5).
/export/shared 192.168.1.0/24(fsid=1,rw,subtree_check,insecure)

# NFSv4 - pseudo filesystem root
/export 192.168.1.0/24(ro,fsid=0,root_squash,no_subtree_check,hide)


/etc/fstab from a client (Xubuntu 20.10):

# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/nvme0n1p7 during installation
UUID=a1616d5e-3728-4549-940b-f072a4272256 / ext4 errors=remount-ro 0 1
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=94B7-D6DE /boot/efi vfat umask=0077 0 1
# /home was on /dev/nvme0n1p8 during installation
UUID=5d8be7b1-8a59-453e-8c9e-8fb0ed344593 /home ext4 defaults 0 2
/swapfile none swap sw 0 0
/dev/disk/by-uuid/A62493FD2493CEA5 /mnt/win10 auto nosuid,nodev,nofail 0 0

# rpi4 nfs shared
192.168.1.140:/export/shared /mnt/shared nfs proto=tcp,async 0 0



The server is RPi4 + Raspberry Pi OS (previously called Raspbian) based on Debian Buster + OpenMediaVault NAS software. All up-to-date.

Not sure what you mean about the command when I modify something.
I don't modify anything on the server side.
I previously tried different options in the fstab line for NFS entry, but now I have your line. Also tried to add _netdev, but didn't help so it is now as above.

I don't know what you mean 'nested NFS shares'. I don't thing it's nested. It is on a USB3 external drive connected to Raspberry Pi server. Other RPi's can connect to it on boot with no problem. I also can manually mount it on Xubuntu. Just mount on boot from fstab doesn't work.

I hope that's clear now.

Thanks for your help.
Greg

[TheFu]

192.168.1.140:/export
and
192.168.1.140:/export/shared
is nested. One is above the other. Nested. Disable one for now. Might change the order too in the exports file. It shouldn't matter, but who knows?
I assume the IP address for the NFS server never changes?

After you modify the fstab, did you run sudo mount -a? What are the errors, if any? Look in the system logs and dmesg. There isn't any need to reboot for this stuff until everything is working. The reboot would be the final test.

BTW, "automount" is a very specific term that has NOTHING to do with the fstab. I'd assumed you were using amd - the automount daemon tool when you said that. You want it to mount at boot automatically, but aren't using amd.

On the server, please should the mounted storage for:

/export/shared
/export
I'd use df -Th and remove all the junk except those areas and the header. Thanks.

[SeijiSensei]

My fstab entry looks like that:

192.168.1.140:/export/shared/ /mnt/shared/ nfs auto,defaults,nofail,noatime,nolock,tcp,rsize=8192 ,wsize=8192,actimeo=1800,timeo=14 0 0

I'd get rid of the rsize and wsize parameters, or increase them considerably to something like 65536. The value of 8192 was established back when most networks ran 10MBit Ethernet.

[kjur]

is nested. One is above the other. Nested. Disable one for now. Might change the order too in the exports file. It shouldn't matter, but who knows?
I assume the IP address for the NFS server never changes?

After you modify the fstab, did you run sudo mount -a? What are the errors, if any? Look in the system logs and dmesg. There isn't any need to reboot for this stuff until everything is working. The reboot would be the final test.

BTW, "automount" is a very specific term that has NOTHING to do with the fstab. I'd assumed you were using amd - the automount daemon tool when you said that. You want it to mount at boot automatically, but aren't using amd.

On the server, please should the mounted storage for:

/export/shared
/export


Sorry, been busy with other things.
I understand now what nested means in that case. I guess the extra /export/ part in the NFS path is added by default in NFS v3. Removed in v4 if I'm correct.

A server IP never changes and it's always on (definitely it is on when booting a laptop to automount the share.

I was actually rebooting the laptop completely. Didn't use sudo mount -a.

I didn't hear about the amd. I thought happens automatically if there is an entry in /etc/fstab. I'll look into it.
I used autofs before, but as mentioned unmounting was taking ages.

Will modify export entries as you suggested and let you know.


@SeijiSensei - rsize and wsize are removed now as per instructions of TheFu in other posts. Still no luck

Funny enough another RPi with Raspberry Pi OS (based on Debian Buster) mounts the share straight away with an entry:


192.168.1.140:/export/shared/mp3 /mnt/mp3 nfs auto,_netdev 0 0

[TheFu]

Did you change the exports file?

Perhaps I'm too close now. Some fresh eyes needed.

[kjur]

Hi, sorry, not yet. Not access to this computer at the moment.
However look at my export file one of the posts above. /export/shared is listed first, then /export. So what to change? fsid=1 and fsid=0?

[TheFu]

192.168.1.140:/export/shared/mp3
shouldn't work. It isn't a valid export.

[kjur]

It shouldn't, but it definitely does.
Obviously the /mp3 directory exists on a network drive.

EDIT: On my laptop the command sudo mount -a works. I understand it uses all the entries from fstab, but for some reason the same entires on boot don't work.
Perhaps it's because the network is not ready yet at the time of mounting? But in theory parameter _netdev should solve it (a system waits with mounting until a network is ready).

EDIT2: Check my boot process and I have services: -.mount, mount-win10.mount, boot-efi.mount and home.mount, then nfs-client.target, then some other services and NetworkManager.service and networkd-dispatcher.service, network.target and network-online.target much later.
Have a look here: https://svgur.com/s/UVG

Do you think this may be a problem?


EDIT3: found another similar problem here: https://ubuntuforums.org/showthread.php?t=2450781
and tried x-systemd.after=network-online.target parameter, but didn't work.
Also found on other forums that disabling ipv6 may help, but in my case it didn't help too.


EDIT4: seems like it was a network issue (network not ready). Added a Systemd service with a small delay and mount. 3 sec delay was too short. It was reporting as Network not reachable. Sleep 10 does the job. It mounts NFS share as intended.

[scottbomb]

And as you've found, systemd doesn't give a damn if the network is ready or not, it just runs through fstab with whatever options are there and if it doesn't mount (because the system doesn't have an IP address yet), no matter. This seems to me to be a serious bug. Having to create a service that adds a delay and a mount command is a workaround, one it looks like I'll have to try as well until this is actually fixed.

[TheFu]

https://www.freedesktop.org/software/systemd/man/systemd.mount.html has the documentation for systemd.mount.

In theory:


fstab
....
The NFS mount option bg for NFS background mounts as documented in nfs(5) is detected by systemd-fstab-generator
and the options are transformed so that systemd fulfills the job-control implications of that option. Specifically
systemd-fstab-generator acts as though "x-systemd.mount-timeout=infinity,retry=10000" was prepended to the option
list, and "fg,nofail" was appended. Depending on specific requirements, it may be appropriate to provide some of these
options explicitly, or to make use of the "x-systemd.automount" option described below instead of using "bg".

_netdev

Normally the file system type is used to determine if a mount is a "network mount", i.e. if it should only be started after
the network is available. Using this option overrides this detection and specifies that the mount requires network.

Network mount units are ordered between remote-fs-pre.target and remote-fs.target, instead of local-fs-pre.target and
local-fs.target. They also pull in network-online.target and are ordered after it and network.target.
...

Default Dependencies

The following dependencies are added unless DefaultDependencies=no is set:
...
Network mount units automatically acquire After= dependencies on remote-fs-pre.target, network.target and
network-online.target, and gain a Before= dependency on remote-fs.target unless nofail mount option is set. Towards the
latter a Wants= unit is added as well.

[scottbomb]

God bless you, TheFu. I've tried everything. _netdev is mentioned in a lot of places on the 'net but has never worked.

Tonight I read man nfs and thought I had found the answer with options: bg,timeo=150, _netdev,nofail ... and it STILL didn't work. I'd grep nfs syslog and get "mount.nfs: Network is unreachable" shown only once, indicating it didn't retry after the first fail. A reading of the man page suggests it would retry every 15 seconds with those options but nope. Once is all. Network not up yet, systemd says screw it.

What finally made it work is this: x-systemd.automount,nofail

Those are my ONLY options now for 3 NFS mounts. And it finally freaking worked.

It's too bad I didn't see that option mentioned anywhere in the man file, which is irritating. I really wish I could find some thorough, offical, documentation on systemd as I find myself learning it bit-by-bit in my troubleshooting at home and on the servers I manage at work. Combing through forums and posting like a madman eventually gets me what I expect the OS to do but I'll be damned if I can find everything documented the way it actually works. Your description of the services is helpful too, but I do need to read up on all this wants vs. requires vs. after syntax that debian makes a half-assed approach at describing but not as well as you do here. Now, finally, every time this living room PC reboots, I don't have to fire up a command line and run mount -a just so we can access the NFS server. Thanks again.

[TheFu]

God bless you, TheFu. I've tried everything. _netdev is mentioned in a lot of places on the 'net but has never worked.


I prefer to thank Bob, my imaginary friend. He's always right.
Rather than searching the internet for answers from people who often know know anymore than we do, best to read the manpages ALREADY on your computer.

I've bypassed all these issues by using autofs the last 25+ yrs, though it was called automountd in the 1990s. Any storage that isn't internally gets mounted via the autofs settings here - that's NFS, USB, eSATA storage. That way the system doesn't get stuck and, more importantly, systemd-mount isn't used.

Systemd-mount took over the /etc/fstab file and clearly didn't honor the prior settings correctly for many options. It happened in Ubuntu sometime after 16.04 was released with the update pushed out to 16.04 systems.

Always remember that systemd documentation is "aspirational" - it describes what they hope to achieve at some time in the future, not what actually works today. Because of this, I try to avoid it - all of it. systemd-timesync, systemd-resolved, systemd-mount, systemd-{whatever} as much as possible. Usually there are excellent, simple, options. Chrony, /etc/resolv.conf (manually edited), autofs, are examples, respectively.

I haven't mounted NFS storage using the /etc/fstab in at least a decade, but NFS really was the first use for the amd in the 1990s. Whenever people in these forums use the term "automount", it is confusing because 99% of the time, they aren't really talking about the automountd process. They are talking about static mounts listed in /etc/fstab.

[scottbomb]

Duplicate post deleted. Please see below.

[scottbomb]

Well, the odd thing is, since it worked on one PC, I changed the fstab on another. The first PC automounts the NFS shares and the second doesn't (although It did with the previous settings). They're both running the exact same Kubuntu 20.04 too. This is most infuriating.

I say automount because I got used to using fstab to automatically mount hard drives when I first learned to use Linux, when 11.04 was the current release and the systemd cancer hadn't reared its ugly head. Things just worked then but no, some folks decided they had to change it for some reason and now we have this more complicated setup that doesn't work as documented. I'll read up on systemd-mount since we're stuck with it. I just wonder why they even kept fstab if systemd-mount was supposed to take over. It makes things very confusing for those of us who are used to doing things a certain way and the tools that we expect to work are still there. It's like Ford giving me a hand throttle to control my speed but leaving the gas pedal in place, but disconnected.

So I read up on autofs and found the man files to be a bit confusing. I found a tutorial that seems pretty straightforward here: https://websiteforstudents.com/how-to-mount-nfs-filesystems-with-autofs-on-ubuntu-20-04/

I created /etc/auto.master and populated the last two lines with the following, which refer to folders that exist in /media with 777 permissions:


/media/Television /etc/auto.nfs
/media/Movies /etc/auto.nfs


I then created /etc/auto.nfs and populated it according to the tutorial with:



/media/Television -fstype=nfs4 192.168.1.2:/media/Romulus/Television
/media/Movies -fstype=nfs4 192.168.1.2:/media/Kronos/Movies


Then I did



sudo systemctl restart autofs.service


I navigate to /media and check my Television and Movie directories and they're empty when I expect to see a listing of the same contents as are on the NFS server.

What in the world am I missing?

[scottbomb]

I also tried these:

for /etc/auto.master


/media /etc/auto.nfs



for /etc/auto.nfs


/media -fstype=nfs4,rw 192.168.1.2:/media/Romulus/Television
/media -fstype=nfs4,rw 192.168.1.2:/media/Kronos/Movies


But of course it still doesn't work. I'm continuing trying different combinations as I search for more tutorials on this.

EDIT:

OK, I think I have this working now.
Content of /etc/auto.master (last line):


/media /etc/auto.nfs


Apparently all it cares about here is where to find the mount points, which are both in /media.

Content of /etc/auto.nfs


Television -fstype=nfs4,rw 192.168.1.2:/media/Romulus/Television
Movies -fstype=nfs4,rw 192.168.1.2:/media/Kronos/Movies


This file defines each mount point within /media.

I'll do a reboot and see if it holds.

EDIT: After the reboot, my other hard drives mounted by fstab that were also in /media were gone. It seems autofs now has complete control over /media. I made some changes and created a new folder, /nfs to use for these mounts instead of /media.

I rebooted again. I navigate to /nfs and there are no folders. BUT, if I do cd Television. Voila, it appears! Same with Movies. ***? So these aren't mounted until I try to navigate to them. The answer to this was to put options in /etc/auto.master with of "--timeout 0 browse". It's a weird syntax they expect, which isn't consistent with anything else in Linux, but there are to be no commas in the options. --timeout 0 prevents autofs from UN-mounting the remote folder (like I ever want that to happen) and browse makes the mounted folders appear before actually accessing them (so they're no longer invisible). They WERE, however, still invisible in the file manager (Dolphin) until I made symbolic links to them in my home directory so that when I open Dolphin, it resolves those sybolic links to their respective places in /nfs.

So the last line in /etc/auto.master is


/nfs /etc/auto.nfs --timeout 0 browse



What a ****ing mess, but at least it seems to be working now on both PCs. I liked it much better when it was just a one-liner in fstab but thanks to systemd, it's a lot more complicated now. I suppose in another 10 years we'll have more poorly-documented changes as they move from systemd to systemf or whatever since we can never seem to leave well enough ALONE.