large NFS copy locks up/hang client with large files (again) (lucid)

**ScarletPea** · May 9th, 2010

Good day all,

On lucid 10.04 client, with debian (sid) nfs server, using nfsv4, the client machine locks up/hangs on transfer of large files ~GiB+ to the nfs mount.

I am not alone, this was reported by:
http://ubuntuforums.org/showthread.p...ght=nfs+lockup
and sim.
http://ubuntuforums.org/showthread.p...ght=nfs+lockup

Same behaviour with cp or nautilus.
No problems with scp, rsync, samba(except its super slow - no not an option)

Read from the server seems ok.

With the mouse free to move and everything else stuck I thought this could be the nic module or interrupts or something. Loading the module with debug enabled showed nothing.

The client machine is Dell XPS M1710 with:
Broadcom Corporation NetXtreme BCM5752 Gigabit Ethernet adaptor.
(so using wired in case you ask), standard mtu, connecting through a fast ethernet switch to via epia-en server, nic:
VIA Technologies, Inc. VT6120/VT6121/VT6122 Gigabit Ethernet Adapter

NFS is mounted as (from fstab)

Code:

192.168.1.9:					/media/minihaha nfs4	noauto,users,rw,hard,intr 0 	0

and exported as (/etc/exports)

Code:

[/nfs4exports 192.168.1.0/24(rw,fsid=0,sync,insecure,no_subtree_check,nohide)

Given the range of machines that this is occuring on, I doubt it is one nic. It may be a gigabit thing...

Can we amass the user experience here and see if there are any common factors.

Other than that, what commands etc. can I do to really log what is going on to try and determine what is causing this.

The weird thing is it doesnt happen straight away, it takes a finite (~40s) ammount of time, which I assume is why noone notices for small files. So there could be a buffer or somthing that get full...
Anyway, help and collaboration would be appreciated.

Thank you.

Jasper

**ScarletPea** · May 10th, 2010

From http://ubuntuforums.org/showthread.p...ght=nfs+lockup,
The configurations with trouble are:

dist: Lucid,
nic: ath5k, wireless

dist:Lucid/Karmic
nic: atl1c, "Atheros Communications AR8131 Gigabit Ethernet"
nfs server: debian 5.0

dist: karmic
arch: 64bit (32bit fine aparently)

I am:
dist: Lucid
arch: 64bit
nic: tg3, Broadcom Corporation NetXtreme BCM5752 Gigabit Ethernet
nfs server: debian (sid) (NFS V4)

**Lepy** · May 13th, 2010

I've been experiencing this problem as well. NFS uploads to a Freenas seem to lockup the ubuntu systems (graphically and sometimes ssh) and cause transfer speeds to slow down dramatically or stop altogether.

A desktop and mythbuntu system that are both gigabit wired to the Freenas exhibit these symptoms. A laptop with a wireless g connection does not seem to show these symptoms (or I am too lazy to transfer a large file at g-speeds). All systems including Freenas have different nic cards/drivers.

I have noticed that transferring to the nas causes a spikey graph over gigabit while the wireless seems to be a pretty constant throughput (wireless is much slower, so makes sense). I could blame it on Freenas or hardware but 1.) a reboot of the uploading machine doesn't break any of the nas's functions or subsequent connections, it keeps chugging along like nothing ever happened for the rest of the systems 2.) someone has the same problem using openfiler.

I've never had a problem with downloads, and none of the logs seem to show any signs; however, I have experienced a dmesg "task X blocked for more than 120 seconds." as jeebustrain. I believe I have only seen this on the mythbuntu system, but it only seems to bork the systems lirc usb remote until a reboot.

I recently upgraded the Freenas to the latest nightly build, and the problem still occurs. These symptoms were present in many kernels under 9.10 and seems to have gotten worse in 10.04. On 9.10 I could transfer a few 100 gigs to the nas no problem and it would occasionally lock up. Now, it seems like it happens on many 600+ meg files.

Other than specific hardware, I'm not sure if there is much more to say that hasn't been covered in the other threads you linked. I hope we can get this figured out. This is pretty killer for the mythtv box because many auto transcodes/exports to the nas of 40+ minute shows (with commercials cut) seem to be locking the system causing recordings to not take place or mythwelcome to not function. This is only the transfer too as the transcoding is done on the local hdd.

Anyway, long post, but a big problem!

**dr_latino999** · May 18th, 2010

I am experiencing the same effect on my test setup at work, where even a 700mb ISO brings NFS and Nautilus to a screeching halt for 5-45 seconds then resumes as if nothing happened. Both client and server are 10.04.

**ScarletPea** · May 21st, 2010

Can I confirm that ALL the above problems are with Gigabit NIC's
I hae a 10/100 switch.

Where to from here? a bug report maybee, but if _anyone_ out there has another distro handy, can they check this out, so <other distro> -> deb/ubuntu nfs server, with gigabit, and without. So as to determine wherther this is a Deb/Ubuntu issue or something for upstream

Then, is this an nfs bug or a kernel bug, if kernel, what part, certainly not the nic driver.

*when* I get some of the stack of broken laptops working I can try fast-ethernet. But at this stage it looks like gigabit is the issue.

Other transfers dont seem to do this, then again, most others are much slower (aka samba, scp)

This is pretty killer for any home network, cant believe more people do not experience this... what is it with our setups?

NFS version? I use 4 on the server.
blah!

**greenhat** · May 25th, 2010

I'm having the same kind of problems with an unresponsive/semi-frozen system doing a large copy (50 files/147 Gb) from a local Lucid ext4 HD connected with a gigabit ethernet nic on a 10/100 switch to an NFS share on another Lucid box (ext4) with gigabit ethernet.

I say it's semi-frozen because it's still working according to the system monitor which updates every few minutes, but I can't really do anything else on the box because it's soooo slooooow.

**Lepy** · May 25th, 2010

The problem may be related to gigabit as I have only experienced complete lock-ups using gigabit. I don't usually do large transfers from laptops because I lack an N router, and while I have noticed nautilus lag/system slowness during G speed transfers, they always complete without a lock-up.

All computers are gigabit wired to a netgear 1000 switch, including a 10/100/g router that assigns static IPs. The heaviest used v4 NFS shares reside on a FreeNAS with raidz1 ZFS v13 pools.

I have attached a netgraph image with a wired (top) and wireless (bottom) nfs transfer. The same file was first moved from the nas and then moved back to the nas. Spikeyness may be due to the way ZFS checksums all files.

**JeffPH** · May 28th, 2010

This is happening to me to...

My System:

Intel Atom Ion with Atheros gigabit network card.
Ubuntu 10.04 LTS
2.6.32-22-generic kernel

When I use FTP from Nautilus to transfer large files (basically mkv's) from PC to NAS, NAS to NAS my desktop PC hangs after 1-2 mins of file transfer. This is usually faster about 13-15 MB/sec.

When i just use samba/nfs transfers there are no problems but the speed is slow at 2-3 MB/secs.

I have been searching the net and it seems there are a lot people having problems with this too..... Hope someone can find a solution as I am backing up 4TB and this will take days

**leespa** · June 4th, 2010

Hi All,

I am experiencing very similar NFSv4 sporadic file transfers as well. This was preceded by crappy Samba performance that could also corrupt large 10+ GB files after it dropped from 20 MB/s to 12 MB/s; Samba sometimes finished, sometimes didn't.

NFS seems to finish, but actually appears to slower overall than Samba was.

It's difficult to get a handle on actual NFS write to server speed I am getting since Nautilus reports the xfer happening in 200-300 MB 'gulps' and then stops.

Server:
Atom 330 mobo [ECS 945GCD-M (V1.0)], 4GB DDR2, Mythbuntu 9.10, NFSv4

Client:
Core 2 e8400, ubuntu 9.04, NFSv4, 4 GB DDR2

Network:
ALL Gigabit, 2 x 8 port DLink DGS-1008D

I have been banging my head against the wall RE this for quite some time. Normal networking tests are all A-OK. TCP iperf test from client to server for instance can push down 840 Mbps consistently.

I believe the NIC on the server is an Atheros as well...

*-network
description: Ethernet interface
product: Attansic Technology Corp.
vendor: Attansic Technology Corp.

NIC on client is NVIDIA
*-network
description: Ethernet interface
product: nVidia Corporation
vendor: nVidia Corporation

I'll do some more digging as well and report back here if I find anything.