Results 1 to 8 of 8

Thread: Samba server, good write but bad read performance

  1. #1
    Join Date
    Oct 2009
    Beans
    Hidden!

    Samba server, good write but bad read performance

    Running Samba on Ubuntu Server, one Win7 machine connected via GBit switch.

    What I've done:
    -set both to Jumbo frames 9k
    -fiddled with Samba socket options, to no avail
    -set some options via sysctl (those are set the same for r/w, so I guess that's not the culprit for this asynchronous behaviour)

    resulted in:
    -writes TO the share at around 60MB/s (I guess thats hd limited, just a single TB drive, no raid)
    -reads FROM the share stayed at 16-17MB/s, whatever I changed

    Any ideas to solve this would be appreciated.

  2. #2
    Join Date
    Apr 2008
    Location
    Columbia, SC
    Beans
    282

    Re: Samba server, good write but bad read performance

    First step: undo all the tweaking you already did. This hurts you at LEAST as often as it helps you; modern network stacks have yielded pretty good performance on gigabit for several years now.

    For reference, I just went through this in another Samba performance thread; these are the results from me installing Samba 3.3.6 on a bone-stock FreeBSD 7.3-R machine, and accessing it across a gigabit LAN from a bone-stock Samba 3.4.0 installation on Ubuntu Karmic.

    Code:
    me@banshee:~$ sudo mount.cifs //192.168.0.2/test /mnt -ouser=me
    Password:
    me@banshee:~$ dd if=/mnt/1G.bin bs=8M of=/dev/null
    128+0 records out
    1073741824 bytes (1.1 GB) copied, 13.7859 s, 77.9 MB/s
    Someone in that thread said that users frequently complain that GUI transfers are slower than command line transfers, so I did the same thing again with a Nautilus copy-and-paste.



    Note that in this case, we're bottlenecked by write performance on the client: there's no way to dump the file to /dev/null from Nautilus.

    Anyway, people frequently get red-herringed on this topic because it's EASY to get bottlenecked by something OTHER than the network when futzing around with gigabit LAN, and the 'net is chock-full of extremely outdated tweak guides based on TCP stacks that have been obsolete for YEARS. So the first step is, take a deep breath, undo EVERYTHING you did to the TCP stack (and to any smb/cifs connection options) on BOTH boxes... then figure out a way to avoid getting bottlenecked by your local filesystem on either end... THEN retest.

    Once you do that, we can start over.

  3. #3
    Join Date
    Oct 2007
    Beans
    668
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Samba server, good write but bad read performance

    Speculation: Could your problem have to do with the overhead of reading from an ext3 or ext4 volume to an NTFS volume?

    One problem with that hypothesis: I would think writing from NTFS to an ext3/4 volume would be slow as well, for the same reason.

    Copying from one NTFS volume to another on the same machine, I get sustained rates of 60-70 MB/s, under either Windows XP or Ubuntu 9.10.

    I'll have to do some testing later, to see whether ext3->NTFS transfers (and vice versa) are significantly different from NTFS->NTFS or ext3->ext3 transfers.
    Last edited by Objekt; March 18th, 2010 at 10:13 PM.
    Samsung Series 3 15.6" notebook (NP300-V5A-A02US)

  4. #4
    Join Date
    Apr 2008
    Location
    Columbia, SC
    Beans
    282

    Re: Samba server, good write but bad read performance

    Quote Originally Posted by Objekt View Post
    Speculation: Could your problem have to do with the overhead of reading from an ext3 or ext4 volume to an NTFS volume?
    Nope. Samba neither knows nor cares about the underlying filesystem; there's no "additional overhead" involved if one fs is different than the other.

    Raw performance of each underlying filesystem in and of itself, of course, does matter.

  5. #5
    Join Date
    Oct 2009
    Beans
    Hidden!

    Re: Samba server, good write but bad read performance

    Quote Originally Posted by jrssystemsnet View Post
    First step: undo all the tweaking you already did. This hurts you at LEAST as often as it helps you; modern network stacks have yielded pretty good performance on gigabit for several years now.

    For reference, I just went through this in another Samba performance thread; these are the results from me installing Samba 3.3.6 on a bone-stock FreeBSD 7.3-R machine, and accessing it across a gigabit LAN from a bone-stock Samba 3.4.0 installation on Ubuntu Karmic.

    Code:
    me@banshee:~$ sudo mount.cifs //192.168.0.2/test /mnt -ouser=me
    Password:
    me@banshee:~$ dd if=/mnt/1G.bin bs=8M of=/dev/null
    128+0 records out
    1073741824 bytes (1.1 GB) copied, 13.7859 s, 77.9 MB/s
    Someone in that thread said that users frequently complain that GUI transfers are slower than command line transfers, so I did the same thing again with a Nautilus copy-and-paste.



    Note that in this case, we're bottlenecked by write performance on the client: there's no way to dump the file to /dev/null from Nautilus.

    Anyway, people frequently get red-herringed on this topic because it's EASY to get bottlenecked by something OTHER than the network when futzing around with gigabit LAN, and the 'net is chock-full of extremely outdated tweak guides based on TCP stacks that have been obsolete for YEARS. So the first step is, take a deep breath, undo EVERYTHING you did to the TCP stack (and to any smb/cifs connection options) on BOTH boxes... then figure out a way to avoid getting bottlenecked by your local filesystem on either end... THEN retest.

    Once you do that, we can start over.
    ´Hmh, you don't happen to have the defaults around for:

    net.core.rmem_max
    net.core.wmem_max
    net.core.rmem_default
    net.core.wmem_default
    net.ipv4.tcp_mem
    net.ipv4.tcp_rmem
    net.ipv4.tcp_wmem



    Which is all I remember changing. The thing is why only the reads from the share would be so badly affected? I'm pretty ok with the writes though.

  6. #6
    Join Date
    Apr 2008
    Location
    Columbia, SC
    Beans
    282

    Re: Samba server, good write but bad read performance

    Quote Originally Posted by wmd View Post
    ´Hmh, you don't happen to have the defaults around for:

    net.core.rmem_max
    net.core.wmem_max
    net.core.rmem_default
    net.core.wmem_default
    net.ipv4.tcp_mem
    net.ipv4.tcp_rmem
    net.ipv4.tcp_wmem



    Which is all I remember changing. The thing is why only the reads from the share would be so badly affected? I'm pretty ok with the writes though.
    Code:
    me@banshee:~$ sudo sysctl -a | egrep "net.(core|ipv4.tcp).(r|w)?mem.*"
    error: "Invalid argument" reading key "fs.binfmt_misc.register"
    error: permission denied on key 'net.ipv4.route.flush'
    error: permission denied on key 'net.ipv6.route.flush'
    net.core.wmem_max = 131071
    net.core.rmem_max = 131071
    net.core.wmem_default = 129024
    net.core.rmem_default = 129024
    net.ipv4.tcp_mem = 767424	1023232	1534848
    net.ipv4.tcp_wmem = 4096	16384	4194304
    net.ipv4.tcp_rmem = 4096	87380	4194304
    There ya go; that's defaults for Karmic amd64.

    As for why read performance sucks worse than write... well, that could be TCP stack problem, could be a Samba buffering problem, or could be a filesystem problem. I tend to think it's a filesystem problem; are you certain the files you're reading from aren't heavily fragmented? In particular, anything you got via the transmission bittorrent client is likely to be EXTREMELY heavily fragmented, as in 80,000 or more frags in a single file... which coincides pretty nicely with the read speed you said you were getting.

    To try to troubleshoot this, since you can't read from /dev/zero over Samba, you might try this:

    Code:
    you@box:~$ sudo apt-get install pv
    pv is acronymic for pipe viewer - it shows you a nice progress bar and estimated time-to-complete for ANY operation you can pipe through it. If you didn't already know about pv... you'll be glad you do now.

    OK, now...

    Code:
    you@box:~$ sudo cat /path/to/big/file | pv > /dev/null
    you@box:~$ sudo cat /path/to/big/file | pv > /dev/null
    OK, now you have a good idea what the local filesystem speed is for a normal read from the big file... and you've immediately tried doing it with the same file again, to see if running it through the filesystem cache helped any. RIGHT NOW, while it should still be as fresh as possible in the filesystem cache, try accessing it over Samba from your other machine.

    How does that speed compare with the local speed you got by cat-ing the file to /dev/null?

  7. #7
    Join Date
    Oct 2009
    Beans
    Hidden!

    Re: Samba server, good write but bad read performance

    First of all thanks for your reply, without changing anything I installed pv (nice little tool indeed).

    Reading the file into nirvana and measuring with pv resulted in:

    Code:
    702MB 0:00:22 [31,2MB/s]
    with a improvement of 0.4MB/s from the first to the second try. Now I read the file from the client via Samba, which resulted in ~17MB/s. Since the writes to the share are full speed and unaffected I'm guessing it's some kind of Samba setting?! The files aren't fragmented at all, this is not a torrent partition which I'm reading from.

    I rebooted in the meantime, it seems whatever I changed to the tcp stack didn't stick, everything was set as the default settings yout posted, except for net.ipv4.tcp_mem which was for all values set for roughly the half. So I entered your reference values and repeated the procedure, but it didn't change anything.

    Edit: The 50-60MB/s are not correct it seems, I've got this value from a network traffic mini-app. The Windows file copy dialog shows 27MB/s which is pretty close to the directly measured read speed.
    Last edited by wmd; March 22nd, 2010 at 12:18 AM.

  8. #8
    Join Date
    Oct 2009
    Beans
    Hidden!

    Re: Samba server, good write but bad read performance

    bumpety-bump.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •