Results 1 to 9 of 9

Thread: Unable to exploit network speed potential

  1. #1
    Join Date
    Nov 2019
    Beans
    5

    Unable to exploit network speed potential

    Hi,

    I have a server and a NAS connected to the same 10 Gbit network
    • MacMini with Sonnet 10Gbit TB2 adapter, running Ubuntu 19.10 server
    • Synology NAS DiskStation DS1819+ with 10Gbit NIC, running DMS DSM 6.2.2-24922 Update 3
    • UniFi Switch XG 16
    • Both devices are connected directly to the switch with Cat 6 ethernet cables (< 3m)


    I have ran an iPerf3 test between the two devices, and I can't seem to reach more than ~ 5 Gbit/s, so there is also a network issue I need to address, but the biggest problem is that I'm far from exploiting even the 5 Gbit/s when transferring files. I'm quite new to Ubuntu server, and I'm not able to identify whether this is only a network issue or something else (maybe network drive mounting?).

    When I do a scp between the server and the NAS (both directions), I don't achieve more than 80 MB/s.

    Here is the output from my /etc/netplan/*.yaml
    Code:
    # This file is generated from information provided by# the datasource.  Changes to it will not persist across an instance.
    # To disable cloud-init's network configuration capabilities, write a file
    # /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
    # network: {config: disabled}
    network:
        ethernets:
            ens1:
                dhcp4: true
                mtu: 9000
        version: 2

    This is the output from ifconfig ens1
    Code:
    ens1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
            inet 192.168.5.46  netmask 255.255.255.0  broadcast 192.168.5.255
            inet6 fe80::230:93ff:fe11:a8b  prefixlen 64  scopeid 0x20<link>
            ether 00:30:93:11:0a:8b  txqueuelen 1000  (Ethernet)
            RX packets 163190936  bytes 211725794759 (211.7 GB)
            RX errors 0  dropped 89  overruns 0  frame 0
            TX packets 118980097  bytes 195761653265 (195.7 GB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    This is the mount I have done in /etc/fstab
    Code:
    //192.168.5.45/Media\040Vault /media/gringotts nfs credentials=/home/ole/.smbcredentials,vers=3.0,rsize=1048576,wsize=1048576,uid=1000,gid=1000 0 0
    MTU is also sat to 9000 on the Synology network interface


    Please let me know if there is anything else I can provide to make it easier to troubleshoot, and also sorry if I'm just overlooking something really basic. I'm new to this whole Ubuntu Server and 10Gbit thing.

    Thank you in advance.

  2. #2
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Unable to exploit network speed potential

    First, 10 Gbps is 1.25 GB/s. Ensure you convert to the same units. So, 80 MBps == 640 Mbps.

    The slowest part of the total transfer setup will limit everything else. Storage is usually the slowest part, not the network.

    scp is a slow protocol. sftp is usually faster. Both require support from the CPU on both sides to encrypt/decrypt the transferred bytes. There are different encryption ciphers available, many faster, for less security.
    If you are transferring multiple files, rsync is often much more convenient. It uses ssh by default and if there is any issue, re-running the exact command should quickly skip over files already transferred or send only the updated parts, as necessary.

    Check the performance of the storage.

    The mount options are wrong. NFSv3 doesn't need any credentials, no uid, no gid and the size settings prevent the client and server from negotiating the best options for the specific connection. Spaces in the mounts should be avoided. Don't mix NFS options and CIFS options.

    Try this:
    Code:
    192.168.5.45:/Media\040Vault      /media/gringotts     nfs      vers=3,async     0      0
    With 1 Gbps and faster connections, it might be wise to force tcp to be used, not udp (default) for NFS. UDP can lose data. With tcp and a "soft" mount, performance might improve, but it will always be limited by the actual capabilities of the physical disks and the NAS disk subsystem.

    For example, my NFS server is a Pentium G3258 CPU with 16.04 server.
    A client with a new Ryzen 2600, both connected over GigE.
    Code:
    $ dd if=/dev/zero of=./zeros bs=4M count=500
    500+0 records in
    500+0 records out
    2097152000 bytes (2.1 GB, 2.0 GiB) copied, 30.7922 s, 68.1 MB/s
    2nd run:
    2097152000 bytes (2.1 GB, 2.0 GiB) copied, 31.629 s, 66.3 MB/s
    The NFS mount options are:
    Code:
    $ mount |grep DD
    /etc/auto.nfs on /DD type autofs (rw,relatime,fd=6,pgrp=3114,timeout=60,minproto=5,maxproto=5,direct,pipe_ino=38278)
    The mount configuration actually uses autofs:
    Code:
    /DD      -fstype=nfs,hard,intr,rw,async,vers=3  istar:/DD
    That's 544 Mbps for the first run over NFS that was able to use the disk's cache.

    Running the same command on the NFS server,
    Code:
    2097152000 bytes (2.1 GB, 2.0 GiB) copied, 15.1733 s, 138 MB/s
    About 1104 Mbps. There's no way for NFS to be that fast. Can't be faster than the physical storage, right?

    Also, reading and writing all zeros is a contrived test. If you really want to know storage performance, use a better tool, like bonnie++.

    Few people have 10 Gbps networks away from work. In a work environment, servers should be running 18.04 LTS and only have non-LTS releases inside lab environments. IMHO.

    While you run the NFS or scp/sftp/ssh/rsync tests, watch the CPU use. If a slow CPU gets busy, then cheaper network devices will be stunted. Usually, Intel NICs reduce the amount of CPU processing needed for network transfers. Some other, cheaper, NICs use the CPU much more and cause many more interruptes to transfer the same bytes. There is always a trade off.

  3. #3
    Join Date
    Nov 2019
    Beans
    5

    Re: Unable to exploit network speed potential

    TheFu,

    thank you very much for the swift response and insightful feedback. I really appreciate it!

    I guess I have messed up the mount settings a bit. I have a username and password set for the mount volume, so then I guess the NFS part of the mounting is incorrect.

    What will be the correct CIFS options?

    Thanks!

  4. #4
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Unable to exploit network speed potential

    Quote Originally Posted by oleronning2 View Post
    What will be the correct CIFS options?
    I avoid CIFS because it doesn't support native permissions (chmod/chown don't work!), usually has less performance, and there are many added complexities. If you like, there are lots of CIFS mount examples in these forums. My attempts to be helpful in those haven't always worked.

  5. #5
    Join Date
    Nov 2019
    Beans
    5

    Re: Unable to exploit network speed potential

    Thank you for the response! I'm sorry I'm a little slow on this stuff, but I'm learning loads.

    So, I have volume on my NAS that only one specific user is supposed to have access to. Is it possible to mount this with NFS at all? I like the idea of the performance gain with NFS, so I'm happy to adjust my setup based on the NFS requirements.

    Thank you

  6. #6
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Unable to exploit network speed potential

    I don't know.

    I've never used any low-end NAS devices. With devices that support NFS properly, only NFSv4 uses Kerberos for either server-to-server or user-to-server authentication. NFSv3 uses strictly IP and export configuration to manage security.

    How much a Synology NAS properly supports any specific standard is something only Synology can answer. With a normal NFSv3 server, no credentials are needed if the server is configured to allow a specific hostname/IP.

    In short, try my fstab config line. Either it will work, which is what I expect, or there will be an error output to the command sudo mount -a or in the logs if you just have the systemd subsystem restart storage mounts.

    Just noticed this:
    Code:
            RX errors 0  dropped 89  overruns 0  frame 0
    Dropped frames aren't good. Definitely need to use TCP for the NAS connection. I'd have to look it up just like you do. Oh, looked it up, the option to be added is proto=tcp. Reran the dd command after restarting the autofs service,
    Code:
    2097152000 bytes (2.1 GB, 2.0 GiB) copied, 30.0127 s, 69.9 MB/s
    Doesn't seem to have degraded NFS performance here at all.

    Also, using DHCP is a bad idea for non-portable systems, unless you've setup DHCP reservations so the clients effectively get static IPs. IHMO. At home, even my portable devices have DHCP reservations, so I can count on their IP address always. Good network performance often comes back to static IP addresses, so we don't have to wait for protocol timeouts as different name resolution techniques are attempted.

    And just for completeness, https://help.ubuntu.com/community/SettingUpNFSHowTo is the community created NFS how-to which addresses some edge situations.
    Last edited by TheFu; 4 Weeks Ago at 10:05 PM.

  7. #7
    Join Date
    Nov 2019
    Beans
    5

    Re: Unable to exploit network speed potential

    So, I was able to set up NFS 4.1 on the Synology NAS with proper access control.

    So, next to the mounting. This is now the contents of /etc/fstab
    Code:
    # Media Vault on Gringotts192.168.5.45:/volume1/media-vault      /media/gringotts     nfs      vers=4.1,async     0      0
    I'm able to mount the volume, but there is something not right with the permissions.

    ls -al in media gives
    Code:
    d---------  1 ole  ole   138 Nov  7 16:36 gringotts/
    and I'm not able to cd into the folder.

  8. #8
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Unable to exploit network speed potential

    You need to understand Unix file and directory permissions. That is a huge plus with NFS. Use the normal commands - chown and chmod - just like the disk is local.
    If you want strict access just for 1 user, then chmod 700 /media/gringotts is what you want. This can be run only by the owner, ole, of the directory .... On my NFS servers, I don't allow a client root/sudo to have any special permissions, so sudo chmod won't work.
    https://help.ubuntu.com/community/FilePermissions is a beginner's guide for Unix permissions.
    Last edited by TheFu; 4 Weeks Ago at 10:25 PM.

  9. #9
    Join Date
    Nov 2019
    Beans
    5

    Re: Unable to exploit network speed potential

    I've now been able to mount everything correctly, and I've ruled out the mounting part of the equation.

    Next steps will be to do some more research on the transfer speeds, including checking storage and transfer method performance.

    Thank you for the support!

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •