Results 1 to 10 of 10

Thread: Network Bottleneck Check

  1. #1
    Join Date
    Feb 2012
    Beans
    8

    Network Bottleneck Check

    Hi all,

    I've set up a nice Ubuntu+KVM based DRBD cluster and everything seems to fine except for when I try to view one of the websites running inside one of my virtual machines - it's really slow... I have LXDE installed on the host and have tested viewing the VM websites there and they are extremely fast - only about 2-3 seconds for a fairly complex page (compared to about 25 seconds when I view the same site externally). Because of this, I don't think it's a problem with Apache or KVM disk I/O.

    I suspect that the problem is with the new switch I put the servers behind, but I can't be 100% sure. What tools should I use to detect where a network bottleneck is? I'm toying around with iperf at the moment, but my boss is looking to me for an absolute as to what and why the connection is slow.

    Thanks

  2. #2
    Join Date
    Mar 2010
    Location
    India
    Beans
    7,377
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Network Bottleneck Check

    I don't think I can help you with this kind of setup but providing a few more details may help those who can. Details like -

    • What kind of networking there is between the host and the VM - NAT/Bridged... etc? I'm asking this with the only experience of virtualbox and vmware. I don't know if this kind complexity even exists or not with KVM.
    • Please confirm whether the host is connected to the router via cable or wifi ? I'm assuming it must be cable for speed and less-complexity benefits.
    • The outputs of following from host -

    Code:
    ifconfig -a
    nm-tool
    Plus any extra details you think noteworthy.
    Varun
    Help others by marking threads as [SOLVED], if they are. (See how)
    Wireless Script | Use Code Tags

  3. #3
    Join Date
    Feb 2012
    Beans
    8

    Re: Network Bottleneck Check

    • The machine is connected through a bridged interface.
    • Connected to network by cable.

    Code:
    #ifconfig -a (from within the vm)
    eth0 Link encap:Ethernet HWaddr 52:54:00:e7:21:f5
    inet addr:192.168.10.153 Bcast:192.168.10.255 Mask:255.255.255.0
    inet6 addr: fe80::5054:ff:fee7:21f5/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
    RX packets:2697007 errors:0 dropped:0 overruns:0 frame:0
    TX packets:1082547 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000
    RX bytes:36793743884 (36.7 GB) TX bytes:5048650580 (5.0 GB)
    lo Link encap:Local Loopback
    inet addr:127.0.0.1 Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING MTU:16436 Metric:1
    RX packets:28338 errors:0 dropped:0 overruns:0 frame:0
    TX packets:28338 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0
    RX bytes:2129272 (2.1 MB) TX bytes:2129272 (2.1 MB)
    Not sure if I should really install network-manager on this system, so I can't check nm-tool output right now.

    I've also installed varnish on it to see if it was just Apache being slow, but even with the varnish cache the images loaded slowly. I should mention that everything is fast except for images - they take ages...

    I'm going to try VirtualBox and see how it goes, maybe it's some strange bug with KVM networking...

    EDIT:
    I've replaced the ethernet cables with brand new ones and tested connections with the server being behind the hub vs directly connected to the firewall in front of it with no difference- the FreeBSD server is fast no matter where it's placed on the network. I now believe it must have something to do with KVM.
    Should this be moved to the Server category instead of Networking?
    Last edited by sandyd; November 16th, 2012 at 04:39 PM. Reason: kode tags

  4. #4
    Join Date
    Mar 2010
    Location
    India
    Beans
    7,377
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Network Bottleneck Check

    Quote Originally Posted by monarckco View Post
    #ifconfig -a (from within the vm)
    Code:
    eth0 Link encap:Ethernet HWaddr 52:54:00:e7:21:f5
    inet addr:192.168.10.153 Bcast:192.168.10.255 Mask:255.255.255.0
    inet6 addr: fe80::5054:ff:fee7:21f5/64 Scope:Link
    Actually I was interested in the ifconfig output of host, not the vm, because of this -
    Quote Originally Posted by monarckco View Post
    I have LXDE installed on the host and have tested viewing the VM websites there and they are extremely fast - only about 2-3 seconds for a fairly complex page (compared to about 25 seconds when I view the same site externally).
    Means, you get the fast loading speed when you view the pages on the host machine, right ? Please correct me if I'm misunderstanding you.

    If that's true, please post the output of ifconfig from host also. And it's okay if nm is not installed. Just tell us the gateway and DNS you are using on the host and the remote machines.

    If you think it can be a DNS lookup delay, you may try disabling IPv6 on the interfaces (both on host and guest). Although I don't think it is going to help since you said everything loads fine except images.

    Also, please make it a habit to wrap the codes between 'Code' tags, like I have done above in your quoted text. It can be auto-generated by clicking on # at the top of the edit box in which you create new posts, then copy-paste the code text in-between those tags. It preserves the output formatting and makes it more readable.

    Quote Originally Posted by monarckco View Post
    Should this be moved to the Server category instead of Networking?
    By the description of problem, sounds reasonable to me.
    Varun
    Help others by marking threads as [SOLVED], if they are. (See how)
    Wireless Script | Use Code Tags

  5. #5
    Join Date
    Feb 2012
    Beans
    8

    Re: Network Bottleneck Check

    Alright, well I've done some switching around with the cables and it seems as if the firewall is slowing me down. When I plug the servers directly into the switch that comes before the firewall, everything is much faster. Perhaps the firewall doesn't like my network settings? I don't have any settings in the firewall to block this server's IP or anything, but it is definitely slowing down the images (html, css, etc. are all done in an instant). The strange thing is that the FreeBSD machine is behind the same firewall, but is not affected in the least.

    Here's the hosts output:

    Code:
     
    #ifconfig -a
    br0       Link encap:Ethernet  HWaddr 5c:f3:fc:f2:35:91
              inet addr:192.168.10.151  Bcast:192.168.10.255  Mask:255.255.255.0
              inet6 addr: fe80::5ef3:fcff:fef2:3591/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:1978647 errors:0 dropped:0 overruns:0 frame:0
              TX packets:1279774 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:817184548 (817.1 MB)  TX bytes:40806693222 (40.8 GB)
     
    eth0      Link encap:Ethernet  HWaddr 5c:f3:fc:f2:35:91
              inet addr:192.168.10.150  Bcast:192.168.10.255  Mask:255.255.255.0
              inet6 addr: fe80::5ef3:fcff:fef2:3591/64 Scope:Link
              UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
              RX packets:9131368 errors:0 dropped:0 overruns:0 frame:0
              TX packets:13704705 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:5640014748 (5.6 GB)  TX bytes:15492416713 (15.4 GB)
              Interrupt:17 Memory:91a80000-91aa0000
     
    eth1      Link encap:Ethernet  HWaddr 5c:f3:fc:f2:35:92
              inet addr:10.128.1.1  Bcast:10.128.255.255  Mask:255.255.0.0
              inet6 addr: fe80::5ef3:fcff:fef2:3592/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:10204713 errors:0 dropped:0 overruns:0 frame:0
              TX packets:39988333 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:1464121436 (1.4 GB)  TX bytes:47578413107 (47.5 GB)
              Interrupt:19 Memory:91980000-919a0000
     
    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:76978 errors:0 dropped:0 overruns:0 frame:0
              TX packets:76978 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:18402880 (18.4 MB)  TX bytes:18402880 (18.4 MB)
     
    usb0      Link encap:Ethernet  HWaddr 5e:f3:fc:f2:35:94
              BROADCAST MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
     
    virbr0    Link encap:Ethernet  HWaddr f6:5b:8f:0c:da:d4
              inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
              UP BROADCAST MULTICAST  MTU:1500  Metric:1
              RX packets:0 errors:0 dropped:0 overruns:0 frame:0
              TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
     
    vnet0     Link encap:Ethernet  HWaddr fe:54:00:e7:21:f5
              inet6 addr: fe80::fc54:ff:fee7:21f5/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:1178577 errors:0 dropped:0 overruns:0 frame:0
              TX packets:2892334 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:500
              RX bytes:5266669650 (5.2 GB)  TX bytes:36853993657 (36.8 GB)
     
    vnet1     Link encap:Ethernet  HWaddr fe:54:00:7d:de:7f
              inet6 addr: fe80::fc54:ff:fe7d:de7f/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:16047 errors:0 dropped:0 overruns:0 frame:0
              TX packets:36475 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:500
              RX bytes:13662918 (13.6 MB)  TX bytes:9032684 (9.0 MB)
     
    vnet2     Link encap:Ethernet  HWaddr fe:54:00:d5:53:8d
              inet6 addr: fe80::fc54:ff:fed5:538d/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:47974 errors:0 dropped:0 overruns:0 frame:0
              TX packets:89944 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:500
              RX bytes:69686551 (69.6 MB)  TX bytes:38172335 (38.1 MB)
     
    vnet3     Link encap:Ethernet  HWaddr fe:54:00:20:4e:1e
              inet6 addr: fe80::fc54:ff:fe20:4e1e/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:39226 errors:0 dropped:0 overruns:0 frame:0
              TX packets:56638 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:500
              RX bytes:22966613 (22.9 MB)  TX bytes:14984757 (14.9 MB)
    • We have two servers set up as DNS servers, 192.168.10.231 and 192.168.10.153. We also use google as a last resort, 8.8.8.8.
    I checked my gateway settings and it seems that I did make a mistake there. I fixed it and reset networking, but it still seems too slow. Is it possible that the firewall just doesn't like my networking setup or something?

  6. #6
    Join Date
    Jan 2008
    Location
    Manchester UK
    Beans
    13,658
    Distro
    Ubuntu

    Re: Network Bottleneck Check

    Thread moved to Server Platforms.

  7. #7
    Join Date
    Mar 2010
    Location
    India
    Beans
    7,377
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Network Bottleneck Check

    Quote Originally Posted by monarckco View Post
    The strange thing is that the FreeBSD machine is behind the same firewall, but is not affected in the least.
    Is it a physical server or a vm ?

    Quote Originally Posted by monarckco View Post
    Perhaps the firewall doesn't like my network settings?
    ..
    ..
    .. Is it possible that the firewall just doesn't like my networking setup or something?
    Is it a dedicated hardware firewall ? It may be interesting to know what model it is.

    I won't pretend to know a lot about firewalls and stuff.. but with my very limited experience with both software (isa 2004) and hardware (sonicwall nsa 2400) firewalls, I can say that complex set of rules or extensive port/protocol filtering (in short any kind of complexity) can sometimes play wonderful games with you. In such situations, the dedicated hardware firewall, IMHO, is easier to analyze and troubleshoot.

    You may try to compare and single-out the basic differences between your BSD server packets and the problematic one's. Like MTU, txqueuelen, etc.

    I'd also like to mention that in last three days, I've noticed at least three different threads discussing network speed problems with VMs. So I suspect it may just be a vm factor - especially if your BSD one (which you say is working fine) is a physical machine and this current host has installed any recent updates. However, all the three threads I noticed were about 12.10 server and vmware/virtualbox.


    With this I'll leave this thread for people who really understand and have experience with this kind of setup, and know what they are talking about

    Good luck !
    Varun
    Help others by marking threads as [SOLVED], if they are. (See how)
    Wireless Script | Use Code Tags

  8. #8
    Join Date
    Feb 2012
    Beans
    8

    Re: Network Bottleneck Check

    Well, neither? It's a jail inside of FreeBSD 9.

    The firewall is a dedicated hardware firewall, Fortinet 50B I believe. I've been in contact with them trying to sort this out and they said that they've seen problems like this before so maybe its a step in the right direction.

    I have a feeling that it's not the firewall though, because it's only the images on the initial connection that are slow. Once you have one page loaded, all of the other pages are lightning fast. DNS? I'll try some of the utilities mentioned and see what they can get me.

    Anyone else have this issue?

  9. #9
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    8,619
    Distro
    Kubuntu 14.04 Trusty Tahr

    Re: Network Bottleneck Check

    I'll try a wild guess and suggest reducing the MTU of your Ethernet adapter. Since the firewall manufacturer admits they've seen the problem before, maybe it has a problem handling full-sized (1500 byte) packets correctly. No promises that this will fix things, though.
    If you ask for help, please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  10. #10
    Join Date
    Feb 2012
    Beans
    8

    Re: Network Bottleneck Check

    Well, I would try changing the MTU, but it's only slow on the first load. After loading the first page, all of the other pages are very quick to respond and load.

    I've looked around and there are a few other people with the same problem as me (slow first load) and a lot of them are blaming it on the lack of reverse DNS. I'll take a look at my DNS server tomorrow and make sure it's not slowing me down.

    Any other ideas as to why it would be slow on the first load, but not the rest. And note that I have tested the slow pages with Firebug and it told me that the transfer was slow, not DNS...

    EDIT:
    I just did some tests with and without the firewall and got some interesting results.
    With the firewall, one of our image-heavy sites took ~15 seconds while without it only ~4 seconds.
    Because of this I am sold that it is a firewall issue. Strange that it's only affecting the Ubuntu machine and not the FreeBSD machine though...
    Last edited by monarckco; November 19th, 2012 at 01:58 AM.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •