PDA

View Full Version : [ubuntu] Network Bottleneck Check



monarckco
November 15th, 2012, 05:32 AM
Hi all,

I've set up a nice Ubuntu+KVM based DRBD cluster and everything seems to fine except for when I try to view one of the websites running inside one of my virtual machines - it's really slow... I have LXDE installed on the host and have tested viewing the VM websites there and they are extremely fast - only about 2-3 seconds for a fairly complex page (compared to about 25 seconds when I view the same site externally). Because of this, I don't think it's a problem with Apache or KVM disk I/O.

I suspect that the problem is with the new switch I put the servers behind, but I can't be 100% sure. What tools should I use to detect where a network bottleneck is? I'm toying around with iperf at the moment, but my boss is looking to me for an absolute as to what and why the connection is slow.

Thanks :)

varunendra
November 16th, 2012, 12:16 AM
I don't think I can help you with this kind of setup but providing a few more details may help those who can. Details like -


What kind of networking there is between the host and the VM - NAT/Bridged... etc? I'm asking this with the only experience of virtualbox and vmware. I don't know if this kind complexity even exists or not with KVM.
Please confirm whether the host is connected to the router via cable or wifi ? I'm assuming it must be cable for speed and less-complexity benefits.
The outputs of following from host -


ifconfig -a
nm-tool

Plus any extra details you think noteworthy.

monarckco
November 16th, 2012, 04:48 AM
The machine is connected through a bridged interface.
Connected to network by cable.


#ifconfig -a (from within the vm)
eth0 Link encap:Ethernet HWaddr 52:54:00:e7:21:f5
inet addr:192.168.10.153 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fee7:21f5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2697007 errors:0 dropped:0 overruns:0 frame:0
TX packets:1082547 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:36793743884 (36.7 GB) TX bytes:5048650580 (5.0 GB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:28338 errors:0 dropped:0 overruns:0 frame:0
TX packets:28338 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2129272 (2.1 MB) TX bytes:2129272 (2.1 MB)

Not sure if I should really install network-manager on this system, so I can't check nm-tool output right now.

I've also installed varnish on it to see if it was just Apache being slow, but even with the varnish cache the images loaded slowly. I should mention that everything is fast except for images - they take ages...

I'm going to try VirtualBox and see how it goes, maybe it's some strange bug with KVM networking...

EDIT:
I've replaced the ethernet cables with brand new ones and tested connections with the server being behind the hub vs directly connected to the firewall in front of it with no difference- the FreeBSD server is fast no matter where it's placed on the network. I now believe it must have something to do with KVM.
Should this be moved to the Server category instead of Networking?

varunendra
November 16th, 2012, 07:33 AM
#ifconfig -a (from within the vm)

eth0 Link encap:Ethernet HWaddr 52:54:00:e7:21:f5
inet addr:192.168.10.153 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::5054:ff:fee7:21f5/64 Scope:Link
Actually I was interested in the ifconfig output of host, not the vm, because of this -

I have LXDE installed on the host and have tested viewing the VM websites there and they are extremely fast - only about 2-3 seconds for a fairly complex page (compared to about 25 seconds when I view the same site externally).
Means, you get the fast loading speed when you view the pages on the host machine, right ? Please correct me if I'm misunderstanding you.

If that's true, please post the output of ifconfig from host also. And it's okay if nm is not installed. Just tell us the gateway and DNS you are using on the host and the remote machines.

If you think it can be a DNS lookup delay, you may try disabling IPv6 on the interfaces (both on host and guest). Although I don't think it is going to help since you said everything loads fine except images.

Also, please make it a habit to wrap the codes between 'Code' tags, like I have done above in your quoted text. It can be auto-generated by clicking on # at the top of the edit box in which you create new posts, then copy-paste the code text in-between those tags. It preserves the output formatting and makes it more readable.


Should this be moved to the Server category instead of Networking?By the description of problem, sounds reasonable to me.

monarckco
November 16th, 2012, 09:21 AM
Alright, well I've done some switching around with the cables and it seems as if the firewall is slowing me down. When I plug the servers directly into the switch that comes before the firewall, everything is much faster. Perhaps the firewall doesn't like my network settings? I don't have any settings in the firewall to block this server's IP or anything, but it is definitely slowing down the images (html, css, etc. are all done in an instant). The strange thing is that the FreeBSD machine is behind the same firewall, but is not affected in the least.

Here's the hosts output:



#ifconfig -a
br0 Link encap:Ethernet HWaddr 5c:f3:fc:f2:35:91
inet addr:192.168.10.151 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::5ef3:fcff:fef2:3591/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1978647 errors:0 dropped:0 overruns:0 frame:0
TX packets:1279774 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:817184548 (817.1 MB) TX bytes:40806693222 (40.8 GB)

eth0 Link encap:Ethernet HWaddr 5c:f3:fc:f2:35:91
inet addr:192.168.10.150 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::5ef3:fcff:fef2:3591/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:9131368 errors:0 dropped:0 overruns:0 frame:0
TX packets:13704705 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5640014748 (5.6 GB) TX bytes:15492416713 (15.4 GB)
Interrupt:17 Memory:91a80000-91aa0000

eth1 Link encap:Ethernet HWaddr 5c:f3:fc:f2:35:92
inet addr:10.128.1.1 Bcast:10.128.255.255 Mask:255.255.0.0
inet6 addr: fe80::5ef3:fcff:fef2:3592/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:10204713 errors:0 dropped:0 overruns:0 frame:0
TX packets:39988333 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1464121436 (1.4 GB) TX bytes:47578413107 (47.5 GB)
Interrupt:19 Memory:91980000-919a0000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:76978 errors:0 dropped:0 overruns:0 frame:0
TX packets:76978 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:18402880 (18.4 MB) TX bytes:18402880 (18.4 MB)

usb0 Link encap:Ethernet HWaddr 5e:f3:fc:f2:35:94
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

virbr0 Link encap:Ethernet HWaddr f6:5b:8f:0c:da:d4
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vnet0 Link encap:Ethernet HWaddr fe:54:00:e7:21:f5
inet6 addr: fe80::fc54:ff:fee7:21f5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1178577 errors:0 dropped:0 overruns:0 frame:0
TX packets:2892334 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:5266669650 (5.2 GB) TX bytes:36853993657 (36.8 GB)

vnet1 Link encap:Ethernet HWaddr fe:54:00:7d:de:7f
inet6 addr: fe80::fc54:ff:fe7d:de7f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:16047 errors:0 dropped:0 overruns:0 frame:0
TX packets:36475 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:13662918 (13.6 MB) TX bytes:9032684 (9.0 MB)

vnet2 Link encap:Ethernet HWaddr fe:54:00:d5:53:8d
inet6 addr: fe80::fc54:ff:fed5:538d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:47974 errors:0 dropped:0 overruns:0 frame:0
TX packets:89944 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:69686551 (69.6 MB) TX bytes:38172335 (38.1 MB)

vnet3 Link encap:Ethernet HWaddr fe:54:00:20:4e:1e
inet6 addr: fe80::fc54:ff:fe20:4e1e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:39226 errors:0 dropped:0 overruns:0 frame:0
TX packets:56638 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:22966613 (22.9 MB) TX bytes:14984757 (14.9 MB)



We have two servers set up as DNS servers, 192.168.10.231 and 192.168.10.153. We also use google as a last resort, 8.8.8.8.
I checked my gateway settings and it seems that I did make a mistake there. I fixed it and reset networking, but it still seems too slow. Is it possible that the firewall just doesn't like my networking setup or something?

nothingspecial
November 16th, 2012, 10:04 AM
Thread moved to Server Platforms.

varunendra
November 16th, 2012, 11:08 AM
The strange thing is that the FreeBSD machine is behind the same firewall, but is not affected in the least.
Is it a physical server or a vm ?


Perhaps the firewall doesn't like my network settings?
..
..
.. Is it possible that the firewall just doesn't like my networking setup or something?
Is it a dedicated hardware firewall ? It may be interesting to know what model it is.

I won't pretend to know a lot about firewalls and stuff.. but with my very limited experience with both software (isa 2004) and hardware (sonicwall nsa 2400) firewalls, I can say that complex set of rules or extensive port/protocol filtering (in short any kind of complexity) can sometimes play wonderful games with you. In such situations, the dedicated hardware firewall, IMHO, is easier to analyze and troubleshoot.

You may try to compare and single-out the basic differences between your BSD server packets and the problematic one's. Like MTU, txqueuelen, etc.

I'd also like to mention that in last three days, I've noticed at least three different threads discussing network speed problems with VMs. So I suspect it may just be a vm factor - especially if your BSD one (which you say is working fine) is a physical machine and this current host has installed any recent updates. However, all the three threads I noticed were about 12.10 server and vmware/virtualbox.


With this I'll leave this thread for people who really understand and have experience with this kind of setup, and know what they are talking about :)

Good luck !

monarckco
November 16th, 2012, 01:46 PM
Well, neither? It's a jail inside of FreeBSD 9.

The firewall is a dedicated hardware firewall, Fortinet 50B I believe. I've been in contact with them trying to sort this out and they said that they've seen problems like this before so maybe its a step in the right direction.

I have a feeling that it's not the firewall though, because it's only the images on the initial connection that are slow. Once you have one page loaded, all of the other pages are lightning fast. DNS? I'll try some of the utilities mentioned and see what they can get me.

Anyone else have this issue?

SeijiSensei
November 16th, 2012, 03:52 PM
I'll try a wild guess and suggest reducing the MTU (http://www.ubuntugeek.com/how-to-change-mtu-maximum-transmission-unit-of-network-interface-in-ubuntu-linux.html) of your Ethernet adapter. Since the firewall manufacturer admits they've seen the problem before, maybe it has a problem handling full-sized (1500 byte) packets correctly. No promises that this will fix things, though.

monarckco
November 18th, 2012, 03:41 PM
Well, I would try changing the MTU, but it's only slow on the first load. After loading the first page, all of the other pages are very quick to respond and load.

I've looked around and there are a few other people with the same problem as me (slow first load) and a lot of them are blaming it on the lack of reverse DNS. I'll take a look at my DNS server tomorrow and make sure it's not slowing me down.

Any other ideas as to why it would be slow on the first load, but not the rest. And note that I have tested the slow pages with Firebug and it told me that the transfer was slow, not DNS...

EDIT:
I just did some tests with and without the firewall and got some interesting results.
With the firewall, one of our image-heavy sites took ~15 seconds while without it only ~4 seconds.
Because of this I am sold that it is a firewall issue. Strange that it's only affecting the Ubuntu machine and not the FreeBSD machine though...