Page 2 of 2 FirstFirst 12
Results 11 to 19 of 19

Thread: Installing jperf to test network speed

  1. #11
    Join Date
    Jun 2005
    Location
    Switzerland
    Beans
    776
    Distro
    Ubuntu 11.04 Natty Narwhal

    Re: Installing jperf to test network speed

    Quote Originally Posted by cbonilla View Post

    Here is the lspci output
    Please add "Code" marks to text output that you copy from the console - it is more comfortable to read. Just copy/paste the text from the console to here, highlight it with the mouse and click the "#" button in the small toolbar above the editing window.


    Quote Originally Posted by cbonilla View Post
    Code:
    carlos@PowerEdge:~$ sudo lspci
    sudo: unable to resolve host PowerEdge
    [sudo] password for carlos: 
    00:00.0 Host bridge: Broadcom CMIC-WS Host Bridge (GC-LE chipset) (rev 13)
    00:00.1 Host bridge: Broadcom CMIC-WS Host Bridge (GC-LE chipset)
    00:00.2 Host bridge: Broadcom CMIC-LE
    00:04.0 Class ff00: Dell Embedded Remote Access or ERA/O
    00:04.1 Class ff00: Dell Remote Access Card III
    00:04.2 IPMI SMIC interface: Dell Embedded Remote Access: BMC/SMIC device
    00:0e.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
    00:0f.0 Host bridge: Broadcom CSB5 South Bridge (rev 93)
    00:0f.1 IDE interface: Broadcom CSB5 IDE Controller (rev 93)
    00:0f.2 USB Controller: Broadcom OSB4/CSB5 OHCI USB Controller (rev 05)
    00:0f.3 ISA bridge: Broadcom CSB5 LPC bridge
    00:10.0 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
    00:10.2 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
    00:11.0 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
    00:11.2 Host bridge: Broadcom CIOB-X2 PCI-X I/O Bridge (rev 03)
    02:06.0 RAID bus controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02)
    03:06.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15)
    03:08.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15)
    04:08.0 PCI bridge: Intel Corporation 80303 I/O Processor PCI-to-PCI Bridge (rev 01)
    04:08.1 RAID bus controller: Dell PowerEdge Expandable RAID Controller 3/Di (rev 01)
    ok, we see that NICs and the RAID Controllers are not on the same PCI bus, so there's probably no congestion there.

    /dev/md0:
    Timing cached reads: 892 MB in 2.00 seconds = 446.15 MB/sec
    Timing buffered disk reads: 208 MB in 3.02 seconds = 68.89 MB/sec
    See? That should be good for way over 700Mbit/s

    And the large file test

    Code:
    carlos@PowerEdge:~$ sudo iperf -c 192.168.1.2 -n 600M -f /dev/mdo/video1.mpg
    sudo: unable to resolve host PowerEdge
    [sudo] password for carlos: 
    ------------------------------------------------------------
    Client connecting to 192.168.1.2, TCP port 5001
    TCP window size:   131 Kbit (default)
    ------------------------------------------------------------
    [  3] local 192.168.1.252 port 58497 connected with 192.168.1.2 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-43.8 sec  5.03 Gbits    115 Mbits/sec
    That's way too slow. Does it get any faster if you repeat the sending of the 600MBytes? With 2GBytes of RAM, the second time, the whole 600Megs should come from cache, not from disk - and should be noticeably faster.


    Hmm... Broadcom 5700 series onboard NICs... there are some stories around of them performing badly in some products, also on Dell PowerEdge 2650. If I remember correctly, HP even swapped out Mainboards on DL38x systems, taking back the Broadcom equipped ones and replacing them with Intel-NICced boards.

    2650 Problem with Broadcom Dual Onboard NIC's with Red Hat 7.2

    And then I found something very interesting. This is from a guy that had Broadcom NICs in an IBM server:

    Troubleshooting VMware ESX network perform...

    In a nutshell: looking at /proc/interrupts, he found out that the Broadcom NIC was sharing the Interrupt with the USB subsystem. Upon removing the USB kernel module, his Broadcom NIC started to fly!

    Can you show us /proc/interrupts from your server?

    Code:
    user@server $ cat /proc/interrupts
    regards

    Marc
    The story so far: In the beginning the Universe was created.This has made a lot of people very angry and has been widely regarded as a bad move. Douglas Adams, The Restaurant at the End of the Universe

  2. #12
    Join Date
    Aug 2008
    Location
    DC
    Beans
    18

    Re: Installing jperf to test network speed

    Thanks for the confirmation that I'm not imagining it

    Here's the output

    carlos@PowerEdge:~$ sudo cat /proc/interrupts
    [sudo] password for carlos:
    Code:
               CPU0       CPU1       CPU2       CPU3       
      0:         92          0          0          0   IO-APIC-edge      timer
      1:        450          0          0          0   IO-APIC-edge      i8042
      3:          4          0          0          0   IO-APIC-edge    
      4:          4          0          0          0   IO-APIC-edge    
      8:          2          0          0          0   IO-APIC-edge      rtc0
      9:          0          0          0          0   IO-APIC-fasteoi   acpi
     11:          0          0          0          0   IO-APIC-fasteoi   ohci_hcd:usb1
     12:      17640          0          0          0   IO-APIC-edge      i8042
     14:      22935          0          0          0   IO-APIC-edge      pata_serverworks
     15:          0          0          0          0   IO-APIC-edge      pata_serverworks
     24:       6078          0          0          0   IO-APIC-fasteoi   sata_sil24
     28:     299146          0          0          0   IO-APIC-fasteoi   eth0
     29:          1          0          0          0   IO-APIC-fasteoi   eth1
     30:      16112          0          0          0   IO-APIC-fasteoi   aacraid
    NMI:          0          0          0          0   Non-maskable interrupts
    LOC:      85345     218076     127022     221991   Local timer interrupts
    RES:       9126       8280      21755       9145   Rescheduling interrupts
    CAL:        404        830        499        835   function call interrupts
    TLB:        586        682       1362       1511   TLB shootdowns
    SPU:          0          0          0          0   Spurious interrupts
    ERR:          0
    MIS:          0
    Nice trick for marking as code, wasn't aware of that. As I said, I'm new to Ubuntu. FyI, my network connection is via eth0.

    Not sure if its relevant but at boot I'm getting the following error : MP-BIOS BUG 8254 Timer not connected to IO-APIC. I think that error only started when I upgraded from Hardy to Itrepid although the speed was unaffected.
    Last edited by cbonilla; February 19th, 2009 at 11:50 PM. Reason: add more info

  3. #13
    Join Date
    Jun 2005
    Location
    Switzerland
    Beans
    776
    Distro
    Ubuntu 11.04 Natty Narwhal

    Re: Installing jperf to test network speed

    Quote Originally Posted by netztier View Post

    In a nutshell: looking at /proc/interrupts, he found out that the Broadcom NIC was sharing the Interrupt with the USB subsystem. Upon removing the USB kernel module, his Broadcom NIC started to fly!
    Guess what! It's sensational! I always felt a bit stumped by low transfer rates on my AMD based desktop (Shuttle SN95Gv3 barebone and an AMD64 Venice 3000+). And I was p*ssed off by stuttering sound output during iPerf tests, that would barely ever go beyond 40 MByte/s on a good day.

    Then I looked an my own /proc/interrupts:

    Code:
               CPU0       
      0:        127    XT-PIC-XT        timer
      1:          2    XT-PIC-XT        i8042
      2:          0    XT-PIC-XT        cascade
      3:    1918827    XT-PIC-XT        ohci_hcd:usb2, ohci_hcd:usb5, eth0
      4:          3    XT-PIC-XT      
      5:    9551333    XT-PIC-XT        hhci_hcd:usb3, ehci_hcd:usb4, eth1_rename, NVidia CK8S
      7:         35    XT-PIC-XT        parport0
      8:          2    XT-PIC-XT        rtc0
      9:          0    XT-PIC-XT        acpi
     11:    1038651    XT-PIC-XT        ehci_hcd:usb1, ohci_hcd:usb6, ohci1394, sata_nv, nvidia
     14:          0    XT-PIC-XT        pata_amd
     15:     120846    XT-PIC-XT        pata_amd
    NMI:          0   Non-maskable interrupts
    LOC:    2300376   Local timer interrupts
    RES:          0   Rescheduling interrupts
    CAL:          0   function call interrupts
    TLB:          0   TLB shootdowns
    SPU:          0   Spurious interrupts
    ERR:          1
    MIS:          0
    D'OH! eth1_rename (the device name the somewhat strange Marvell Chip sometimes appears as) shares the interrupt with the CK8S sound device!

    I went into the BIOS and started to deactivate unneeded devices like the parallel port, floppy controller, first IDE ( the hardisk is on SATA and DVD drive on second IDE). Apparently, this freed up some interrupt allocation space and no we have this:

    Code:
    marc@cube:/tmp$ more /proc/interrupts 
               CPU0       
      0:        127    XT-PIC-XT        timer
      1:          2    XT-PIC-XT        i8042
      2:          0    XT-PIC-XT        cascade
      3:     240024    XT-PIC-XT        ehci_hcd:usb1, NVidia CK8S
      4:          2    XT-PIC-XT      
      5:   17347530    XT-PIC-XT        ehci_hcd:usb4, eth0
      7:     135470    XT-PIC-XT        ohci_hcd:usb5, nvidia
      8:          2    XT-PIC-XT        rtc0
      9:          0    XT-PIC-XT        acpi
     10:       3290    XT-PIC-XT        ohci_hcd:usb2, ohci_hcd:usb6, ohci1394
     11:      65870    XT-PIC-XT        ohci_hcd:usb3, sata_nv
     14:          0    XT-PIC-XT        pata_amd
     15:      20677    XT-PIC-XT        pata_amd
    NMI:          0   Non-maskable interrupts
    LOC:     399179   Local timer interrupts
    RES:          0   Rescheduling interrupts
    CAL:          0   function call interrupts
    TLB:          0   TLB shootdowns
    SPU:          0   Spurious interrupts
    ERR:          1
    MIS:          0
    The Marvell chip is now eth0 again, has almost it's own interrupt, and we have 70-75MByte/s in and out of the interface with no sound stutter at all!

    Let's call THAT a day!

    Marc
    The story so far: In the beginning the Universe was created.This has made a lot of people very angry and has been widely regarded as a bad move. Douglas Adams, The Restaurant at the End of the Universe

  4. #14
    Join Date
    Aug 2008
    Location
    DC
    Beans
    18

    Re: Installing jperf to test network speed

    I'll go give that a try and let you know

    Carlos

  5. #15
    Join Date
    Aug 2008
    Location
    DC
    Beans
    18

    Re: Installing jperf to test network speed

    OK, went into the BIOS and flagged the floopy as not present and disabled the floppy and USB controler. Here's the output from

    Code:
    carlos@PowerEdge:~$ sudo cat /proc/interrupts
    [sudo] password for carlos: 
               CPU0       CPU1       CPU2       CPU3       
      0:         92          0          0          0   IO-APIC-edge      timer
      1:         65          0          0          0   IO-APIC-edge      i8042
      3:          4          0          0          0   IO-APIC-edge    
      4:          4          0          0          0   IO-APIC-edge    
      8:          2          0          0          0   IO-APIC-edge      rtc0
      9:          0          0          0          0   IO-APIC-fasteoi   acpi
     12:       7194          0          0          0   IO-APIC-edge      i8042
     14:       1013          0          0          0   IO-APIC-edge      pata_serverworks
     15:          0          0          0          0   IO-APIC-edge      pata_serverworks
     24:       1227          0          0          0   IO-APIC-fasteoi   sata_sil24
     28:       8622          0          0          0   IO-APIC-fasteoi   eth0
     29:          1          0          0          0   IO-APIC-fasteoi   eth1
     30:      11759          0          0          0   IO-APIC-fasteoi   aacraid
    NMI:          0          0          0          0   Non-maskable interrupts
    LOC:      14525      11449      10258      13279   Local timer interrupts
    RES:       4883       6844       8481       6877   Rescheduling interrupts
    CAL:        454        750        446        726   function call interrupts
    TLB:        523        606       1211       1404   TLB shootdowns
    SPU:          0          0          0          0   Spurious interrupts
    ERR: 0

    I'll try a file transfer in a moment

  6. #16
    Join Date
    Aug 2008
    Location
    DC
    Beans
    18

    Re: Installing jperf to test network speed

    I ran iperf again. All I can say is WOW. Throughput leapt from 115Mbits sec to over 700. Truly amazing. Its too late or I'd offer to name my first born after you

    Code:
    carlos@PowerEdge:~$ sudo iperf -c 192.168.1.2 -n 600M -f /dev/md0/video1.mpg
    ------------------------------------------------------------
    Client connecting to 192.168.1.2, TCP port 5001
    TCP window size:   131 Kbit (default)
    ------------------------------------------------------------
    [  3] local 192.168.1.252 port 48840 connected with 192.168.1.2 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0- 7.1 sec  5.03 Gbits    705 Mbits/sec

    and then ran it a second time

    Code:
    carlos@PowerEdge:~$ sudo iperf -c 192.168.1.2 -n 600M -f /dev/md0/video1.mpg
    ------------------------------------------------------------
    Client connecting to 192.168.1.2, TCP port 5001
    TCP window size:   131 Kbit (default)
    ------------------------------------------------------------
    [  3] local 192.168.1.252 port 58683 connected with 192.168.1.2 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0- 7.1 sec  5.03 Gbits    706 Mbits/sec
    Last edited by cbonilla; February 20th, 2009 at 12:45 AM.

  7. #17
    Join Date
    Jun 2005
    Location
    Switzerland
    Beans
    776
    Distro
    Ubuntu 11.04 Natty Narwhal

    Re: Installing jperf to test network speed

    Quote Originally Posted by cbonilla View Post
    I ran iperf again. All I can say is WOW. Throughput leapt from 115Mbits sec to over 700. Truly amazing. Its too late or I'd offer to name my first born after you


    Congrats for fixing it. From the outputs of /proc/interrupts alone, I couldn't tell where the difference is and what actually happened that unleashed your NIC.

    So - now we know that iPerf/jPerf show decent throughput rates, but that's nothing but raw TCP. What about Samba/CIFS or NFS performance?

    regards

    Marc
    Last edited by netztier; February 20th, 2009 at 12:43 PM.
    The story so far: In the beginning the Universe was created.This has made a lot of people very angry and has been widely regarded as a bad move. Douglas Adams, The Restaurant at the End of the Universe

  8. #18
    Join Date
    Aug 2008
    Location
    DC
    Beans
    18

    Re: Installing jperf to test network speed

    No, the congrats on fixing it belong to you, not me. I just followed directions

    Carlos

  9. #19
    Join Date
    Aug 2008
    Location
    DC
    Beans
    18

    Re: Installing jperf to test network speed

    Marc -In trying to understand what you accomplished, I came across this. Thought it might be of interest in case similar problems arise.

    Carlos

    Local APICs

    LAPICs manage all external interrupts for the processor that it is part of. In addition, it is able to accept and generate inter-processor interrupts (IPIs) between LAPICs. LAPICs may support up to 224 usable IRQ vectors from an I/O APIC. Vectors numbers 0 to 31, out of 0 to 255, are reserved for exception handling by x86 processors.

    [edit] I/O APICs

    I/O APICs contain a redirection table, which is used to route the interrupts it receives from peripheral buses to one or more Local APICs.

    [edit] Considerations

    [edit] Hardware Bugs

    There are a number of known bugs in implementations of APIC systems, especially with concern to how the 8259 is connected.

    There are defective BIOSes which do not set up interrupt routing properly. This includes the errors in the implementation of ACPI tables and Intel Multiprocessor Specification tables.

    [edit] Operating System Issues

    It can be a cause of system failure, as some versions of some operating systems do not support it properly. If this is the case, disabling I/O APIC may cure the problem. For Linux, try the 'noapic nolapic' kernel parameters; for FreeBSD, the 'hint.apic.0.disabled' kernel environment variable; For NetBSD, drop into userconf ("boot -c" from the boot prompt), then use "disable ioapic" and "exit" to continue the boot process.

    In NetBSD with nforce chipsets from Nvidia, having IOAPIC support enabled in the kernel can cause "nfe0: watchdog timeout" errors.[1]

    In Linux, problems with I/O APIC are one of several causes of error messages concerning "spurious 8259A interrupt: IRQ7.". It is also possible that I/O APIC causes problems with network interfaces based on via-rhine driver, causing a transmission time out. Uniprocessor kernels with APIC enabled can cause spurious interrupts to be generated.

Page 2 of 2 FirstFirst 12

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •