NetXtreme II BCM57810 10 GB SFP+ Network Card Issues

[mallinss]

Hello All,

I have recently added a NetXtreme II BCM57810 10 GB SFP+ Network Card to my Ubuntu server (22.04) in addition to the onboard network interface, for the life of me, I can't get this card to work correctly. I have tested this card in a HP Server and it works just fine, so I'm certain the card is all good. I notice that there is no link lights lighting up on the card or the 10 switch, so clearly something going on. (Note - I'm using SFP+ Optics, not copper)

I have three (3) Netplan .yaml config files, one(1) for each interface, the eno2 NIC is working fine on 192.168.253.160 fixed IP, I set the other two for the SPF+ interfaces to DHCP, also have outputs below.

Here are some outputs from the command line as below.

Any ideas on what I have done wrong or missing would be greatly appreciated.

Regards Steve


$ sudo lshw -C network


*-network:0
description: Ethernet interface
product: NetXtreme II BCM57810 10 Gigabit Ethernet
vendor: Broadcom Inc. and subsidiaries
physical id: 0
bus info: pci@0000:02:00.0
logical name: enp2s0f0
version: 10
serial: a0:d3:c1:fe:d1:80
capacity: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd
configuration: autonegotiation=off broadcast=yes driver=bnx2x driverversion=5.15.0-91-generic firmware=bc 7.13.75 latency=0 link=no multicast=yes port=fibre
resources: irq:17 memory:99000000-997fffff memory:99800000-99ffffff memory:9b000000-9b00ffff memory:df480000-df4fffff memory:9b020000-9b21ffff memory:9b420000-9b49ffff
*-network:1
description: Ethernet interface
product: NetXtreme II BCM57810 10 Gigabit Ethernet
vendor: Broadcom Inc. and subsidiaries
physical id: 0.1
bus info: pci@0000:02:00.1
logical name: enp2s0f1
version: 10
serial: a0:d3:c1:fe:d1:84
capacity: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd
configuration: autonegotiation=off broadcast=yes driver=bnx2x driverversion=5.15.0-91-generic firmware=bc 7.13.75 latency=0 link=no multicast=yes port=fibre
resources: irq:18 memory:9a000000-9a7fffff memory:9a800000-9affffff memory:9b010000-9b01ffff memory:df400000-df47ffff memory:9b220000-9b41ffff memory:9b4a0000-9b51ffff
*-network
description: Ethernet interface
product: I210 Gigabit Network Connection
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:06:00.0
logical name: eno2
version: 03
serial: ac:1f:6b:dd:13:81
size: 1Gbit/s
capacity: 1Gbit/s
width: 32 bits
clock: 33MHz
capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.15.0-91-generic duplex=full firmware=3.25, 0x800005cc ip=192.168.253.160 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
resources: irq:16 memory:df200000-df27ffff ioport:d000(size=32) memory:df280000-df283fff
*-network DISABLED
description: Ethernet interface
product: Ethernet Connection (2) I219-LM
vendor: Intel Corporation
physical id: 1f.6
bus info: pci@0000:00:1f.6
logical name: eno1
version: 31
serial: ac:1f:6b:dd:13:80
size: 1Gbit/s
capacity: 1Gbit/s
width: 32 bits
clock: 33MHz
capabilities: pm msi bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=5.15.0-91-generic duplex=full firmware=0.8-4 latency=0 link=no multicast=yes port=twisted pair speed=1Gbit/s
resources: irq:134 memory:df600000-df61ffff

$ sudo ifconfig -a
eno1: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether ac:1f:6b:dd:13:80 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xdf600000-df620000


eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.253.160 netmask 255.255.255.0 broadcast 192.168.253.255
inet6 fe80::ae1f:6bff:fedd:1381 prefixlen 64 scopeid 0x20<link>
inet6 fd6d:8fd1:d580:cb46:ae1f:6bff:fedd:1381 prefixlen 64 scopeid 0x0<global>
ether ac:1f:6b:dd:13:81 txqueuelen 1000 (Ethernet)
RX packets 644389 bytes 954533337 (954.5 MB)
RX errors 0 dropped 383 overruns 0 frame 0
TX packets 268561 bytes 18310196 (18.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xdf200000-df27ffff


enp2s0f0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether a0:d3:c1:fe:d1:80 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17 memory 0x99000000-997fffff


enp2s0f1: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether a0:d3:c1:fe:d1:84 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 18 memory 0x9a000000-9a7fffff


lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 4928 bytes 1644948 (1.6 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4928 bytes 1644948 (1.6 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0


$ sudo dmesg | grep -e bnx2x -e enp
[ 1.822141] bnx2x 0000:02:00.0: msix capability found
[ 1.832916] bnx2x 0000:02:00.0: part number 0-0-0-0
[ 1.974785] [bnx2x_iov_init_one:1210(eth%d)]ARI not supported (check pci bridge ARI forwarding), SRIOV can not be enabled
[ 2.113932] bnx2x 0000:02:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 2.114465] bnx2x 0000:02:00.1: msix capability found
[ 2.119332] bnx2x 0000:02:00.1: part number 0-0-0-0
[ 2.269890] [bnx2x_iov_init_one:1210(eth%d)]ARI not supported (check pci bridge ARI forwarding), SRIOV can not be enabled
[ 2.330158] bnx2x 0000:02:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[ 2.401679] bnx2x 0000:02:00.0 enp2s0f0: renamed from eth0
[ 2.704503] bnx2x 0000:02:00.1 enp2s0f1: renamed from eth2
[ 26.423130] bnx2x 0000:02:00.1 enp2s0f1: using MSI-X IRQs: sp 163 fp[0] 165 ... fp[7] 172
[ 27.146635] bnx2x 0000:02:00.0 enp2s0f0: using MSI-X IRQs: sp 130 fp[0] 138 ... fp[7] 146


sudo lshw
plex-server
description: System
product: Super Server (To be filled by O.E.M.)
vendor: Supermicro
version: 0123456789
serial: 0123456789
width: 64 bits
capabilities: smbios-3.0.0 dmi-3.0.0 smp vsyscall32
configuration: boot=normal chassis=server family=To be filled by O.E.M. sku=To be filled by O.E.M. uuid=00000000-0000-0000-0000-ac1f6bdd1380
*-core
description: Motherboard
product: X11SAE-F
vendor: Supermicro
physical id: 0
version: 1.03A
serial: ZM19BS017843
slot: To be filled by O.E.M.
*-firmware
description: BIOS
vendor: American Megatrends Inc.
physical id: 0
version: 3.0
date: 07/04/2019
size: 64KiB
capacity: 16MiB



/etc/netplan$ ls -al
total 44
drwxr-xr-x 2 root root 4096 Jan 5 13:59 .
drwxr-xr-x 166 root root 12288 Jan 4 13:01 ..
-rw-r--r-- 1 root root 263 Jun 25 2023 00-installer-config.yaml.backup
-rw-r--r-- 1 root root 119 Jan 5 13:59 1
-rw-r--r-- 1 root root 263 Dec 31 15:40 eno2.yaml
-rw-r--r-- 1 root root 263 Jan 5 13:05 eno2.yaml.backup
-rw-r--r-- 1 root root 119 Jan 5 13:59 enp2s0f0.yaml
-rw-r--r-- 1 root root 119 Jan 5 13:58 enp2s0f1.yaml
-rw-r--r-- 1 root root 119 Jan 5 13:56 enp2s0f1.yaml.backup


/etc/netplan$ more eno2.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
eno2:
addresses: ['192.168.253.160/24']
gateway4: 192.168.253.1
nameservers:
addresses: [1.1.1.1, 1.0.0.1, 8.8.8.8]
version: 2

/etc/netplan$ more enp2s0f0.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
enp2s0f0:
dhcp4: true
version: 2

/etc/netplan$ more enp2s0f1.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
enp2s0f1:
dhcp4: true
version: 2

[fkarakaya97]

Did you find a way to fix this ?