2 NICs on KVM host

[aljames2]

I want to use my extra NIC card (1 port), that I have plugged into a PCIe slot, and assign it to my WG VPN vm. Trying to figure out how to either assign or passthru this nic. The KVM host is on Ubuntu 22.04 with a netplan file creating a bridge (br0) for my Nextcloud server to run on. I am not seeing a way to assign this other physical interface in virt-manager. The only option I see under network source is either br0 or Macvtap. Perhaps Macvtap? Or, can I set up a second bridge (br1) in the same .yaml file for this interface assignment? I’ve read some about PCI passthrough and the process seems tricky and not always reliable. I appreciate any ideas

[QIII]

One use case in my situation:

Use ifconfig to get the names of your interfaces. I usually assign the VM a Macvtap device with the name of the interface. Bear in mind that macvtap does not usually facilitate direct host-to-guest communication, so you'll have to treat the VM as just another machine on your LAN. I have 14 available NICs on this particular machine, so I can assign individual NICs to VMs permanently. I also have a 24 port switch that everything is individually connected to. You may not find this the best solution for your use case.

You can find details at https://www.linux-kvm.org/page/Networking and make your choices.

[aljames2]

Much thanks QIII, this helps, some new information I hadn’t found yet. I should have this sorted out today.

[MAFoElffen]

RE: https://wenchma.github.io/2016/09/07...deep-dive.html

I did mine a bit differently... With my old Dell PE T720, I had many NIC's as that machine was designed as a virt host... But my newer workstation only came with a single onboard NIC.

What I added to my workstation was a 2 port PCIe x4 lane Intel 82576 chip'ed NIC. That is SR-IOV capable where I set one of the ports as my Bridge, and the other I setup as SR-IOV, with 8 virtual ports. The NIC was $35 new. These cards come in PCIe x1 & x4 for the same price.

I have mine setup by just adding two boot parameters, and defined in my netplan yaml file. Then a short XML file to add it into the KVM networking. Fairly painless.

It worked so well, and was so much easier than I remember it being prior to 18.04... That if I did it again, I would do it a bit differently... See below.

A Dell Intel i350-T4 is Quad-port and is PCIe x4 lane. Each port can be split into 8 virtual ports... And can be had between $35 to $85 dollars. That comes out to 32 ports, for about $2.65 a port, if you picked it up on the high end of that scale ($85). A very good value, and a wise long-term investment.

I used PCIe x4, because I was worried about throughput. (Even though SR-IOV spec's are rated high) I thrash that machine and the loads I put on it... But I do have my media server on it. If it affects my wife watching what she is used to and wants to watch... It gets scary. I need to keep her happy and content. LOL.

[aljames2]

                WAN
                |
             ___|_________
            |  pfSense    |
            |   Router    |
            |____ (en01)__|
                    |
                    |
                    |-------(VLAN - en01.2) 192.168.23.0 (subnet)
                    |
                    |
                    |10.11.37.0 (trunk subnet)
                    |
                ____|___
               | switch |
               |________|(dumb/unmanaged sw)
                 |  |
              ___|  |
                    |
                    |
                ____|_______
               | KVM Host   |
               |            |
               |_nic1__nic2_|
                  |     |
                  |     |
                  |     |
     10.11.37.5 (br0)   |
                  |     |
                  |     |
             (NC VM)   (WG-vpn VM)
         10.11.37.10     192.168.23.112 (VLAN)

Thought I'd try some ascii art.

This is what I am trying to do. To use 1 port off the router, through my dumb switch, and onto my KVM host machine. There are 2 VMs (Nextcloud & Wireguard vpn). I have other physical machines connected to this switch as well but I'm focusing on this branch. I am thinking of involving the router to create a vlan to help with some isolation between the VPN & the KVM host, by putting the VPN on a VLAN subnet. Essentially isolate the VPN server.

At present, I have 2 NICs on the KVM host but neither support SR-IOV. I can run 2 cables between my dumb switch & the host but this doesn't seem to achieve the goal. I would have each VM on a separate NIC but there would be no isolation as I see it. The switch is not VLAN smart and cannot help control traffic. I prefer not to have a managed switch.

@MAFoElffen, I did just order that PCIe Dell quad port device for the KVM host. I do know that my MB supports SR-IOV & IOMMU. Not sure I'll know how to use it when I get it, LOL... Will I even need to deal with using a VLAN if I can get this virtual networking set up right on the host?

[MAFoElffen]

At present, I have 2 NICs on the KVM host but neither support SR-IOV. I can run 2 cables between my dumb switch & the host but this doesn't seem to achieve the goal. I would have each VM on a separate NIC but there would be no isolation as I see it. The switch is not VLAN smart and cannot help control traffic. I prefer not to have a managed switch.

@MAFoElffen, I did just order that PCIe Dell quad port device for the KVM host. I do know that my MB supports SR-IOV & IOMMU. Not sure I'll know how to use it when I get it, LOL... Will I even need to deal with using a VLAN if I can get this virtual networking set up right on the host?

I figure with a KVM host, you can never have enough network ports to play with!!!

VLANs -- Though I have trained on them and done them, while training for CISCO CCNA... I don't find a need for them here. Besides, then to get out of that VLAN to something else, then you have to route to there.

For SR-IOV, at first I did via a vfio.conf file in modules, excluding... But then after doing that that way, I found I severely overthought that!!! It can be done a lot simpler and easier...

First your find the ID of the NIC in lspci

mafoelffen@msi-ubuntu:~$ lspci -nnnk | grep -E 'Ethernet'
    DeviceName: Onboard - Ethernet
06:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:125c] (rev 04)
07:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)

Then in the GRUB_CMDLINE_LINUX_DEFAULT line of /etc/default/grub:

mafoelffen@msi-ubuntu:~$ grep 'GRUB_CMDLINE_LINUX_DEFAULT' /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="-- splash intel_iommu=on iommu=pt igb.max_vfs=7"

Then, after a reboot, I do 'ip a' to confirm it's split out, then do my netplan file

mafoelffen@msi-ubuntu:~$ grep . /etc/netplan/00-installer-config.yaml
# This is the custom network config written by MAFoElffen
network:
  version: 2
  renderer: networkd
  ethernets:
    enp6s0:
      dhcp4: false
      dhcp6: false
      addresses:
          - 10.0.0.5/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      routes:
          - to: default
            via: 10.0.0.1
            metric: 100
            on-link: true
      mtu: 1500
## Begin SR-IOV on enp7s0f0 
    enp7s0f0:
      virtual-function-count: 7
      dhcp4: false
      dhcp6: false
      optional: yes
    enp7s0f0v0:
      link: enp7s0f0
      addresses:
          - 10.0.0.10/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      dhcp4: false
      dhcp6: false
      optional: yes
    enp7s0f0v1:
      link: enp7s0f0
      addresses:
          - 10.0.0.11/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      dhcp4: false
      dhcp6: false
      optional: yes
    enp7s0f0v1:
      link: enp7s0f0
      addresses:
          - 10.0.0.11/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      dhcp4: false
      dhcp6: false
      optional: yes
    enp7s0f0v2:
      link: enp7s0f0
      addresses:
          - 10.0.0.12/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      dhcp4: false
      dhcp6: false
      optional: yes
    enp7s0f0v3:
      link: enp7s0f0
      addresses:
          - 10.0.0.13/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      dhcp4: false
      dhcp6: false
      optional: yes
    enp7s0f0v4:
      link: enp7s0f0
      addresses:
          - 10.0.0.14/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      dhcp4: false
      dhcp6: false
      optional: yes
    enp7s0f0v5:
      link: enp7s0f0
      addresses:
          - 10.0.0.15/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      dhcp4: false
      dhcp6: false
      optional: yes
    enp7s0f0v6:
      link: enp7s0f0
      addresses:
          - 10.0.0.16/8
      nameservers:
          addresses: [8.8.8.8, 8.8.4.4]
      dhcp4: false
      dhcp6: false
      optional: yes
## End SR-IOV on enp7s0f0
## Begin Bridge on enp7s0f1
    enp7s0f1:
      dhcp4: false
      dhcp6: false
      optional: yes
  bridges:
    br0:
      interfaces: [enp7s0f1]
      addresses: [10.0.0.6/8]
      routing-policy:
         - from: 10.0.0.6
           table: 10
      # ** ADDITIONAL ROUTING POLICY **
         - to: 172.16.1.0/24
           table: 172
      # *******************************
      routes:
         - to: 0.0.0.0/0
           via: 10.0.0.1
           table: 10
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4, 1.1.1.1, 1.0.0.1]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no
      optional: yes
## End Bridge on enp7s0f1

Then apply

sudo netplan generate
sudo netplan apply

Done.

[aljames2]

Thanks for the details, this helps!
That new… (new to me)… card arrived today, I will be working on this very soon. The extra ports will come in handy because I do want to add a few more servers to this host.

[aljames2]

An update

I received the Dell I350-T4 Quad port card I bought, I thought it was new, at least it was advertised as such, but it wasn't. Anyhow, after running lspci, I noticed it did not show the SR-IOV capability. I made sure the IOMMU & SR-IOV setting was enabled on the MB as well as in grub. After some reading on the Intel website, I learned that some of the Dell OEM cards are know to have the SR-IOV disabled on the card. In particular the Gigabit cards. I think there is another version, perhaps I350-T4V2 that is fine. If you get the Intel retail version of the card then it comes with SR-IOV enabled. I think Dell figured users of a gigabit card would not need SR-IOV. Perhaps they are right. I shipped that card back.

I had another card lying around that I had thought a few years ago wasn't working but turns out it does work. I guess I didn't know what I was doing with it previously It is a Intel Gigabit ET 82576 Quad Port server adapter. At first, I had no lights on any of the ports, so I figured my original theory of a broken card was true. Then I found a "Gen" setting in my Asus Bios for the PCIe 16_2 x4 slot. The default setting was Auto which didn't work. I knew this older card was likely not Gen-3 or Gen-4, so I set it to Gen-2 and voila, the card works. I have the VFs set up now & so far so good.

Wondering if I should let the Host be assigned by itself to the MB integrated port, and put the VMs all on this card, that's what I am thinking..

[MAFoElffen]

Good job. It may help others here to share what settings and changes you made to your /etc/default/grub and /etc/netplan/<Named>.yaml file so they can do the same...

Just a thought.

[aljames2]

Good job. It may help others here to share what settings and changes you made to your /etc/default/grub and /etc/netplan/<Named>.yaml file so they can do the same...

Thanks, I do plan to share all I’ve done here soon. I am still trying to sort out getting my PCIe nic to pass through. My issue is an error that virt-manager gives about some devices in the same IOMMU group need to be bound to a VFIO driver. It is IOMMU group# 15. I have 18 devices all lumped in this group. Some of which are upstream root based devices, such as a few cpu & bridge functions. The root devices are not the issue, but all of my nics, VFs, & an onboard NVMe are also in this group. So I either need to pass all the non-root devices through to the VM which I don’t want, or somehow detach the device I want to pass. I have read about overriding ACU which eliminates isolation and is a last resort.