Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: 2 NICs on KVM host

  1. #1
    Join Date
    Feb 2019
    Location
    Virginia
    Beans
    366
    Distro
    Xubuntu 22.04 Jammy Jellyfish

    2 NICs on KVM host

    I want to use my extra NIC card (1 port), that I have plugged into a PCIe slot, and assign it to my WG VPN vm. Trying to figure out how to either assign or passthru this nic. The KVM host is on Ubuntu 22.04 with a netplan file creating a bridge (br0) for my Nextcloud server to run on. I am not seeing a way to assign this other physical interface in virt-manager. The only option I see under network source is either br0 or Macvtap. Perhaps Macvtap? Or, can I set up a second bridge (br1) in the same .yaml file for this interface assignment? I’ve read some about PCI passthrough and the process seems tricky and not always reliable. I appreciate any ideas

  2. #2
    Join Date
    Jul 2008
    Location
    The Left Coast of the USA
    Beans
    Hidden!
    Distro
    Kubuntu

    Re: 2 NICs on KVM host

    One use case in my situation:

    Use ifconfig to get the names of your interfaces. I usually assign the VM a Macvtap device with the name of the interface. Bear in mind that macvtap does not usually facilitate direct host-to-guest communication, so you'll have to treat the VM as just another machine on your LAN. I have 14 available NICs on this particular machine, so I can assign individual NICs to VMs permanently. I also have a 24 port switch that everything is individually connected to. You may not find this the best solution for your use case.

    You can find details at https://www.linux-kvm.org/page/Networking and make your choices.
    Last edited by QIII; January 5th, 2024 at 09:54 PM.
    Please read The Forum Rules and The Forum Posting Guidelines

    A thing discovered and kept to oneself must be discovered time and again by others. A thing discovered and shared with others need be discovered only the once.
    This universe is crazy. I'm going back to my own.

  3. #3
    Join Date
    Feb 2019
    Location
    Virginia
    Beans
    366
    Distro
    Xubuntu 22.04 Jammy Jellyfish

    Re: 2 NICs on KVM host

    Much thanks QIII, this helps, some new information I hadn’t found yet. I should have this sorted out today.

  4. #4
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 2 NICs on KVM host

    RE: https://wenchma.github.io/2016/09/07...deep-dive.html

    I did mine a bit differently... With my old Dell PE T720, I had many NIC's as that machine was designed as a virt host... But my newer workstation only came with a single onboard NIC.

    What I added to my workstation was a 2 port PCIe x4 lane Intel 82576 chip'ed NIC. That is SR-IOV capable where I set one of the ports as my Bridge, and the other I setup as SR-IOV, with 8 virtual ports. The NIC was $35 new. These cards come in PCIe x1 & x4 for the same price.

    I have mine setup by just adding two boot parameters, and defined in my netplan yaml file. Then a short XML file to add it into the KVM networking. Fairly painless.

    It worked so well, and was so much easier than I remember it being prior to 18.04... That if I did it again, I would do it a bit differently... See below.

    A Dell Intel i350-T4 is Quad-port and is PCIe x4 lane. Each port can be split into 8 virtual ports... And can be had between $35 to $85 dollars. That comes out to 32 ports, for about $2.65 a port, if you picked it up on the high end of that scale ($85). A very good value, and a wise long-term investment.

    I used PCIe x4, because I was worried about throughput. (Even though SR-IOV spec's are rated high) I thrash that machine and the loads I put on it... But I do have my media server on it. If it affects my wife watching what she is used to and wants to watch... It gets scary. I need to keep her happy and content. LOL.
    Last edited by MAFoElffen; January 8th, 2024 at 12:23 AM.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  5. #5
    Join Date
    Feb 2019
    Location
    Virginia
    Beans
    366
    Distro
    Xubuntu 22.04 Jammy Jellyfish

    Re: 2 NICs on KVM host

    Code:
                    WAN
                    |
                 ___|_________
                |  pfSense    |
                |   Router    |
                |____ (en01)__|
                        |
                        |
                        |-------(VLAN - en01.2) 192.168.23.0 (subnet)
                        |
                        |
                        |10.11.37.0 (trunk subnet)
                        |
                    ____|___
                   | switch |
                   |________|(dumb/unmanaged sw)
                     |  |
                  ___|  |
                        |
                        |
                    ____|_______
                   | KVM Host   |
                   |            |
                   |_nic1__nic2_|
                      |     |
                      |     |
                      |     |
         10.11.37.5 (br0)   |
                      |     |
                      |     |
                 (NC VM)   (WG-vpn VM)
             10.11.37.10     192.168.23.112 (VLAN)

    Thought I'd try some ascii art.

    This is what I am trying to do. To use 1 port off the router, through my dumb switch, and onto my KVM host machine. There are 2 VMs (Nextcloud & Wireguard vpn). I have other physical machines connected to this switch as well but I'm focusing on this branch. I am thinking of involving the router to create a vlan to help with some isolation between the VPN & the KVM host, by putting the VPN on a VLAN subnet. Essentially isolate the VPN server.

    At present, I have 2 NICs on the KVM host but neither support SR-IOV. I can run 2 cables between my dumb switch & the host but this doesn't seem to achieve the goal. I would have each VM on a separate NIC but there would be no isolation as I see it. The switch is not VLAN smart and cannot help control traffic. I prefer not to have a managed switch.

    @MAFoElffen, I did just order that PCIe Dell quad port device for the KVM host. I do know that my MB supports SR-IOV & IOMMU. Not sure I'll know how to use it when I get it, LOL... Will I even need to deal with using a VLAN if I can get this virtual networking set up right on the host?

  6. #6
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 2 NICs on KVM host

    Quote Originally Posted by aljames2 View Post
    At present, I have 2 NICs on the KVM host but neither support SR-IOV. I can run 2 cables between my dumb switch & the host but this doesn't seem to achieve the goal. I would have each VM on a separate NIC but there would be no isolation as I see it. The switch is not VLAN smart and cannot help control traffic. I prefer not to have a managed switch.

    @MAFoElffen, I did just order that PCIe Dell quad port device for the KVM host. I do know that my MB supports SR-IOV & IOMMU. Not sure I'll know how to use it when I get it, LOL... Will I even need to deal with using a VLAN if I can get this virtual networking set up right on the host?
    I figure with a KVM host, you can never have enough network ports to play with!!!

    VLANs -- Though I have trained on them and done them, while training for CISCO CCNA... I don't find a need for them here. Besides, then to get out of that VLAN to something else, then you have to route to there.

    For SR-IOV, at first I did via a vfio.conf file in modules, excluding... But then after doing that that way, I found I severely overthought that!!! It can be done a lot simpler and easier...

    First your find the ID of the NIC in lspci
    Code:
    mafoelffen@msi-ubuntu:~$ lspci -nnnk | grep -E 'Ethernet'
        DeviceName: Onboard - Ethernet
    06:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:125c] (rev 04)
    07:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
    Then in the GRUB_CMDLINE_LINUX_DEFAULT line of /etc/default/grub:
    Code:
    mafoelffen@msi-ubuntu:~$ grep 'GRUB_CMDLINE_LINUX_DEFAULT' /etc/default/grub
    GRUB_CMDLINE_LINUX_DEFAULT="-- splash intel_iommu=on iommu=pt igb.max_vfs=7"
    Then, after a reboot, I do 'ip a' to confirm it's split out, then do my netplan file
    Code:
    mafoelffen@msi-ubuntu:~$ grep . /etc/netplan/00-installer-config.yaml
    # This is the custom network config written by MAFoElffen
    network:
      version: 2
      renderer: networkd
      ethernets:
        enp6s0:
          dhcp4: false
          dhcp6: false
          addresses:
              - 10.0.0.5/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          routes:
              - to: default
                via: 10.0.0.1
                metric: 100
                on-link: true
          mtu: 1500
    ## Begin SR-IOV on enp7s0f0 
        enp7s0f0:
          virtual-function-count: 7
          dhcp4: false
          dhcp6: false
          optional: yes
        enp7s0f0v0:
          link: enp7s0f0
          addresses:
              - 10.0.0.10/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          dhcp4: false
          dhcp6: false
          optional: yes
        enp7s0f0v1:
          link: enp7s0f0
          addresses:
              - 10.0.0.11/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          dhcp4: false
          dhcp6: false
          optional: yes
        enp7s0f0v1:
          link: enp7s0f0
          addresses:
              - 10.0.0.11/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          dhcp4: false
          dhcp6: false
          optional: yes
        enp7s0f0v2:
          link: enp7s0f0
          addresses:
              - 10.0.0.12/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          dhcp4: false
          dhcp6: false
          optional: yes
        enp7s0f0v3:
          link: enp7s0f0
          addresses:
              - 10.0.0.13/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          dhcp4: false
          dhcp6: false
          optional: yes
        enp7s0f0v4:
          link: enp7s0f0
          addresses:
              - 10.0.0.14/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          dhcp4: false
          dhcp6: false
          optional: yes
        enp7s0f0v5:
          link: enp7s0f0
          addresses:
              - 10.0.0.15/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          dhcp4: false
          dhcp6: false
          optional: yes
        enp7s0f0v6:
          link: enp7s0f0
          addresses:
              - 10.0.0.16/8
          nameservers:
              addresses: [8.8.8.8, 8.8.4.4]
          dhcp4: false
          dhcp6: false
          optional: yes
    ## End SR-IOV on enp7s0f0
    ## Begin Bridge on enp7s0f1
        enp7s0f1:
          dhcp4: false
          dhcp6: false
          optional: yes
      bridges:
        br0:
          interfaces: [enp7s0f1]
          addresses: [10.0.0.6/8]
          routing-policy:
             - from: 10.0.0.6
               table: 10
          # ** ADDITIONAL ROUTING POLICY **
             - to: 172.16.1.0/24
               table: 172
          # *******************************
          routes:
             - to: 0.0.0.0/0
               via: 10.0.0.1
               table: 10
          nameservers:
            addresses: [8.8.8.8, 8.8.4.4, 1.1.1.1, 1.0.0.1]
          parameters:
            stp: true
            forward-delay: 4
          dhcp4: no
          dhcp6: no
          optional: yes
    ## End Bridge on enp7s0f1
    Then apply
    Code:
    sudo netplan generate
    sudo netplan apply
    Done.
    Last edited by MAFoElffen; January 17th, 2024 at 07:01 PM.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  7. #7
    Join Date
    Feb 2019
    Location
    Virginia
    Beans
    366
    Distro
    Xubuntu 22.04 Jammy Jellyfish

    Re: 2 NICs on KVM host

    Thanks for the details, this helps!
    That new… (new to me)… card arrived today, I will be working on this very soon. The extra ports will come in handy because I do want to add a few more servers to this host.

  8. #8
    Join Date
    Feb 2019
    Location
    Virginia
    Beans
    366
    Distro
    Xubuntu 22.04 Jammy Jellyfish

    Re: 2 NICs on KVM host

    An update

    I received the Dell I350-T4 Quad port card I bought, I thought it was new, at least it was advertised as such, but it wasn't. Anyhow, after running lspci, I noticed it did not show the SR-IOV capability. I made sure the IOMMU & SR-IOV setting was enabled on the MB as well as in grub. After some reading on the Intel website, I learned that some of the Dell OEM cards are know to have the SR-IOV disabled on the card. In particular the Gigabit cards. I think there is another version, perhaps I350-T4V2 that is fine. If you get the Intel retail version of the card then it comes with SR-IOV enabled. I think Dell figured users of a gigabit card would not need SR-IOV. Perhaps they are right. I shipped that card back.

    I had another card lying around that I had thought a few years ago wasn't working but turns out it does work. I guess I didn't know what I was doing with it previously It is a Intel Gigabit ET 82576 Quad Port server adapter. At first, I had no lights on any of the ports, so I figured my original theory of a broken card was true. Then I found a "Gen" setting in my Asus Bios for the PCIe 16_2 x4 slot. The default setting was Auto which didn't work. I knew this older card was likely not Gen-3 or Gen-4, so I set it to Gen-2 and voila, the card works. I have the VFs set up now & so far so good.

    Wondering if I should let the Host be assigned by itself to the MB integrated port, and put the VMs all on this card, that's what I am thinking..

  9. #9
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 2 NICs on KVM host

    Good job. It may help others here to share what settings and changes you made to your /etc/default/grub and /etc/netplan/<Named>.yaml file so they can do the same...

    Just a thought.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  10. #10
    Join Date
    Feb 2019
    Location
    Virginia
    Beans
    366
    Distro
    Xubuntu 22.04 Jammy Jellyfish

    Re: 2 NICs on KVM host

    Quote Originally Posted by MAFoElffen View Post
    Good job. It may help others here to share what settings and changes you made to your /etc/default/grub and /etc/netplan/<Named>.yaml file so they can do the same...
    Thanks, I do plan to share all I’ve done here soon. I am still trying to sort out getting my PCIe nic to pass through. My issue is an error that virt-manager gives about some devices in the same IOMMU group need to be bound to a VFIO driver. It is IOMMU group# 15. I have 18 devices all lumped in this group. Some of which are upstream root based devices, such as a few cpu & bridge functions. The root devices are not the issue, but all of my nics, VFs, & an onboard NVMe are also in this group. So I either need to pass all the non-root devices through to the VM which I don’t want, or somehow detach the device I want to pass. I have read about overriding ACU which eliminates isolation and is a last resort.

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •