Page 2 of 2 FirstFirst 12
Results 11 to 13 of 13

Thread: 2 NICs on KVM host

  1. #11
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 2 NICs on KVM host

    I do not pass my NICs through... I setup virtual networks in KVM, and use those virtual network switches from the guests.

    RE:
    https://www.linux-kvm.org/page/Networking
    https://wiki.qemu.org/Documentation/Networking
    https://help.ubuntu.com/community/KVM/Networking

    Then, for example, I create a DD-WRT and PFsense VM router appliances, that I can route between those virtual networks, or an Ubuntu Server VM, that I can use as a Router appliance.

    I get a bit creative in replicating virtual network schemes to satisfy what my testcases are or what I am trying to accomplish.

    KVM/Qemu makes many things possible in your virtual world, if you use your imagination. The idea is to create things virtually, which would normally be in real hardware. Just because you only have one switch and router "physically", should not limit what you can do within your virtual world. Those (now_ extra 4 physical ports) now extend how your Virtual host can communicate outside that virtual network.

    The easiest way to use a NIC port as dedicated is to set it as a bridged port device.
    Last edited by MAFoElffen; January 30th, 2024 at 04:31 PM.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

  2. #12
    Join Date
    Feb 2019
    Location
    Virginia
    Beans
    366
    Distro
    Xubuntu 22.04 Jammy Jellyfish

    Re: 2 NICs on KVM host

    I gave up on trying to do PCI passthrough. Learned my lesson once and for all, never again to try that. I can explain in detail why, but it wouldn't apply to everyone, some have had success but not always possible depending on your hardware..

    I ended up making a setup on this subnet that does make use of my extra quad-port ethernet card and the SR-IOV feature it supports. The sr-iov essentially makes up to 8 nics out of each physical nic, 32 total. Althought, I have only initialized 2 virtual nics on 1 physical nic for now, can easily initialize more as needed.

    Since I won't be doing PCI passthrough to my VM, I will add an SR-IOV network adapter to a KVM VM as a Virtual Function (VF) network adapter connected to a macvtap on the host.

    Some of this inspired from here:
    https://www.intel.com/content/www/us...linux-kvm.html

    And some clues from @MAFoElffen's netplan shared earlier, thanks again !

    Some facts:
    - Hardware is AMD 5600x, on Asus B550-A Gaming mobo, supports IOMMU, SR-IOV, & SVM
    - Ubuntu 22.04 KVM host
    - My Wireguard VM started out running on my KVM bridge, same as my Host & NC server
    - Goal to move WG server to its own physical interface, and have fun doing it
    - This process is editorialized without all the outputs, instead explanation of what you should see, not always what my output was. YMMV

    Here is what I did:

    Code:
    # Shut down computer
    # Installed quad port nic, no wire attached yet  (WG server still on br0 for now)
    # Start computer
    
    ip a  (mobo NIC name changed as expected with the introduction of another adapter)
    
    # See 4 new NIC ports and their names
    
    # edit /etc/netplan/kvm-host.yaml
    # ...correct my ethernet name definition in order to bring up the host network again
    
    sudo netplan generate
    sudo netplan apply
    reboot  (for good measure)
    
    ip a  (Now our mobo nic is back up)
    
    ----------------------------
    
    # Restart computer & enter MB BIOS (F2 or DEL)
    # Enable:  SR-IOV, IOMMU, & SVM
    # In Advanced, Go to "Onboard Connections Configuration", & set PCIe 16_2 slot to "Gen 3"  (Auto did not work)
    # Save & restart computer
    
    ----------------------------
    
    # Now, check if this old/(new to me), NIC supports SR-IOV
    
    sudo lshw -c network -businfo
    sudo lspci -vs 0000:06:00.0  (using the pci address from previous command)
    
    # Look for something like this in the output:
    Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
    
    # Do this lspci for each NIC port address to be sure each are SR-IOV capable
    # Assuming our device supports SR-IOV, continue...
    
    ----------------------------
    
    # Edit /etc/default/grub
    # Add these within the quotes: amd_iommu=on iommu=pt
     (my line has some extra stuff and looks like the following)
    
    GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 amd_iommu=on iommu=pt"
    
    sudo update-grub
    
    # Attach ethernet wire between my chosen NIC port & the switch
    
    reboot
    
    # Check our switch & interface, lights should be flashing. If no lights something is wrong.
    
    ip a  (make note of our top port name: enp6s0f0)
    
    ----------------------------
    
    # We will now initialize SR-IOV VFs (Virtual Functions) on this port we chose.
    
    # become root
    sudo -i
    
    cd /sys/class/net
    ls -al  (see our device names)
    cd ./enp6s0f0/device
    ls -al  (find sriov_numvfs file)
    cat sriov_numvfs  (current value should be 0)
    
    echo 2 > sriov_numvfs
    # I chose to only initialize 2 VFs for now.
    # This will manually initialize 2 VFs on this port, but they won't survive boot, so we'll use our new output to write a new netplan file which will make the virtual function definitions permanent.
    
    cat sriov_numvfs  (value should now be 2)
    exit  (root)
    
    ip a  (notice our 2 virtual nics now exist on this physical port & each with its own MAC address)
        enp6s0f0v0
        enp6s0f0v1
    
    sudo lshw -c network -businfo  (another look to see our VFs each have a different bus address as well)
    
    # Edit my netplan file
    cd /etc/netplan
    
    # Save a copy of the original file first
    sudo cp kvm-host.yaml kvm-host.yaml.bak
    
    # My netplan file is this:
    
    ----------------------------
    
    #  [ for KVM + bridge + SR-IOV ]
    network:
      version: 2
      renderer: networkd
    # Define our Host NIC - bridge will use this NIC below
      ethernets:
        enp12s0:
          dhcp4: false
          dhcp6: false
    # Define our SR-IOV NIC (Physical Function)
        enp6s0f0:
          virtual-function-count: 2
          dhcp4: false
          dhcp6: false
          optional: yes
    # Define our SR-IOV VFs (Virtual Functions) on enp6s0f0
        enp6s0f0v0:
          link: enp6s0f0
          dhcp4: false
          dhcp6: false
          optional: yes
        enp6s0f0v1:
          link: enp6s0f0
          dhcp4: false
          dhcp6: false
          optional: yes
    # End SR-IOV Definitions
      bridges:
        br0:
          interfaces: [enp12s0]
          addresses: [10.11.37.81/24]
          routes:
          - to: default
            via: 10.11.37.1
            metric: 100
            on-link: true
          mtu: 1500
          nameservers:
            addresses: [10.11.37.1]
          parameters:
            stp: true
            forward-delay: 4
          dhcp4: no
          dhcp6: no
    
    ----------------------------
    
    sudo netplan generate
    sudo netplan apply
    
    ip a  (see our VFs are now set with IPs)
    
    reboot
    
    ----------------------------
    
    sudo lshw -c network -businfo
    
    Bus info          Device          Class       Description
    =========================================================
    pci@0000:06:00.0  enp6s0f0        network     82576 Gigabit Network Connection
    pci@0000:06:00.1  enp6s0f1        network     82576 Gigabit Network Connection
    pci@0000:08:00.0  enp8s0f0        network     82576 Gigabit Network Connection
    pci@0000:08:00.1  enp8s0f1        network     82576 Gigabit Network Connection
    pci@0000:0c:00.0  enp12s0         network     Ethernet Controller I225-V
    pci@0000:07:10.0  enp6s0f0v0      network     82576 Virtual Function
    pci@0000:07:10.2  enp6s0f0v1      network     82576 Virtual Function
    
    
    # Find our VF interface name we want to use & make a note of it
    # I chose: enp6s0f0v0
    
    # Since my VM is running on the bridge we will be editing our VM configuration to move it over to our new Virtual Function on our new interface.
    
    virsh list --all  (shutdown VM if running)
    
    virsh edit ubuntu-server2204_WG
    
    # In my case, I changed this excert to match the following:
    
    ----------------------------
    
    <devices>
    …
       <interface type='direct'>
          <source dev='enp6s0f0v0' mode='passthrough'/>
       </interface>
    …
    </devices>
    
    ----------------------------
    
    # Launch virt-manager & start the VM
    
    # If errors, check:
    # If no errors, also check these virtual nic settings in virt-manager:
        - Network source shows macvtap
        - Device name is our VF name (enp6s0f0v0)
        - Our MAC address exists
        - Device model is virtio
    
    # As the VM starts, KVM creates a macvtap adapter ‘macvtap0’ on the VF specified.
    # I had no errors but these are things to check even if working.
    
    ip a  (on the host shows)
    
    11: macvtap0@enp6s0f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 500....(including it's mac address)
    
    # It works for me
    My diagram is something like this now:
    Code:
                    WAN
                    |
                 ___|_________
                |  pfSense    |
                |   Router    |
                |_____(en01)__|
                  | |   |   |
               ___| |   |   |___
              ______|   |
                        |
                        |10.11.37.0 (trunk subnet)
                        |
                    ____|_____
                   | switch   |
                   |__________|(dumb/unmanaged sw)
                      |     |
                      |     |
             (enp12s0)|     |(enp6s0f0v0 VF of enp6s0f0)
                      |     |
                    __|_____|___
                   | KVM Host   |
                   |            |
                   |_nic1__nic2_|
                      |     |
                      |     |
                      |     |
        10.11.37.81 [br0]  [macvtap passthrough to SR-IOV VF on physical nic]
                      |     |
                      |     |
                 (NC VM)   (WG-vpn VM)
             10.11.37.10     10.11.37.117

  3. #13
    Join Date
    Mar 2010
    Location
    USA
    Beans
    Hidden!
    Distro
    Ubuntu Development Release

    Re: 2 NICs on KVM host

    Good job.

    Yes. I had to go through my own trials and errors with different combinations, until I found what worked best for me, and with what I do.

    I found that passing through the NIC, is not what I want to do, as it is all one IOMMU group, so then you lose it all from the host side, and made it less valuable and flexible to me, in what I could with it from the host side, and creating differing virtual network appliances using the differing host ports that were available.

    That is why I don't do the SR-IOV virtual network scheme inside KVM. I lost a lot of flexibility of my resources with that option. Just because it is possible, didn't make that a good idea. Lessons learned.

    I'm still playing with that in different ways, to see what is possible, and creates the best value for me. Time will tell.
    Last edited by MAFoElffen; February 7th, 2024 at 04:48 PM.

    "Concurrent coexistence of Windows, Linux and UNIX..." || Ubuntu user # 33563, Linux user # 533637
    Sticky: Graphics Resolution | UbuntuForums 'system-info' Script | Posting Guidelines | Code Tags

Page 2 of 2 FirstFirst 12

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •