Weird Behavior with Bridged network

[crono1412]

I have kinda a long story. TLDR is that I had a working network bridge to share with vnets from KVM instances, but now it doesn't work in strange ways.

I have an Ubuntu 20.04 (upgraded over the years from much earlier versions) server running file shares, plex, and various other services to my home and wide network. The server uses an ethernet connection directly to the router. Other devices on the network are working with the internet properly regardless of the issues I'm having on this server.

Originally this was just a file/plex server, but I started playing with KVM, GPU Passthrough, and Home Assistant virtual machines. The KVM with GPU Passthrough is an experiment to see if I can create a headless steam gaming server (like Stadia). To get the various SteamLink hardware to talk to that VM, I needed to create a network bridge which bridged the physical NIC with the virtual ones. I set this up and it worked great for months, no problems.

brctl addbr br0
brctl addif enp12s0 vnet0 vnet1

Then updated my /etc/network/interfaces file. The br0 used DHCP

Then last friday I decided I wanted to set up a pihole on this same server. This required changing things to that I had a static address. I went and downloaded/installed the pihole script from their website, and got everything set up. Seemed good until a reboot, then I had no network connectivity. Changing the br0 to be DHCP again restored function, but obviously wouldn't work for the pihole. I fought with this for a few days then decided to give up, and I removed the PiHole software from the server using its built in script.

Problem is, throughout my troubleshooting, I've somehow configured it so that whenever the br0 is up, I lose complete connectivity to the internet. After much research, I thought the issue was ip routes. So today I messed around with ip routing (deleted default route which used my physical interface, so the only route to the gateway was through the bridge). This seemed to fix internet connectivity... but weirdly enough I cannot ping my gateway, and no devices on the local network can see the server. In spite of this, Plex (which phones home for ease of use outside the local network) works just fine, and I can browse the internet from the server with no problem. Nothing can see the server from the LAN, but the server can talk to WAN through the same gateway, even though it can't ping the gateway.

Another possibly related issue is that even when I completely delete the br0 gateway, like a zombie it will reappear on reboot or restart of networking service. This is after changing all the VMs to use NAT (which doesn't work for my purposes) instead of br0, and removing it from the interfaces file, and using brctl to delete it. It somehow keeps coming back. I think that this must be because some leftover config from pihole must reference it and keeps resurrecting it. In fact, all these issues might relate back to pihole configs.

Any advice would be appreciated. I've been pulling my hair out all weekend trying to find a solution. Since its a network drive primarily, not having access within my home is a big deal. Thanks in advance for any ideas.

EDIT: I thought maybe I had some conflicting network mangers messing with things, so I removed ifupdown and tried to set this up with netplan. Bridge created with netplan has the exact same behavior. Here's my netplan yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    enp12s0:
      dhcp4: true
      dhcp6: true
  bridges:
    br0:
      interfaces: [enp12s0, vnet0, vnet1]
      parameters:
        stp: true
        forward-delay: 4
      addresses: [ 192.168.1.4/24 ]
  vlans:
    vnet0:
      accept-ra: no
      id: 0
      link: enp12s0
    vnet1:
      accept-ra: no
      id: 1
      link: enp12s0

I've tried this config without enp12s0 as a bridge interface (as the example yaml here https://netplan.io/examples/ for virtio bridge states), but that just results in no network connection for the VMs.

[TheFu]

Less explanation and more config files and commands showing the facts, please.

[crono1412]

Well, what would you like to see?

[crono1412]

Here is the current configuration, netplan.

master@HomeServer:/etc/netplan$ sudo netplan get all
network:
  bridges:
    br0:
      addresses:
      - 192.168.1.4/24
      dhcp4: false
      gateway4: 192.168.1.1
      interfaces:
      - enp12s0
      nameservers:
        addresses:
        - 192.168.1.1
        - 8.8.8.8
      parameters:
        forward-delay: 4
        stp: true
  ethernets:
    enp12s0:
      dhcp4: false
      dhcp6: false
  renderer: networkd
  version: 2

With this configuration, I can access the internet, and the VMs get the IP addresse that it should get (192.168.1.5). But I cannot ping the gateway (192.168.1.1) from the server, and I cannot ping the server (192.168.1.4) from other devices on the same network. I also cannot ping the VMs from the host (192.168.1.4 to 192.168.1.5).

Here's the ifconfig

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.4  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::d607:2828:d728:82d1  prefixlen 64  scopeid 0x20<link>
        ether 0e:8a:7f:b5:d9:c9  txqueuelen 1000  (Ethernet)
        RX packets 38143  bytes 9269734 (9.2 MB)
        RX errors 0  dropped 488  overruns 0  frame 0
        TX packets 8111  bytes 1326416 (1.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


enp12s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.4  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 0e:8a:7f:b5:d9:c9  txqueuelen 1000  (Ethernet)
        RX packets 39845  bytes 10076494 (10.0 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14534  bytes 1817361 (1.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 13062  bytes 1903783 (1.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13062  bytes 1903783 (1.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 10.8.0.1  netmask 255.255.255.255  destination 10.8.0.2
        inet6 fe80::fe5b:e1a6:8022:a25  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 183  bytes 21534 (21.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.165.48  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::fc54:ff:fede:61b5  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::f7b0:e9a1:6eb1:6097  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:de:61:b5  txqueuelen 1000  (Ethernet)
        RX packets 358  bytes 40923 (40.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9295  bytes 1147353 (1.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ip route:

default via 192.168.1.1 dev br0 proto static 
default via 192.168.1.1 dev enp12s0 src 192.168.1.4 metric 202 
default via 192.168.1.1 dev br0 src 192.168.1.4 metric 203 
10.8.0.0/24 via 10.8.0.2 dev tun0 
10.8.0.2 dev tun0 proto kernel scope link src 10.8.0.1 
169.254.0.0/16 dev vnet0 scope link src 169.254.165.48 metric 209 
192.168.1.0/24 dev enp12s0 proto dhcp scope link src 192.168.1.4 metric 202 
192.168.1.0/24 dev br0 proto dhcp scope link src 192.168.1.4 metric 203

I moved my /etc/network/interfaces file to interfaces.old

What else would you like to see?

[TheFu]

The netplan, the routing table, the ip address for the VM host, the ip address for each guest VM ... for starters.

Whenever a VM is involved, the VM host shouldn't use DHCP addresses. It should have a static IP.

I don't see any gateway set or DNS servers. There are example netplan files in these forums. I posted one in the last week which immediately fixed someone else's problem. Did you see that one? He was trying to use DHCP too. Bad idea.

[TheFu]

tun? Nobody mentioned having a VPN running before.

enp12s0 shouldn't have any IP address assigned.

route -n is easier to read.
or use
ip r | column -t
to make is readable.

[crono1412]

master@HomeServer:/etc/netplan$ route -nKernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 br0
0.0.0.0         192.168.1.1     0.0.0.0         UG    202    0        0 enp12s0
0.0.0.0         192.168.1.1     0.0.0.0         UG    203    0        0 br0
10.8.0.0        10.8.0.2        255.255.255.0   UG    0      0        0 tun0
10.8.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
169.254.0.0     0.0.0.0         255.255.0.0     U     209    0        0 vnet0
192.168.1.0     0.0.0.0         255.255.255.0   U     202    0        0 enp12s0
192.168.1.0     0.0.0.0         255.255.255.0   U     203    0        0 br0

The tun is ancient and hasn't been used. I put openVPN on here some months/years ago, and it didn't work out. Guess I never bothered to remove it. Removed now, no change in behavior. Here's the new ifconfig.

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.4  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::d607:2828:d728:82d1  prefixlen 64  scopeid 0x20<link>
        ether 0e:8a:7f:b5:d9:c9  txqueuelen 1000  (Ethernet)
        RX packets 74179  bytes 15878954 (15.8 MB)
        RX errors 0  dropped 1063  overruns 0  frame 0
        TX packets 13991  bytes 2474919 (2.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


enp12s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.4  netmask 255.255.255.0  broadcast 192.168.1.255
        ether 0e:8a:7f:b5:d9:c9  txqueuelen 1000  (Ethernet)
        RX packets 76512  bytes 17284412 (17.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 24827  bytes 3283681 (3.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 20599  bytes 3241216 (3.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 20599  bytes 3241216 (3.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.165.48  netmask 255.255.0.0  broadcast 169.254.255.255
        inet6 fe80::fc54:ff:fede:61b5  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::f7b0:e9a1:6eb1:6097  prefixlen 64  scopeid 0x20<link>
        ether fe:54:00:de:61:b5  txqueuelen 1000  (Ethernet)
        RX packets 787  bytes 76676 (76.6 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34269  bytes 4274544 (4.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The vnet address in ifconfig doesn't make too much sense to me, as the VM reports its address as 192.168.1.5, and my router also reflects that. I've found while troubleshooting this issue that ifconfig doesn't always report the whole truth.

How do I prevent enp12s0 from getting an IP address? Since this is the physical link, doesn't it need an address?

EDIT: found your post you mentioned. The only material difference between yours and mine is setting dhcp6 to false. I did that and regenerated/applied the netplan. No change. For the first 5 seconds or so after changing the config I can reach my routers landing page at 192.168.1.1, but as soon as the bridge takes over it becomes unreachable. I also disabled network manager because I hear it can cause problems with non-standard network configs.

master@HomeServer:/etc/netplan$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
From 192.168.1.4 icmp_seq=1 Destination Host Unreachable
From 192.168.1.4 icmp_seq=2 Destination Host Unreachable
^C
--- 192.168.1.1 ping statistics ---
3 packets transmitted, 0 received, +2 errors, 100% packet loss, time 2007ms
pipe 2
master@HomeServer:/etc/netplan$ ping www.google.com
PING www.google.com (172.217.164.68) 56(84) bytes of data.
64 bytes from atl26s18-in-f4.1e100.net (172.217.164.68): icmp_seq=1 ttl=57 time=9.01 ms
64 bytes from atl26s18-in-f4.1e100.net (172.217.164.68): icmp_seq=2 ttl=57 time=9.23 ms
64 bytes from atl26s18-in-f4.1e100.net (172.217.164.68): icmp_seq=3 ttl=57 time=9.21 ms
^C
--- www.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 9.010/9.147/9.226/0.097 ms

[Doug S]

How do I prevent enp12s0 from getting an IP address? Since this is the physical link, doesn't it need an address?

Good question. It isn't clear to me why it is getting an IP address. I agree with TheFu, it shouldn't. And, no it doesn't need an address, br0 gets the address.

What do you get for example this:

doug@s19:~$ brctl show br0
bridge name     bridge id               STP enabled     interfaces
br0             8000.3c7c3f0d9983       no              enp3s0

[crono1412]

Here it is.

brctl showbridge name    bridge id        STP enabled    interfaces
br0        8000.0e8a7fb5d9c9    yes        enp12s0
                            vnet0

[TheFu]

EDIT: found your post you mentioned. The only material difference between yours and mine is setting dhcp6 to false. I did that and regenerated/applied the netplan. No change. For the first 5 seconds or so after changing the config I can reach my routers landing page at 192.168.1.1, but as soon as the bridge takes over it becomes unreachable. I also disabled network manager because I hear it can cause problems with non-standard network configs.

In theory, that is correct, but sometimes order matters.

Humor me. Put that config in (fixed for your NIC and subnet) and see if it works.
I remember something about the renderer being 2nd as important. Yes, it shouldn't matter, but then there is the real world.