Page 1 of 2 12 LastLast
Results 1 to 10 of 19

Thread: Weird Behavior with Bridged network

  1. #1
    Join Date
    Sep 2013
    Beans
    21

    Weird Behavior with Bridged network

    I have kinda a long story. TLDR is that I had a working network bridge to share with vnets from KVM instances, but now it doesn't work in strange ways.

    I have an Ubuntu 20.04 (upgraded over the years from much earlier versions) server running file shares, plex, and various other services to my home and wide network. The server uses an ethernet connection directly to the router. Other devices on the network are working with the internet properly regardless of the issues I'm having on this server.

    Originally this was just a file/plex server, but I started playing with KVM, GPU Passthrough, and Home Assistant virtual machines. The KVM with GPU Passthrough is an experiment to see if I can create a headless steam gaming server (like Stadia). To get the various SteamLink hardware to talk to that VM, I needed to create a network bridge which bridged the physical NIC with the virtual ones. I set this up and it worked great for months, no problems.

    Code:
    brctl addbr br0
    brctl addif enp12s0 vnet0 vnet1
    Then updated my /etc/network/interfaces file. The br0 used DHCP

    Then last friday I decided I wanted to set up a pihole on this same server. This required changing things to that I had a static address. I went and downloaded/installed the pihole script from their website, and got everything set up. Seemed good until a reboot, then I had no network connectivity. Changing the br0 to be DHCP again restored function, but obviously wouldn't work for the pihole. I fought with this for a few days then decided to give up, and I removed the PiHole software from the server using its built in script.

    Problem is, throughout my troubleshooting, I've somehow configured it so that whenever the br0 is up, I lose complete connectivity to the internet. After much research, I thought the issue was ip routes. So today I messed around with ip routing (deleted default route which used my physical interface, so the only route to the gateway was through the bridge). This seemed to fix internet connectivity... but weirdly enough I cannot ping my gateway, and no devices on the local network can see the server. In spite of this, Plex (which phones home for ease of use outside the local network) works just fine, and I can browse the internet from the server with no problem. Nothing can see the server from the LAN, but the server can talk to WAN through the same gateway, even though it can't ping the gateway.

    Another possibly related issue is that even when I completely delete the br0 gateway, like a zombie it will reappear on reboot or restart of networking service. This is after changing all the VMs to use NAT (which doesn't work for my purposes) instead of br0, and removing it from the interfaces file, and using brctl to delete it. It somehow keeps coming back. I think that this must be because some leftover config from pihole must reference it and keeps resurrecting it. In fact, all these issues might relate back to pihole configs.

    Any advice would be appreciated. I've been pulling my hair out all weekend trying to find a solution. Since its a network drive primarily, not having access within my home is a big deal. Thanks in advance for any ideas.

    EDIT: I thought maybe I had some conflicting network mangers messing with things, so I removed ifupdown and tried to set this up with netplan. Bridge created with netplan has the exact same behavior. Here's my netplan yaml

    Code:
    network:
      version: 2
      renderer: networkd
      ethernets:
        enp12s0:
          dhcp4: true
          dhcp6: true
      bridges:
        br0:
          interfaces: [enp12s0, vnet0, vnet1]
          parameters:
            stp: true
            forward-delay: 4
          addresses: [ 192.168.1.4/24 ]
      vlans:
        vnet0:
          accept-ra: no
          id: 0
          link: enp12s0
        vnet1:
          accept-ra: no
          id: 1
          link: enp12s0
    I've tried this config without enp12s0 as a bridge interface (as the example yaml here https://netplan.io/examples/ for virtio bridge states), but that just results in no network connection for the VMs.
    Last edited by crono1412; June 21st, 2021 at 10:53 PM.

  2. #2
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Weird Behavior with Bridged network

    Less explanation and more config files and commands showing the facts, please.

  3. #3
    Join Date
    Sep 2013
    Beans
    21

    Re: Weird Behavior with Bridged network

    Well, what would you like to see?

  4. #4
    Join Date
    Sep 2013
    Beans
    21

    Re: Weird Behavior with Bridged network

    Here is the current configuration, netplan.

    Code:
    master@HomeServer:/etc/netplan$ sudo netplan get all
    network:
      bridges:
        br0:
          addresses:
          - 192.168.1.4/24
          dhcp4: false
          gateway4: 192.168.1.1
          interfaces:
          - enp12s0
          nameservers:
            addresses:
            - 192.168.1.1
            - 8.8.8.8
          parameters:
            forward-delay: 4
            stp: true
      ethernets:
        enp12s0:
          dhcp4: false
          dhcp6: false
      renderer: networkd
      version: 2
    With this configuration, I can access the internet, and the VMs get the IP addresse that it should get (192.168.1.5). But I cannot ping the gateway (192.168.1.1) from the server, and I cannot ping the server (192.168.1.4) from other devices on the same network. I also cannot ping the VMs from the host (192.168.1.4 to 192.168.1.5).

    Here's the ifconfig
    Code:
    br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 192.168.1.4  netmask 255.255.255.0  broadcast 192.168.1.255
            inet6 fe80::d607:2828:d728:82d1  prefixlen 64  scopeid 0x20<link>
            ether 0e:8a:7f:b5:d9:c9  txqueuelen 1000  (Ethernet)
            RX packets 38143  bytes 9269734 (9.2 MB)
            RX errors 0  dropped 488  overruns 0  frame 0
            TX packets 8111  bytes 1326416 (1.3 MB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    
    enp12s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 192.168.1.4  netmask 255.255.255.0  broadcast 192.168.1.255
            ether 0e:8a:7f:b5:d9:c9  txqueuelen 1000  (Ethernet)
            RX packets 39845  bytes 10076494 (10.0 MB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 14534  bytes 1817361 (1.8 MB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    
    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10<host>
            loop  txqueuelen 1000  (Local Loopback)
            RX packets 13062  bytes 1903783 (1.9 MB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 13062  bytes 1903783 (1.9 MB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    
    tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
            inet 10.8.0.1  netmask 255.255.255.255  destination 10.8.0.2
            inet6 fe80::fe5b:e1a6:8022:a25  prefixlen 64  scopeid 0x20<link>
            unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
            RX packets 0  bytes 0 (0.0 B)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 183  bytes 21534 (21.5 KB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    
    vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 169.254.165.48  netmask 255.255.0.0  broadcast 169.254.255.255
            inet6 fe80::fc54:ff:fede:61b5  prefixlen 64  scopeid 0x20<link>
            inet6 fe80::f7b0:e9a1:6eb1:6097  prefixlen 64  scopeid 0x20<link>
            ether fe:54:00:de:61:b5  txqueuelen 1000  (Ethernet)
            RX packets 358  bytes 40923 (40.9 KB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 9295  bytes 1147353 (1.1 MB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    ip route:
    Code:
    default via 192.168.1.1 dev br0 proto static 
    default via 192.168.1.1 dev enp12s0 src 192.168.1.4 metric 202 
    default via 192.168.1.1 dev br0 src 192.168.1.4 metric 203 
    10.8.0.0/24 via 10.8.0.2 dev tun0 
    10.8.0.2 dev tun0 proto kernel scope link src 10.8.0.1 
    169.254.0.0/16 dev vnet0 scope link src 169.254.165.48 metric 209 
    192.168.1.0/24 dev enp12s0 proto dhcp scope link src 192.168.1.4 metric 202 
    192.168.1.0/24 dev br0 proto dhcp scope link src 192.168.1.4 metric 203
    I moved my /etc/network/interfaces file to interfaces.old

    What else would you like to see?
    Last edited by crono1412; June 22nd, 2021 at 12:00 AM.

  5. #5
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Weird Behavior with Bridged network

    The netplan, the routing table, the ip address for the VM host, the ip address for each guest VM ... for starters.

    Whenever a VM is involved, the VM host shouldn't use DHCP addresses. It should have a static IP.

    I don't see any gateway set or DNS servers. There are example netplan files in these forums. I posted one in the last week which immediately fixed someone else's problem. Did you see that one? He was trying to use DHCP too. Bad idea.

  6. #6
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Weird Behavior with Bridged network

    tun? Nobody mentioned having a VPN running before.

    enp12s0 shouldn't have any IP address assigned.

    route -n is easier to read.
    or use
    ip r | column -t
    to make is readable.

  7. #7
    Join Date
    Sep 2013
    Beans
    21

    Re: Weird Behavior with Bridged network

    Code:
    master@HomeServer:/etc/netplan$ route -nKernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    0.0.0.0         192.168.1.1     0.0.0.0         UG    0      0        0 br0
    0.0.0.0         192.168.1.1     0.0.0.0         UG    202    0        0 enp12s0
    0.0.0.0         192.168.1.1     0.0.0.0         UG    203    0        0 br0
    10.8.0.0        10.8.0.2        255.255.255.0   UG    0      0        0 tun0
    10.8.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
    169.254.0.0     0.0.0.0         255.255.0.0     U     209    0        0 vnet0
    192.168.1.0     0.0.0.0         255.255.255.0   U     202    0        0 enp12s0
    192.168.1.0     0.0.0.0         255.255.255.0   U     203    0        0 br0
    The tun is ancient and hasn't been used. I put openVPN on here some months/years ago, and it didn't work out. Guess I never bothered to remove it. Removed now, no change in behavior. Here's the new ifconfig.

    Code:
    br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 192.168.1.4  netmask 255.255.255.0  broadcast 192.168.1.255
            inet6 fe80::d607:2828:d728:82d1  prefixlen 64  scopeid 0x20<link>
            ether 0e:8a:7f:b5:d9:c9  txqueuelen 1000  (Ethernet)
            RX packets 74179  bytes 15878954 (15.8 MB)
            RX errors 0  dropped 1063  overruns 0  frame 0
            TX packets 13991  bytes 2474919 (2.4 MB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    
    enp12s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 192.168.1.4  netmask 255.255.255.0  broadcast 192.168.1.255
            ether 0e:8a:7f:b5:d9:c9  txqueuelen 1000  (Ethernet)
            RX packets 76512  bytes 17284412 (17.2 MB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 24827  bytes 3283681 (3.2 MB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    
    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10<host>
            loop  txqueuelen 1000  (Local Loopback)
            RX packets 20599  bytes 3241216 (3.2 MB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 20599  bytes 3241216 (3.2 MB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    
    vnet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 169.254.165.48  netmask 255.255.0.0  broadcast 169.254.255.255
            inet6 fe80::fc54:ff:fede:61b5  prefixlen 64  scopeid 0x20<link>
            inet6 fe80::f7b0:e9a1:6eb1:6097  prefixlen 64  scopeid 0x20<link>
            ether fe:54:00:de:61:b5  txqueuelen 1000  (Ethernet)
            RX packets 787  bytes 76676 (76.6 KB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 34269  bytes 4274544 (4.2 MB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    The vnet address in ifconfig doesn't make too much sense to me, as the VM reports its address as 192.168.1.5, and my router also reflects that. I've found while troubleshooting this issue that ifconfig doesn't always report the whole truth.

    How do I prevent enp12s0 from getting an IP address? Since this is the physical link, doesn't it need an address?

    EDIT: found your post you mentioned. The only material difference between yours and mine is setting dhcp6 to false. I did that and regenerated/applied the netplan. No change. For the first 5 seconds or so after changing the config I can reach my routers landing page at 192.168.1.1, but as soon as the bridge takes over it becomes unreachable. I also disabled network manager because I hear it can cause problems with non-standard network configs.

    Code:
    master@HomeServer:/etc/netplan$ ping 192.168.1.1
    PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
    From 192.168.1.4 icmp_seq=1 Destination Host Unreachable
    From 192.168.1.4 icmp_seq=2 Destination Host Unreachable
    ^C
    --- 192.168.1.1 ping statistics ---
    3 packets transmitted, 0 received, +2 errors, 100% packet loss, time 2007ms
    pipe 2
    master@HomeServer:/etc/netplan$ ping www.google.com
    PING www.google.com (172.217.164.68) 56(84) bytes of data.
    64 bytes from atl26s18-in-f4.1e100.net (172.217.164.68): icmp_seq=1 ttl=57 time=9.01 ms
    64 bytes from atl26s18-in-f4.1e100.net (172.217.164.68): icmp_seq=2 ttl=57 time=9.23 ms
    64 bytes from atl26s18-in-f4.1e100.net (172.217.164.68): icmp_seq=3 ttl=57 time=9.21 ms
    ^C
    --- www.google.com ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2003ms
    rtt min/avg/max/mdev = 9.010/9.147/9.226/0.097 ms
    Last edited by crono1412; June 22nd, 2021 at 12:38 AM.

  8. #8
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    2,963
    Distro
    Ubuntu Development Release

    Re: Weird Behavior with Bridged network

    Quote Originally Posted by crono1412 View Post
    How do I prevent enp12s0 from getting an IP address? Since this is the physical link, doesn't it need an address?
    Good question. It isn't clear to me why it is getting an IP address. I agree with TheFu, it shouldn't. And, no it doesn't need an address, br0 gets the address.

    What do you get for example this:

    Code:
    doug@s19:~$ brctl show br0
    bridge name     bridge id               STP enabled     interfaces
    br0             8000.3c7c3f0d9983       no              enp3s0
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  9. #9
    Join Date
    Sep 2013
    Beans
    21

    Re: Weird Behavior with Bridged network

    Here it is.

    Code:
    brctl showbridge name    bridge id        STP enabled    interfaces
    br0        8000.0e8a7fb5d9c9    yes        enp12s0
                                vnet0

  10. #10
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Weird Behavior with Bridged network

    Quote Originally Posted by crono1412 View Post
    EDIT: found your post you mentioned. The only material difference between yours and mine is setting dhcp6 to false. I did that and regenerated/applied the netplan. No change. For the first 5 seconds or so after changing the config I can reach my routers landing page at 192.168.1.1, but as soon as the bridge takes over it becomes unreachable. I also disabled network manager because I hear it can cause problems with non-standard network configs.
    In theory, that is correct, but sometimes order matters.

    Humor me. Put that config in (fixed for your NIC and subnet) and see if it works.
    I remember something about the renderer being 2nd as important. Yes, it shouldn't matter, but then there is the real world.

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •