20.04 Server, can't forward between vlans

**JamesMcT** · May 11th, 2020

I've managed to get the new AutoInstall method up and running, with MAAS and docker up and running, but I've noticed an odd routing problem.

To start off with I have two VLANS:

VLAN10: workstation traffic
IP space: 10.10.0.0/24

VLAN20: Node traffic
IP space 10.20.0.0/24

I'll explain the problem with three systems:

Workstation:
eth0 interface: 10.10.0.11

Server:
bond0 interface is the parent to:
vlan10 interface: 10.10.0.2
vlan20 interface: 10.20.0.1

Node:
eno1 interface: 10.20.0.3
eno1.10 interface: 10.10.0.22 (DHCP assigned from a server in VLAN10)

I am able to ping back and forth between the workstation and the server.
I am able to ping back and forth between the server and the node.
I am able to ping back and forth between the workstation and the 10.10.0.22 address on the node
I am **not** able to ping back and forth between the workstation and the 10.20.0.3 address on the node

The routing table on the server is straight forward:

Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.0.1 0.0.0.0 UG 100 0 0 vlan10
10.10.0.0 0.0.0.0 255.255.255.0 U 0 0 0 vlan10
10.10.0.1 0.0.0.0 255.255.255.255 UH 100 0 0 vlan10
10.20.0.0 0.0.0.0 255.255.255.0 U 0 0 0 vlan20
--- cut docker rules out ---

The routing table on the nodes are also straight forward

Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.0.1 0.0.0.0 UG 100 0 0 eno1.10
10.10.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1.10
10.10.0.1 0.0.0.0 255.255.255.255 UH 100 0 0 eno1.10
10.20.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1

--- cut docker rules out ---

The routing table on the workstation currently has a manual entry for 10.10.0.0/24 gw 10.10.0.2

I have checked and:

cat /proc/sys/net/ipv4/ip_forward
1

I run tcpdump -i bond0 icmp on the server and:
Ping from the workstation to the node, I see the icmp packets from 10.10.0.11 coming in, but nothing else
Ping from the node to the workstation, I see the icmp packets from 10.20.0.3 coming in, but nothing else

I think it's something on the server, the iptables setup by Docker doesn't seem to be blocking it:

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
DROP all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (2 references)
target prot opt source destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere

I'm missing something, I just have no idea what.

**sbuxhofer** · May 12th, 2020

I think your default UFW policy for forward is still set to "DROP" [Chain FORWARD (policy DROP)]

Either edit /etc/default/ufw and set DEFAULT_FORWARD_POLICY to ACCEPT:

Code:

DEFAULT_FORWARD_POLICY="ACCEPT"

or apply forwarding policies to allow traffic to pass from interface to interface.

**JamesMcT** · May 13th, 2020

The default FORWARD rule was set to "DROP", however UFW is inactive.

╰─$ sudo ufw status
Status: inactive

Docker has setup iptables. I didn't see any rules that would affect the forwarding from the bond0 interface. I aded the highlighted ones just to be sure. I'm not seeing anything in the logs.

╰─$ sudo iptables --list -v
Chain INPUT (policy ACCEPT 2227K packets, 716M bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy DROP 14524 packets, 1220K bytes)
pkts bytes target prot opt in out source destination
4686K 1501M DOCKER-USER all -- any any anywhere anywhere
4686K 1501M DOCKER-INGRESS all -- any any anywhere anywhere
4392K 1457M DOCKER-ISOLATION-STAGE-1 all -- any any anywhere anywhere
1709K 565M ACCEPT all -- any br-42e950ccd87b anywhere anywhere ctstate RELATED,ESTABLISHED
1502 90736 DOCKER all -- any br-42e950ccd87b anywhere anywhere
2343K 710M ACCEPT all -- br-42e950ccd87b !br-42e950ccd87b anywhere anywhere
0 0 ACCEPT all -- br-42e950ccd87b br-42e950ccd87b anywhere anywhere
0 0 ACCEPT all -- any br-f1cc37eb06a9 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any br-f1cc37eb06a9 anywhere anywhere
0 0 ACCEPT all -- br-f1cc37eb06a9 !br-f1cc37eb06a9 anywhere anywhere
0 0 ACCEPT all -- br-f1cc37eb06a9 br-f1cc37eb06a9 anywhere anywhere
0 0 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- any docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 !docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 docker0 anywhere anywhere
88197 19M ACCEPT all -- any docker_gwbridge anywhere anywhere ctstate RELATED,ESTABLISHED
2013 121K DOCKER all -- any docker_gwbridge anywhere anywhere
77445 125M ACCEPT all -- docker_gwbridge !docker_gwbridge anywhere anywhere
461K 193M ACCEPT all -- any br-962a6ca70854 anywhere anywhere ctstate RELATED,ESTABLISHED
6921 419K DOCKER all -- any br-962a6ca70854 anywhere anywhere
19577 25M ACCEPT all -- br-962a6ca70854 !br-962a6ca70854 anywhere anywhere
3956 237K ACCEPT all -- br-962a6ca70854 br-962a6ca70854 anywhere anywhere
0 0 DROP all -- docker_gwbridge docker_gwbridge anywhere anywhere
0 0 ACCEPT all -- bond0 any anywhere anywhere
0 0 ACCEPT all -- any bond0 anywhere anywhere
0 0 LOG all -- bond0 any anywhere anywhere LOG level warning
0 0 LOG all -- any bond0 anywhere anywhere LOG level warning

Chain OUTPUT (policy ACCEPT 1944K packets, 908M bytes)
pkts bytes target prot opt in out source destination

Chain DOCKER (5 references)
pkts bytes target prot opt in out source destination
2013 121K ACCEPT tcp -- !docker_gwbridge docker_gwbridge anywhere 172.18.0.3 tcp dpt:9001
2965 181K ACCEPT tcp -- !br-962a6ca70854 br-962a6ca70854 anywhere 172.19.0.2 tcp dpt:http-alt
1502 90736 ACCEPT tcp -- !br-42e950ccd87b br-42e950ccd87b anywhere 172.21.0.5 tcp dpt:8052

Chain DOCKER-INGRESS (1 references)
pkts bytes target prot opt in out source destination
180K 26M ACCEPT tcp -- any any anywhere anywhere tcp dpt:8002
142K 24M ACCEPT tcp -- any any anywhere anywhere state RELATED,ESTABLISHED tcp spt:8002
4399K 1463M RETURN all -- any any anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
2343K 710M DOCKER-ISOLATION-STAGE-2 all -- br-42e950ccd87b !br-42e950ccd87b anywhere anywhere
0 0 DOCKER-ISOLATION-STAGE-2 all -- br-f1cc37eb06a9 !br-f1cc37eb06a9 anywhere anywhere
0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 anywhere anywhere
77445 125M DOCKER-ISOLATION-STAGE-2 all -- docker_gwbridge !docker_gwbridge anywhere anywhere
19577 25M DOCKER-ISOLATION-STAGE-2 all -- br-962a6ca70854 !br-962a6ca70854 anywhere anywhere
4726K 1639M RETURN all -- any any anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (5 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- any br-42e950ccd87b anywhere anywhere
0 0 DROP all -- any br-f1cc37eb06a9 anywhere anywhere
0 0 DROP all -- any docker0 anywhere anywhere
0 0 DROP all -- any docker_gwbridge anywhere anywhere
0 0 DROP all -- any br-962a6ca70854 anywhere anywhere
2440K 860M RETURN all -- any any anywhere anywhere

Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
5049K 1689M RETURN all -- any any anywhere anywhere

**SeijiSensei** · May 13th, 2020

Check /etc/sysctl.conf and make sure you've enabled net.ipv4.ip_forward=1. Reboot. Any better?

**darkod** · May 13th, 2020

I am not using docker so I don't know why it creates so "complex" iptables rules, or whether they are necessary or can be avoided.

Is you issue with traffic not getting through between dockers, or traffic that should be flowing completely outside any docker (unrelated to it)?

If it's unrelated traffic, you can set the iptables FORWARD policy to ACCEPT and flush the rules. That should give you opportunity to check the traffic flowing outside of docker as you expect.

However I am not sure if it can break docker functionality because as I said I haven't worked with it.

In any case, even with docker I think you should be able to control your own iptables rules, which you might find easier than going with these that docker set up. Again, if that is possible at all...

Also, I think the following command gives iptables output much easier to read:

Code:

sudo iptables -L -n -v

**sbuxhofer** · May 13th, 2020

one more question though, why do you have a bond interface? for using and communicating with containers, shouldn't a bridge interface being used?

i do not use docker, but LXC, thats my config, currently there is only one container running:

Code:

# The primary network interfaceauto ens3
iface ens3 inet manual
        # up ip link set ens3 up


auto vbr0
iface vbr0 inet dhcp
    up ip link set vbr0 up
    dns-nameservers 172.17.17.4
    bridge_ports ens3
    bridge_fd 9
    bridge_stp off
    bridge_maxage 12
    bridge-vlan-aware yes
    bridge-vids 2-4094
    bridge-hello 2

Code:

ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:96:96:20 brd ff:ff:ff:ff:ff:ff
3: vbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:96:96:20 brd ff:ff:ff:ff:ff:ff
4: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ae:40:c3:c6:10:b6 brd ff:ff:ff:ff:ff:ff
6: veth60FR04@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:10:0b:b8:21:91 brd ff:ff:ff:ff:ff:ff link-netnsid 0

Code:

brctl show
bridge name    bridge id        STP enabled    interfaces
lxdbr0        8000.000000000000    no        
vbr0          8000.525400969620    no        ens3
                                             veth60FR04

**JamesMcT** · May 19th, 2020

Going through the replies:

@sbuxhofer : That was my first thought, confirmed ip_forward = 1, no change after 100+ reboots.

@darkod : I never configured anything to do with docker and i-tables, so that's what it does out of the box. Can't say I'm a fan of how it sets things up myself, but the docker side of things is working. The issue is not actually in the docker, but between three physical boxes.

Code:

[Workstation]              [Server                                                             ]           [Node                             ]
                           [two ethernet ports, one bond, two vlan interfaces vlan10, vlan20   ]           [two interfaces, eno1, eno1.10    ]
10.10.0.11    <= VLAN10 => if:vlan10 (tagged) 10.10.0.2 ROUTE TO if:vlan20 (tagged) 10.20.0.1 <= VLAN20 => if:eno1 (untagged) with 10.20.0.3
              <= VLAN10 ======================================================================== VLAN10 => if:eno1.10 (tagged) with 10.10.0.22

I can ping from workstation directly to node via VLAN10
I can ping from Workstation via VLAN 10 to server
I can ping from server via VLAN20 to node

I can absolutely not ping from workstation to 10.10.0.22 on the node. The workstation has a 10.20.0.0/24 route via 10.10.0.2, and I can confirm that the pings are coming in on the server interface on the vlan10 (or bond0), but nothing is going out the other side on bond0 or vlan20. They are getting stuck in the server for some reason.

Thanks for the iptables flags. It does clean it up a little.

At this point I'm probably going to break docker just to see if it is the iptables rules, because I can't think of anything else.