Results 1 to 2 of 2

Thread: DNS fails when docker swarm degrades on 20.04 LTS

  1. #1
    Join Date
    Nov 2017
    Beans
    5

    DNS fails when docker swarm degrades on 20.04 LTS

    Hi,

    I can't figure out why DNS resolving breaks as soon as docker swarm set its iptables rules on Ubuntu 20.04.
    The problem started after I lost one of the 3 swarm nodes, and one of the 3 DNS servers. (Degraded)
    Eventually recreating and re-joining the missing swarm node and adding the missing DNS IP address on a healthy node did not fix the problem.


    • /etc/resolvd.conf points to 'nameserver 127.0.0.53' (OK)
    • I can resolve any hostname using the systemd-resolve command:
      Code:
      # systemd-resolve startpage.com 
      startpage.com: 145.131.132.78-- link: ens18
      
      -- Information acquired via protocol DNS in 1.0ms.
      -- Data is authenticated: no
    • the systemd-resolve properly listens on port 53
    • But DNS resolution is broken:
      Code:
      # host startpage.com  
      ;; connection timed out; no servers could be reached

    • By investigating much more, I found out that there is a NAT redirection in the DOCKER-INGRESS nat table chain for DNS traffic. Removing this entry restores DNS:
      Code:
      # iptables -t nat -L DOCKER-INGRESS -n -v --line-numbers 
      Chain DOCKER-INGRESS (2 references) 
      num   pkts bytes target     prot opt in     out     source               destination          
      1        4   240 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:2222 to:172.19.0.2:2222 
      2        3   180 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8082 to:172.19.0.2:8082 
      3        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8080 to:172.19.0.2:8080 
      4        1    59 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 to:172.19.0.2:53 
      5        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:9000 to:172.19.0.2:9000 
      6        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8000 to:172.19.0.2:8000 
      7        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8010 to:172.19.0.2:8010 
      8       32  2437 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            
      # iptables -t nat -D DOCKER-INGRESS 4  # Delete the 4th line in the chain     
      # host startpage.com                      
      startpage.com has address 145.131.132.68 
      startpage.com mail is handled by 10 mx1.startmail.com. 
      startpage.com mail is handled by 10 mx2.startmail.com.



    From my understanding, all queries from the host are sent to the Docker Swarm to resolve container names from the host. But this Docker DNS is broken. (172.19.0.2 is reachable but times out on DNS queries).

    How should I get DNS working ? And why does this happen after the loss of one node ?



    Last edited by pivert2; August 2nd, 2021 at 09:40 AM.

  2. #2
    Join Date
    Nov 2017
    Beans
    5

    Re: DNS fails when docker swarm degrades on 20.04 LTS

    A temporary workaround is to run this poorly crafted oneliner after docker has started on every node:
    Code:
    host startpage.com || iptables -t nat -L DOCKER-INGRESS --line-numbers | grep -E 'DNAT.*udp dpt:domain' | awk
     '{print $1};' | xargs iptables -t nat -D DOCKER-INGRESS

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •