Systemd DNS server losing many requests

[John Nagle]

Intermittent DNS errors that prevent web page loading, etc. nslookup shows


nslookup r.bing.com
;; communications error to 127.0.0.53#53: timed out
;; communications error to 127.0.0.53#53: timed out
;; communications error to 127.0.0.53#53: timed out
;; no servers could be reached


127.0.0.53 is systemd's DNS server. That goes out to the local AT&T fibre, which goes to Sonic.net. DNS is all the ISP default.

Setup is vanilla 22.04 LTS. No manual changes to DNS config. This problem is occurring on two desktop machines, side by side on the same LAN.
This has become much worse since the last update to Ubuntu 22.04 LTS about two days ago.

Internet connection is wired 1GB Ethernet -> AT&T U-Verse fiber modem - AT&T fiber -> Sonic.net (which uses AT&T for the last mile.)

Systemd's report from the last boot:

journalctl -u systemd-resolved -f   
     Jul 21 13:29:47 Nagle-LTS systemd[1]: systemd-resolved.service: Consumed 5.977s CPU time.
-- Boot 975d9f133cfc41fda55952df737f5c53 --
Jul 21 13:30:30 Nagle-LTS systemd[1]: Starting Network Name Resolution...
Jul 21 13:30:31 Nagle-LTS systemd-resolved[900]: Positive Trust Anchors:
Jul 21 13:30:31 Nagle-LTS systemd-resolved[900]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d   
     Jul 21 13:30:31 Nagle-LTS systemd-resolved[900]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Jul 21 13:30:31 Nagle-LTS systemd-resolved[900]: Using system hostname 'Nagle-LTS'.
Jul 21 13:30:31 Nagle-LTS systemd[1]: Started Network Name Resolution.
Jul 21 13:30:50 Nagle-LTS systemd-resolved[900]: enp3s0: Bus client set search domain list to: attlocal.net, attlocal.net
Jul 21 13:30:50 Nagle-LTS systemd-resolved[900]: enp3s0: Bus client set default route setting: yes
Jul 21 13:30:50 Nagle-LTS systemd-resolved[900]: enp3s0: Bus client set DNS server list to: 192.168.1.254, 2600:1700:22f6:2410::1

That looks OK.
This is an intermittent failure. Try once, it may fail. Try 10 times, and it almost always succeeds.
If I set nslookup to use 192.168.1.254 as the DNS server (the same server systemd is supposedly using, connections via systemd fail while connections to the DNS server in the AT&T UVerse box work.

So this looks like some kind of systemd problem.

Setting an explicit DNS server IP address in the "Settings" GUI for IPv4 seems to have no effect.

/etc/resolv.conf reads

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
search attlocal.net

Manually hacking that file has been suggested in other forums, but that's a desperation measure.

There is no "systemd-resolve" installed.

[The Cog]

I think it's systemd-resolved, not systemd-resolve.
It's a couple of years since I decided systemd's DNS server was apallingly unreliable. I generally use the brute-force method of deleting the /etc/resolv.conf symlink and replace it with a file that says something like:

nameserver 1.1.1.1
nameserver 8.8.8.8

There are probably more subtle ways, but I'm an impatient soul.

[halogen2]

This has become much worse since the last update to Ubuntu 22.04 LTS about two days ago.

Could you please be more specific about what update?

Manually hacking that file has been suggested in other forums,

If you've asked for help with this in other forums, could you please link those requests here to avoid duplicating effort?

but that's a desperation measure.

Not if you also stop/disable systemd-resolved and configure NetworkManager to work with such setup. You can use The Cog's suggestion if you edit /etc/NetworkManager/NetworkManager.conf , under [main] section set appropriate values for dns= and rc-manager= . See man NetworkManager.conf for possible values and further details.

To apply the changes, run sudo systemctl restart NetworkManager

Note that doing this stuff wrong results in no DNS at all, so maybe try it in a test system before making changes in your main system?

[currentshaft]

There's really no good reason to use your ISP's default DNS, and many reasons to avoid it.

Can you reproduce this with another DNS provider?

Also, use "dig" instead of nslookup to do network troubleshooting, e.g.:

dig a ubuntuforums.org
dig a ubuntuforums.org @1.1.1.1

[John Nagle]

> systemd-resolved
systemd-resolved: command not found
> sudo apt-get install systemd-resolved
...
E: Unable to locate package systemd-resolved
What was intended there?

> I generally use the brute-force method of deleting the /etc/resolv.conf symlink
Yes, advice I keep getting is "don't use systemd". But then I have nonstandard systems and parts of the GUI stop working.

> Could you please be more specific about what update?
Whatever "software updater" told me to do. "Ubuntu Pro" has not been purchases.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
john@Nagle-LTS:/etc$
> uname -r
5.15.0-116-generic

> Note that doing this stuff wrong results in no DNS at all, so maybe try it in a test system before making changes in your main system?
Which is why I'd rather not get into manually hacking key system files.

This seems to be a systemd bug. Others see similar problems. Do those ever get fixed, or what?

[halogen2]

Yes, advice I keep getting is "don't use systemd". But then I have nonstandard systems and parts of the GUI stop working.

systemd doesn't require you to use systemd-resolved. What nonstandard GUI parts of your system require using systemd-resolved?

> Could you please be more specific about what update?
Whatever "software updater" told me to do.

Those updates should be logged in /var/log/apt/history.log (or one of the /var/log/apt/history.log.*.gz files). Could you please post the log entry of the update that caused this issue?

I'd rather not get into manually hacking key system files.

Then don't. Instead, find good way(s) to modify them. And when working this type stuff out, it's very helpful to have a test environment which you can restore easily and where it doesn't really matter if something goes wrong. Do you not have such test environment?

[John Nagle]

Possibly related to new systemd bug: https://github.com/systemd/systemd/issues/33822

[John Nagle]

Nothing exciting in here:

Start-Date: 2024-07-10  06:36:33
Commandline: /usr/bin/unattended-upgrade
Upgrade: python3.10:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10-
minimal:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), libpython3.10-stdlib:amd64
 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), python3.10-venv:amd64 (3.10.12-1~22.04.
3, 3.10.12-1~22.04.4), libpython3.10:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4
), libpython3.10-dev:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), python3.10-de
v:amd64 (3.10.12-1~22.04.3, 3.10.12-1~22.04.4), python3.10-minimal:amd64 (3.10.1
2-1~22.04.3, 3.10.12-1~22.04.4)
End-Date: 2024-07-10  06:37:10

Start-Date: 2024-07-13  06:50:33
Commandline: /usr/bin/unattended-upgrade
Upgrade: apache2-utils:amd64 (2.4.52-1ubuntu4.10, 2.4.52-1ubuntu4.11)
End-Date: 2024-07-13  06:50:45

Start-Date: 2024-07-13  06:50:50
Commandline: /usr/bin/unattended-upgrade
Upgrade: apache2-bin:amd64 (2.4.52-1ubuntu4.10, 2.4.52-1ubuntu4.11)
End-Date: 2024-07-13  06:50:51

Start-Date: 2024-07-16  06:26:39
Commandline: /usr/bin/unattended-upgrade
Upgrade: ghostscript-x:amd64 (9.55.0~dfsg1-0ubuntu5.7, 9.55.0~dfsg1-0ubuntu5.9),
 libgs9-common:amd64 (9.55.0~dfsg1-0ubuntu5.7, 9.55.0~dfsg1-0ubuntu5.9), ghostsc
ript:amd64 (9.55.0~dfsg1-0ubuntu5.7, 9.55.0~dfsg1-0ubuntu5.9), libgs9:amd64 (9.5
5.0~dfsg1-0ubuntu5.7, 9.55.0~dfsg1-0ubuntu5.9)
End-Date: 2024-07-16  06:26:56

Start-Date: 2024-07-16  06:27:00
Commandline: /usr/bin/unattended-upgrade
Install: linux-headers-5.15.0-116:amd64 (5.15.0-116.126, automatic), linux-modul
es-extra-5.15.0-116-generic:amd64 (5.15.0-116.126, automatic), linux-headers-5.1
5.0-116-generic:amd64 (5.15.0-116.126, automatic), linux-image-5.15.0-116-generi
c:amd64 (5.15.0-116.126, automatic), linux-modules-5.15.0-116-generic:amd64 (5.1
5.0-116.126, automatic)
Upgrade: linux-headers-generic:amd64 (5.15.0.113.113, 5.15.0.116.116), linux-gen
eric:amd64 (5.15.0.113.113, 5.15.0.116.116), linux-image-generic:amd64 (5.15.0.1
13.113, 5.15.0.116.116)
End-Date: 2024-07-16  06:28:50

Start-Date: 2024-07-16  06:28:55
Commandline: /usr/bin/unattended-upgrade
Upgrade: linux-libc-dev:amd64 (5.15.0-113.123, 5.15.0-116.126)
End-Date: 2024-07-16  06:28:57

Start-Date: 2024-07-16  06:29:00
Commandline: /usr/bin/unattended-upgrade
Remove: linux-modules-extra-5.15.0-107-generic:amd64 (5.15.0-107.117)
End-Date: 2024-07-16  06:29:13

Start-Date: 2024-07-16  06:29:16
Commandline: /usr/bin/unattended-upgrade
Remove: linux-modules-5.15.0-107-generic:amd64 (5.15.0-107.117), linux-image-5.1
5.0-107-generic:amd64 (5.15.0-107.117)
End-Date: 2024-07-16  06:29:25

Start-Date: 2024-07-16  06:29:28
Commandline: /usr/bin/unattended-upgrade
Remove: linux-headers-5.15.0-107:amd64 (5.15.0-107.117), linux-headers-5.15.0-10
7-generic:amd64 (5.15.0-107.117)
End-Date: 2024-07-16  06:29:45

Start-Date: 2024-07-17  06:28:25
Commandline: /usr/bin/unattended-upgrade
Upgrade: libgtk2.0-bin:amd64 (2.24.33-2ubuntu2, 2.24.33-2ubuntu2.1), libgtk2.0-d
ev:amd64 (2.24.33-2ubuntu2, 2.24.33-2ubuntu2.1), libgail18:amd64 (2.24.33-2ubunt
u2, 2.24.33-2ubuntu2.1), gir1.2-gtk-2.0:amd64 (2.24.33-2ubuntu2, 2.24.33-2ubuntu
2.1), libgtk2.0-0:amd64 (2.24.33-2ubuntu2, 2.24.33-2ubuntu2.1), libgail-common:a
md64 (2.24.33-2ubuntu2, 2.24.33-2ubuntu2.1)
End-Date: 2024-07-17  06:29:06

Start-Date: 2024-07-17  06:29:12
Commandline: /usr/bin/unattended-upgrade
Upgrade: gtk2-engines-pixbuf:amd64 (2.24.33-2ubuntu2, 2.24.33-2ubuntu2.1)
End-Date: 2024-07-17  06:29:15

Start-Date: 2024-07-17  06:29:19
Commandline: /usr/bin/unattended-upgrade
Upgrade: gir1.2-gtk-3.0:amd64 (3.24.33-1ubuntu2.1, 3.24.33-1ubuntu2.2), libgail-
3-0:amd64 (3.24.33-1ubuntu2.1, 3.24.33-1ubuntu2.2), libgtk-3-0:amd64 (3.24.33-1u
buntu2.1, 3.24.33-1ubuntu2.2), libgtk-3-common:amd64 (3.24.33-1ubuntu2.1, 3.24.3
3-1ubuntu2.2), libgtk-3-bin:amd64 (3.24.33-1ubuntu2.1, 3.24.33-1ubuntu2.2), libg
tk-3-dev:amd64 (3.24.33-1ubuntu2.1, 3.24.33-1ubuntu2.2)
End-Date: 2024-07-17  06:29:38

Start-Date: 2024-07-17  06:29:42
Commandline: /usr/bin/unattended-upgrade
Upgrade: gtk-update-icon-cache:amd64 (3.24.33-1ubuntu2.1, 3.24.33-1ubuntu2.2)
End-Date: 2024-07-17  06:29:45

Start-Date: 2024-07-17  06:29:50
Commandline: /usr/bin/unattended-upgrade
Upgrade: libgtk2.0-common:amd64 (2.24.33-2ubuntu2, 2.24.33-2ubuntu2.1)
End-Date: 2024-07-17  06:29:52

Start-Date: 2024-07-19  06:29:33
Commandline: /usr/bin/unattended-upgrade
Upgrade: apache2-bin:amd64 (2.4.52-1ubuntu4.11, 2.4.52-1ubuntu4.12)
End-Date: 2024-07-19  06:29:46

Start-Date: 2024-07-19  06:29:51
Commandline: /usr/bin/unattended-upgrade
Upgrade: apache2-utils:amd64 (2.4.52-1ubuntu4.11, 2.4.52-1ubuntu4.12)
End-Date: 2024-07-19  06:29:54

Start-Date: 2024-07-19  11:09:29
Commandline: aptdaemon role='role-commit-packages' sender=':1.167'
Install: libodbc2:i386 (2.3.9-5ubuntu0.1, automatic), libgsm1:i386 (1.0.19-1, au
tomatic), libxslt1.1:i386 (1.1.34-4ubuntu0.22.04.1, automatic), libxkbregistry0:
i386 (1.4.0-1, automatic), libcapi20-3:amd64 (1:3.27-3, automatic), libcapi20-3:
i386 (1:3.27-3, automatic)
Upgrade: winehq-stable:amd64 (7.0.0.0~focal-1, 9.0.0.0~jammy-1), libldap-common:
amd64 (2.5.17+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.1), xserver-xorg
-core:amd64 (2:21.1.4-2ubuntu1.7~22.04.10, 2:21.1.4-2ubuntu1.7~22.04.11), libvkd
3d1:amd64 (1.3~focal-1, 1.3~jammy-1), libvkd3d1:i386 (1.3~focal-1, 1.3~jammy-1),
 vkd3d-compiler:amd64 (1.3~focal-1, 1.3~jammy-1), xdg-desktop-portal-gnome:amd64
 (42.1-0ubuntu1, 42.1-0ubuntu2), libvkd3d-shader1:amd64 (1.3~focal-1, 1.3~jammy-
1), libvkd3d-shader1:i386 (1.3~focal-1, 1.3~jammy-1), libpathplan4:amd64 (2.42.2
-6, 2.42.2-6ubuntu0.1), graphviz:amd64 (2.42.2-6, 2.42.2-6ubuntu0.1), libgvpr2:a
md64 (2.42.2-6, 2.42.2-6ubuntu0.1), libgvc6:amd64 (2.42.2-6, 2.42.2-6ubuntu0.1),
 ldap-utils:amd64 (2.5.17+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.1), 
wine-stable-i386:i386 (7.0.0.0~focal-1, 9.0.0.0~jammy-1), libldap-2.5-0:amd64 (2
.5.17+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.1), libldap-2.5-0:i386 (
2.5.17+dfsg-0ubuntu0.22.04.1, 2.5.18+dfsg-0ubuntu0.22.04.1), libcgraph6:amd64 (2
.42.2-6, 2.42.2-6ubuntu0.1), xserver-xorg-legacy:amd64 (2:21.1.4-2ubuntu1.7~22.0
4.10, 2:21.1.4-2ubuntu1.7~22.04.11), ubuntu-desktop:amd64 (1.481.1, 1.481.2), li
bcdt5:amd64 (2.42.2-6, 2.42.2-6ubuntu0.1), xserver-common:amd64 (2:21.1.4-2ubunt
u1.7~22.04.10, 2:21.1.4-2ubuntu1.7~22.04.11), xserver-xorg-dev:amd64 (2:21.1.4-2
ubuntu1.7~22.04.10, 2:21.1.4-2ubuntu1.7~22.04.11), wine-stable-amd64:amd64 (7.0.
0.0~focal-1, 9.0.0.0~jammy-1), xserver-xephyr:amd64 (2:21.1.4-2ubuntu1.7~22.04.1
0, 2:21.1.4-2ubuntu1.7~22.04.11), intel-gpu-tools:amd64 (1.26-2, 1.26-2ubuntu0.1
), wine-stable:amd64 (7.0.0.0~focal-1, 9.0.0.0~jammy-1), liblab-gamut1:amd64 (2.
42.2-6, 2.42.2-6ubuntu0.1), ubuntu-standard:amd64 (1.481.1, 1.481.2), ubuntu-des
ktop-minimal:amd64 (1.481.1, 1.481.2), ubuntu-minimal:amd64 (1.481.1, 1.481.2)
End-Date: 2024-07-19  11:10:50

Start-Date: 2024-07-22  06:11:30
Commandline: /usr/bin/unattended-upgrade
Upgrade: thunderbird:amd64 (1:115.12.0+build3-0ubuntu0.22.04.1, 1:115.13.0+build
5-0ubuntu0.22.04.1), thunderbird-locale-en:amd64 (1:115.12.0+build3-0ubuntu0.22.
04.1, 1:115.13.0+build5-0ubuntu0.22.04.1), thunderbird-gnome-support:amd64 (1:11
5.12.0+build3-0ubuntu0.22.04.1, 1:115.13.0+build5-0ubuntu0.22.04.1)
End-Date: 2024-07-22  06:11:58

Start-Date: 2024-07-22  06:12:01
Commandline: /usr/bin/unattended-upgrade
Upgrade: thunderbird-locale-en-us:amd64 (1:115.12.0+build3-0ubuntu0.22.04.1, 1:1
15.13.0+build5-0ubuntu0.22.04.1)
End-Date: 2024-07-22  06:12:03

Start-Date: 2024-07-23  06:36:14
Commandline: /usr/bin/unattended-upgrade
Upgrade: python3-zipp:amd64 (1.0.0-3, 1.0.0-3ubuntu0.1)
End-Date: 2024-07-23  06:36:27

[John Nagle]

7 month old discussion on Reddit: https://www.reddit.com/r/linux/comme...talking_about/

It's been an intermittent problem for years, but something recently made it much worse.

[halogen2]

It would likely be easier to switch away from systemd-resolved than try to troubleshoot the issue with the current setup.

Are you sure the thing that made it worse was an Ubuntu update? If so, and if you want to go down the investigating/troubleshooting rabbit hole, try rebooting into the previous kernel version?

Other than the kernel update, not noticing anything in that log that's likely to affect networking.