20.04: 'apt-get upgrade' added a huge load of packages and threads

[varenne2]

This is an Ubuntu 20.04 server, running LAMP, obviously "headless". On 11 August 2021 I ran 'apt-get update' followed by 'apt-get upgrade', which I have been doing for over a 18 months without any issues.

But on this day, I noticed that 'apt-get upgrade' reported a huge bulk of packages. Remembering that I haven't done an upgrade for a couple of months, I accepted with 'Y'. The upgrade took ages, over an hour, although this is a very powerful server with full 1 Gb/s connectivity. Most troubling were packages like bluez (Bluetooth!), sound and all sorts of graphical software xorg & co.

I expected a disaster, but the server restarted and is churning further serving LAMP (Apache, MariaDB). But, shockingly, the number of threads have jumped from some 315 to 515! (See munin screen-shot)

How can I track the unwanted services/threads and stop them? And do any other health checks?

[slickymaster]

Thread moved to Server Platforms.

[Doug S]

It almost sounds as though you converted your server version to a desktop version. Have a look at /var/log/apt/history.log as to what was done on August 11th. Post it here if you want us to also have a look. Also have a look at "ps aux" for what is running. I have 227 processes on my 20.04 server, and no "big" upgrade recently. The most recent:

doug@s19:~$ tail -7 /var/log/apt/history.log

Start-Date: 2021-08-17  08:11:37
Commandline: apt upgrade
Requested-By: doug (1000)
Install: linux-headers-5.4.0-81-generic:amd64 (5.4.0-81.91, automatic), linux-modules-extra-5.4.0-81-generic:amd64 (5.4.0-81.91, automatic), linux-modules-5.4.0-81-generic:amd64 (5.4.0-81.91, automatic), linux-image-5.11.0-27-generic:amd64 (5.11.0-27.29~20.04.1, automatic), linux-headers-5.11.0-27-generic:amd64 (5.11.0-27.29~20.04.1, automatic), linux-modules-extra-5.11.0-27-generic:amd64 (5.11.0-27.29~20.04.1, automatic), linux-headers-5.4.0-81:amd64 (5.4.0-81.91, automatic), linux-image-5.4.0-81-generic:amd64 (5.4.0-81.91, automatic), linux-modules-5.11.0-27-generic:amd64 (5.11.0-27.29~20.04.1, automatic), linux-hwe-5.11-headers-5.11.0-27:amd64 (5.11.0-27.29~20.04.1, automatic)
Upgrade: linux-headers-generic-hwe-20.04:amd64 (5.11.0.25.27~20.04.10, 5.11.0.27.29~20.04.11), linux-headers-generic:amd64 (5.4.0.80.84, 5.4.0.81.85), linux-libc-dev:amd64 (5.4.0-80.90, 5.4.0-81.91), linux-image-generic:amd64 (5.4.0.80.84, 5.4.0.81.85), libglapi-mesa:amd64 (21.0.3-0ubuntu0.2~20.04.1, 21.0.3-0ubuntu0.3~20.04.1), language-pack-en:amd64 (1:20.04+20210121, 1:20.04+20210802), libpq5:amd64 (12.7-0ubuntu0.20.04.1, 12.8-0ubuntu0.20.04.1), linux-image-generic-hwe-20.04:amd64 (5.11.0.25.27~20.04.10, 5.11.0.27.29~20.04.11), libgbm1:amd64 (21.0.3-0ubuntu0.2~20.04.1, 21.0.3-0ubuntu0.3~20.04.1), linux-generic-hwe-20.04:amd64 (5.11.0.25.27~20.04.10, 5.11.0.27.29~20.04.11), python3-distupgrade:amd64 (1:20.04.35, 1:20.04.36), ubuntu-release-upgrader-core:amd64 (1:20.04.35, 1:20.04.36), libgl1-mesa-dri:amd64 (21.0.3-0ubuntu0.2~20.04.1, 21.0.3-0ubuntu0.3~20.04.1), language-pack-en-base:amd64 (1:20.04+20210121, 1:20.04+20210802), mesa-vdpau-drivers:amd64 (21.0.3-0ubuntu0.2~20.04.1, 21.0.3-0ubuntu0.3~20.04.1), linux-firmware:amd64 (1.187.15, 1.187.16), mesa-vulkan-drivers:amd64 (21.0.3-0ubuntu0.2~20.04.1, 21.0.3-0ubuntu0.3~20.04.1), linux-generic:amd64 (5.4.0.80.84, 5.4.0.81.85), mesa-va-drivers:amd64 (21.0.3-0ubuntu0.2~20.04.1, 21.0.3-0ubuntu0.3~20.04.1), libglx-mesa0:amd64 (21.0.3-0ubuntu0.2~20.04.1, 21.0.3-0ubuntu0.3~20.04.1)
End-Date: 2021-08-17  08:15:41
doug@s19:~$

[MAFoElffen]

Logs yes... but please, do not post them here... That might be too much for one post in a code block in a post. Especially if it took over an hour to process.

Better yet to post it at: https://paste.ubuntu.com/
...and set an expiration for it, and then post the link to here in this thread. Easier to look at and trace them out there.

I have to agree with "Doug S", that it initially sounds like you have some "new" packages and a configuration change happened somehow.... possible graphical, that is causing more overhead. I didn't see any big updates last week for server.

Having said that... Most of my servers are virtual, running on KVM. There was an issue I had this last week with a Qemu/KVM update on my KVM hosts... that I didn't see and caused all my virtual guests that were running at the time to spike and slow down to a crawl... Somethings that normally, took minutes, took over two hours!!! That went on until I shut them down and restarted everything. Everything is running fine now. I think it was that the guests were running on old (virtual environment) code, until restarted.

That is the one thing that I did see this last week.

[ameinild]

I have a 20.04 server, and I'm running full-upgrade every weekend automatically. Nothing unusual has happened lately.

I agree with Doug, please post /var/log/apt/history.log (at Pastebin or paste.ubuntu.com) to see which packages was installed.

[varenne2]

Hi you both,

Yes, I fear, my server has turned in to a desktop!


Here is the apt/history.log: https://paste.ubuntu.com/p/d4PjXTXWz5/.

Here is the output of 'ps aux': https://paste.ubuntu.com/p/VQ2DvxwZCx/ (both set to expire in 1 month).

The number 515 earlier happen to be the number of threads. The number of processes are around 250.

[Doug S]

Yes, I fear, my server has turned in to a desktop!


Here is the apt/history.log: https://paste.ubuntu.com/p/d4PjXTXWz5/.

It looks as though your computer was always configured as a desktop. Notice that the large update on August 11th is all upgrades and nothing new is installed.

Here is the output of 'ps aux': https://paste.ubuntu.com/p/VQ2DvxwZCx/ (both set to expire in 1 month).

The number 515 earlier happen to be the number of threads. The number of processes are around 250.

So, there are many "gdm" type processes. Perhaps you had that disabled before and somehow it now got enabled. Anyway, I am a server person and do not know how to help further.

[LHammonds]

If you have not disabled bash history, check the command log to see if there were any "apt install" commands issued under root or any user with sudo access.

tail -n100 ~/.bash_history
grep "apt" ~/.bash_history

sudo tail -n100 /root/.bash_history
sudo grep "apt" /root/.bash_history

LHammonds

[MAFoElffen]

Hate to say this... but searching that way adds CPU, and would take a lot of manual work... and I am lazy. Better to narrow down your search terms to get a subset to direct the search... And to be able to get a plan together on how to correct it.

Better to start out with:

apt list --installed | grep "\-desktop"

That one liner, will search on the installed packages and narrow that search down to under 10 lines. That will tell you what DE package(s) was installed (plus a few extraneous) , and to prepare a plan to correct it.

Save those results, so you can use that later for forensics to investigate how that happened. The priority to me, if it was my server, would be to quickly get the server load back down, before investigating the past cause. Uptime of services.

The one liner that will get you back to a server edition: (Take a backup beforehand, for the just in cases...)

sudo apt install ubuntu-server

That will remove all Desktop packages, Desktop Managers (DM), any Linux Graphics Layer Support (beyond normal Server) and take care of all the apt-markings... In one step.

[LHammonds]

In the 1st post, it was said "obviously headless" which indicates this person either forgot they installed desktop, or installed something else that installed desktop or someone else installed something recently. Looking thru command history can help find who did what. If its a single person managing it, no problem...they can do whatever they want but if multiple people have their hands in it, they might need to investigate who did what and find out why....there might be a good reason its there...or not.