Automating removal of old kernels from servers

[Viruk]

Hi,

I have had a couple of problems in the last week when some of the ubuntu servers (running on VMware ESXi) ran out of space in /boot due to the different kernels installed during updates.

I couldn't remove the older kernels due to unmet dependencies:

The following packages have unmet dependencies.
  linux-image-generic-pae: Depends: linux-image-2.6.32-40-generic-pae but it is not going to be installed

I couldn't meet the dependencies because I didn't have the room.

I ended up just deleting some of the older files in /boot and managed to meet dependencies and do the outstanding updates and removing older kernels using apt-get remove.
While looking for a way to remove more of these older kernels, I found the following page: http://ubuntuforums.org/showthread.php?t=1658648
This showed the command:

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

But it is not quite what I am looking for, I would like to leave an older kernel on the system and then create a cron job to periodically run and prevent the /boot partition filling up.

I tested a few options (aren't snapshots of VMs helpful!) including the following option from the same thread above:

OLD=$(ls -tr /boot/vmlinuz-* | head -n -2 | cut -d- -f2- | \
    awk '"'"'{print "linux-image-" $0}'"'"' )
if [ -n "$OLD" ]; then
    apt-get -qy remove --purge $OLD
fi
apt-get -qy autoremove --purge

However this didn't appear to clear out any space on /boot where the first command did clear out all but the latest versions of files in /boot

Can anyone help me with editing one or the other of these options so that I leave an extra kernel on the system, but also clear the space out in /boot?

Thanks very much

[CharlesA]

You cannot remove linux-image-generic-pae package because the kernel depends on it. All you can do is remove the headers and image packages of older kernels.

Can anyone help me with editing one or the other of these options so that I leave an extra kernel on the system, but also clear the space out in /boot?

You just want one previous kernel right?

apt-get purge $(dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | head -n -1) --assume-yes

Note - I haven't tested this on 3.x.x kernels, but it should work fine as the pattern matching uses uname -r for the kernel version:

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d'
linux-headers-2.6.32-34
linux-headers-2.6.32-34-server
linux-headers-2.6.32-35
linux-headers-2.6.32-35-server
linux-headers-2.6.32-38
linux-headers-2.6.32-38-server
linux-image-2.6.32-33-server
linux-image-2.6.32-34-server
linux-image-2.6.32-35-server
linux-image-2.6.32-38-server

[SeijiSensei]

I've been thinking about posting a bug report or wishlist item about kernel management. The installer should really prune very old kernels. I can see leaving the (N-1)st and maybe even the (N-2)nd, but the ones older than that make little sense. The installer should also remove the associated stanzas for these kernels from grub.cfg, too. (This should happen automatically, I think, once the kernel images are deleted and grub-config runs.)

I used to allocate only 256MB to /boot which should be more than ample, but I've doubled that figure on more recent installations to protect against overrunning the partition. This problem particularly arises if you're running a development release. I had half-a-dozen 3.2.0-x kernel images in /boot when I checked just now.

Does this make sense to you, too?

[josephmills]

dpkg-query -l|grep linux-im*

that should match all kernels

need broken down more
like

dpkg-query -l |grep linux-im*|awk '{print $2}'

or a while loop ?

[CharlesA]

I've been thinking about posting a bug report or wishlist item about kernel management. The installer should really prune very old kernels. I can see leaving the (N-1)st and maybe even the (N-2)nd, but the ones older than that make little sense. The installer should also remove the associated stanzas for these kernels from grub.cfg, too. (This should happen automatically, I think, once the kernel images are deleted and grub-config runs.)

Go for it.

I used to allocate only 256MB to /boot which should be more than ample, but I've doubled that figure on more recent installations to protect against overrunning the partition. This problem particularly arises if you're running a development release. I had half-a-dozen 3.2.0-x kernel images in /boot when I checked just now.

Does this make sense to you, too?

I have everything on one partition so, it doesn't dramatically affect me, but I can see your point.

dpkg-query -l|grep linux-im*

that should match all kernels

need broken down more
like

dpkg-query -l |grep linux-im*|awk '{print $2}'

or a while loop ?

You can pipe it to dpkg or use xargs. See the command I posted earlier, or here.

[volkswagner]

Perhaps you can take a different approach.

If you upgrade kernel using:

 sudo aptitude safe-upgrade

Aptitude will automatically remove old kernels.

Then you could create a script to backup images & such to a separate partition plus a cleanup script for the backup location.

If this works for you, it may be safer than going directly at your /boot directory.

[CharlesA]

Hrm, I thought safe-upgrade didn't upgrade kernels? They might be removed because they are marked as obsolete in the newer versions of Ubuntu.

EDIT: I guess you are right:

http://askubuntu.com/questions/221/h...pt-get-upgrade

[Viruk]

Thanks for your replies, I'm juggling some other problems at the minute so will look at this more again in the morning. Here's what I've found so far - although I started writing this response before the last couple of suggestions...

I've been doing some testing...

the test system is up to date with everything installed by apt-get update && apt-get upgrade (at the point I took the snapshot)
the output at this point is:

The following packages have been kept back:
  landscape-common linux-generic-pae linux-headers-generic-pae
  linux-image-generic-pae
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.

uname -r gives
2.6.32-39-generic-pae

I'm not sure why this is, as other servers have updated to 2.6.32-40 using only apt-get update and apt-get upgrade

if I run:

apt-get purge $(dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | head -n -1) --assume-yes

first thing it does is install headers for 2.6.32-40

The following NEW packages will be installed
  linux-headers-2.6.32-40 linux-headers-2.6.32-40-generic-pae
The following packages will be upgraded:
  linux-headers-generic-pae
1 upgraded, 2 newly installed, 38 to remove and 3 not upgraded.
Need to get 10.7MB of archives.
After this operation, 2,218MB disk space will be freed.

it then runs through the loop removing headers for old versions, the final few lines of output are:

Setting up linux-headers-2.6.32-40 (2.6.32-40.87) ...
Setting up linux-headers-2.6.32-40-generic-pae (2.6.32-40.87) ...

Setting up linux-headers-generic-pae (2.6.32.40.47) ...

however when I cd /boot and ls the output is:

abi-2.6.32-39-generic-pae         memtest86+.bin
config-2.6.32-39-generic-pae      System.map-2.6.32-39-generic-pae
grub                              vmcoreinfo-2.6.32-39-generic-pae
initrd.img-2.6.32-39-generic-pae  vmlinuz-2.6.32-39-generic-pae
lost+found

I'm only getting left with 1 kernel version's entries in /boot - any ideas what's going on?
I've even done this multiple times and kept reverting back to the snapshot just to try to rule out any mistakes I could have made, but I've got the same result each time.

I'll try the safeupgrade and let you know what happens then.
Thanks for the help so far

[CharlesA]

You still haven't installed the updated kernel. You would have to make sure the system is totally up-to-date before running that command.

It won't make the system unbootable, thankfully, as it keeps one previous kernel.

sudo apt-get dist-upgrade

Or

sudo aptitude safe-upgrade

Will install the new kernel.

[SeijiSensei]

Go for it.

I added a comment to this bug, which raised a related point about the number of entries in /lib/modules/:

https://bugs.launchpad.net/ubuntu/+s...ty/+bug/690911

That bug is from 2010 with no assignment to a developer. We'll see if my bumping it generates a response.