Having problems freeing space in /boot

[elpidiovaldez5]

I am using Lubuntu 20.04. For some time I have been having problems with automatic updates which seem to be due to a lack of free space in /boot. I have been following a procedure to free space, but that does not work either. I'd appreciate some help fixing the system.

Based on what I have read the following should explain the problem:

paul@clio:~$ df -i
Filesystem        Inodes   IUsed     IFree IUse% Mounted on
udev             4090392     758   4089634    1% /dev
tmpfs            4098333    1275   4097058    1% /run
/dev/nvme0n1p3  31195136 1011315  30183821    4% /
tmpfs            4098333      84   4098249    1% /dev/shm
tmpfs            4098333       7   4098326    1% /run/lock
tmpfs            4098333      18   4098315    1% /sys/fs/cgroup
tmpfs            4098333      41   4098292    1% /tmp
/dev/loop1         10849   10849         0  100% /snap/core18/2344
/dev/loop2         11789   11789         0  100% /snap/core20/1494
/dev/loop3         10857   10857         0  100% /snap/core18/2409
/dev/loop4         11789   11789         0  100% /snap/core20/1518
/dev/loop5         27806   27806         0  100% /snap/gnome-3-28-1804/161
/dev/loop0            29      29         0  100% /snap/bare/5
/dev/loop7           486     486         0  100% /snap/snapd/16010
/dev/loop6         16536   16536         0  100% /snap/libreoffice/254
/dev/loop8         17504   17504         0  100% /snap/gnome-3-38-2004/106
/dev/nvme0n1p2     34240     326     33914    1% /boot
/dev/nvme0n1p1         0       0         0     - /boot/efi
/dev/loop9           486     486         0  100% /snap/snapd/16292
/dev/loop10        76208   76208         0  100% /snap/gtk-common-themes/1535
/dev/loop11        27807   27807         0  100% /snap/gnome-3-28-1804/145
/dev/loop12        76177   76177         0  100% /snap/gtk-common-themes/1534
/dev/loop13        18120   18120         0  100% /snap/gnome-3-38-2004/112
/dev/loop14        17144   17144         0  100% /snap/libreoffice/256
tmpfs            4098333      46   4098287    1% /run/user/1000
/dev/sdb1      900416964 2700844 897716120    1% /media/paul/Seagate Backup Plus Drive

paul@clio:~$ ls -l /boot
total 496124
-rw-r--r-- 1 root root    237942 Feb  3 18:16 config-5.4.0-100-generic
-rw-r--r-- 1 root root    237975 Mar 24 15:48 config-5.4.0-107-generic
-rw-r--r-- 1 root root    237975 Apr  8 09:44 config-5.4.0-109-generic
-rw-r--r-- 1 root root    237975 Apr 14 13:19 config-5.4.0-110-generic
-rw-r--r-- 1 root root    237975 May 18 15:07 config-5.4.0-113-generic
-rw-r--r-- 1 root root    237975 Jun 15 14:13 config-5.4.0-121-generic
-rw-r--r-- 1 root root    237940 Feb  2 16:21 config-5.4.0-99-generic
drwx------ 3 root root      4096 Jan  1  1970 efi
drwxr-xr-x 4 root root      4096 Apr 16 16:20 grub
-rw------- 1 root root  57334711 Mar 31 18:32 initrd.img-5.4.0-100-generic
-rw------- 1 root root  57335089 Mar 31 18:34 initrd.img-5.4.0-107-generic
-rw------- 1 root root 116016571 Aug 25  2021 initrd.img-5.4.0-77-generic
-rw------- 1 root root 116021855 Sep  8  2021 initrd.img-5.4.0-81-generic
-rw------- 1 root root  57331389 Mar 31 18:33 initrd.img-5.4.0-99-generic
drwx------ 2 root root     16384 Jan 22  2021 lost+found
-rw-r--r-- 1 root root    182704 Aug 18  2020 memtest86+.bin
-rw-r--r-- 1 root root    184380 Aug 18  2020 memtest86+.elf
-rw-r--r-- 1 root root    184884 Aug 18  2020 memtest86+_multiboot.bin
-rw------- 1 root root   4758018 Feb  3 18:16 System.map-5.4.0-100-generic
-rw------- 1 root root   4759409 Mar 24 15:48 System.map-5.4.0-107-generic
-rw------- 1 root root   4759493 Apr  8 09:44 System.map-5.4.0-109-generic
-rw------- 1 root root   4759819 Apr 14 13:19 System.map-5.4.0-110-generic
-rw------- 1 root root   4759899 May 18 15:07 System.map-5.4.0-113-generic
-rw------- 1 root root   4761203 Jun 15 14:13 System.map-5.4.0-121-generic
-rw------- 1 root root   4757200 Feb  2 16:21 System.map-5.4.0-99-generic
lrwxrwxrwx 1 root root        25 May 12 17:50 vmlinuz -> vmlinuz-5.4.0-110-generic
-rw------- 1 root root  13664512 Feb  4 17:04 vmlinuz-5.4.0-100-generic
-rw------- 1 root root  13664512 Mar 24 15:52 vmlinuz-5.4.0-107-generic
-rw------- 1 root root  13668608 Apr  8 09:45 vmlinuz-5.4.0-109-generic
-rw------- 1 root root  13668608 Apr 14 13:56 vmlinuz-5.4.0-110-generic
-rw------- 1 root root  13660416 Feb  2 16:23 vmlinuz-5.4.0-99-generic
lrwxrwxrwx 1 root root        25 May 12 17:50 vmlinuz.old -> vmlinuz-5.4.0-109-generic

paul@clio:~$ sudo apt --purge autoremove
[sudo] password for paul:                                                                                                                 
Reading package lists... Done                                                                                                             
Building dependency tree                                                                                                                  
Reading state information... Done                                                                                                         
You might want to run 'apt --fix-broken install' to correct these.                                                                        
The following packages have unmet dependencies.                                                                                           
 linux-image-generic : Depends: linux-image-5.4.0-121-generic but it is not installed                                                     
 linux-modules-extra-5.4.0-113-generic : Depends: linux-image-5.4.0-113-generic but it is not installed or                                
                                                  linux-image-unsigned-5.4.0-113-generic but it is not installed                          
 linux-modules-extra-5.4.0-121-generic : Depends: linux-image-5.4.0-121-generic but it is not installed or                                
                                                  linux-image-unsigned-5.4.0-121-generic but it is not installed                          
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).

paul@clio:~$ uname -r                                                                                                                     
5.4.0-107-generic

I have not tried 'apt --fix-broken install' yet, as I don't understand what it will do, and my top priority is not to break anything more ! I would appreciate some explanation and advice.

[vanadium]

You have a separate boot partition, and too many old kernels lingering around. An old kernel occupies about 500 Mb, so that counts up. Your output does not show that: for a reason unknow, you select to use the "-i" (inode) switch. I however take the freedom to assume that the root partition has no free space anymore.

The problem is that probably your `apt` will not work because it always needs some free space. I nevertheless would attempt with `sudo apt --purge autoremove` first, because that may remove old kernels. If it succeeds, you will have free space again on the /boot partition. If that does not work, you will need to proceed to removal as described in the official Ubuntu documentation.

[elpidiovaldez5]

Many thanks for your help. Sorry I posted output of df -i by mistake. Here is the correct output (shows /boot is full).

paul@clio:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  2.2M  3.2G   1% /run
/dev/nvme0n1p3  468G  137G  307G  31% /
tmpfs            16G   43M   16G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs            16G   17M   16G   1% /tmp
/dev/loop1       56M   56M     0 100% /snap/core18/2344
/dev/loop2       62M   62M     0 100% /snap/core20/1494
/dev/loop3       56M   56M     0 100% /snap/core18/2409
/dev/loop4       62M   62M     0 100% /snap/core20/1518
/dev/loop5      165M  165M     0 100% /snap/gnome-3-28-1804/161
/dev/loop0      128K  128K     0 100% /snap/bare/5
/dev/loop7       47M   47M     0 100% /snap/snapd/16010
/dev/loop6      918M  918M     0 100% /snap/libreoffice/254
/dev/loop8      255M  255M     0 100% /snap/gnome-3-38-2004/106
/dev/nvme0n1p2  510M  494M     0 100% /boot
/dev/nvme0n1p1  532M  5.3M  527M   1% /boot/efi
/dev/loop9       47M   47M     0 100% /snap/snapd/16292
/dev/loop10      92M   92M     0 100% /snap/gtk-common-themes/1535
/dev/loop11     163M  163M     0 100% /snap/gnome-3-28-1804/145
/dev/loop12      82M   82M     0 100% /snap/gtk-common-themes/1534
/dev/loop13     401M  401M     0 100% /snap/gnome-3-38-2004/112
/dev/loop14     915M  915M     0 100% /snap/libreoffice/256
tmpfs           3.2G   16K  3.2G   1% /run/user/1000
/dev/sdb1       1.9T 1008G  856G  55% /media/paul/Seagate Backup Plus Drive

I have previously attempted to use sudo apt-get autoremove --purge. The results are shown in my previous message. It failed.

I tried to follow the procedure in https://help.ubuntu.com/community/Re...ng_Old_Kernels. This also failed. The output is below:

paul@clio:~$ uname -r
5.4.0-107-generic
paul@clio:~$ dpkg -l | tail -n +6 | grep -E 'linux-image-[0-9]+'
pi  linux-image-5.4.0-100-generic                 5.4.0-100.113                        amd64        Signed kernel image generic
ii  linux-image-5.4.0-107-generic                 5.4.0-107.121                        amd64        Signed kernel image generic
iF  linux-image-5.4.0-109-generic                 5.4.0-109.123                        amd64        Signed kernel image generic
iF  linux-image-5.4.0-110-generic                 5.4.0-110.124                        amd64        Signed kernel image generic
rc  linux-image-5.4.0-96-generic                  5.4.0-96.109                         amd64        Signed kernel image generic
pi  linux-image-5.4.0-99-generic                  5.4.0-99.112                         amd64        Signed kernel image generic
paul@clio:~$ sudo update-initramfs -d -k 5.4.0-99-generic
[sudo] password for paul: 
update-initramfs: Deleting /boot/initrd.img-5.4.0-99-generic
paul@clio:~$ sudo dpkg --purge linux-image-5.4.0-99-generic
dpkg: dependency problems prevent removal of linux-image-5.4.0-99-generic:
 linux-modules-extra-5.4.0-99-generic depends on linux-image-5.4.0-99-generic | linux-image-unsigned-5.4.0-99-generic; however:
  Package linux-image-5.4.0-99-generic is to be removed.
  Package linux-image-unsigned-5.4.0-99-generic is not installed.

dpkg: error processing package linux-image-5.4.0-99-generic (--purge):
 dependency problems - not removing
Errors were encountered while processing:
 linux-image-5.4.0-99-generic
paul@clio:~$ sudo dpkg --purge linux-image-5.4.0-99-generic linux-image-extra-5.4.0-99-generic
dpkg: dependency problems prevent removal of linux-image-5.4.0-99-generic:
 linux-modules-extra-5.4.0-99-generic depends on linux-image-5.4.0-99-generic | linux-image-unsigned-5.4.0-99-generic; however:
  Package linux-image-5.4.0-99-generic is to be removed.
  Package linux-image-unsigned-5.4.0-99-generic is not installed.

dpkg: error processing package linux-image-5.4.0-99-generic (--purge):
 dependency problems - not removing
dpkg: warning: ignoring request to remove linux-image-extra-5.4.0-99-generic which isn't installed
Errors were encountered while processing:
 linux-image-5.4.0-99-generic

I have been wrestling with the problem of freeing space in /boot for some time. I think many failed attempts have caused damage to the package structures - I presume pi means 'partially installed'.

[TheFu]

Well, then you should manually remove/purge each specific kernel package that is old. Leave the last 2 working kernels.

Kernels packages typically have modules and header packages that are matched. When the kernel is removed, the match modules and headers can/should be removed too.

There are many different ways to accomplish this. For most people, using Synaptic (a GUI package manager) will be the easiest. But if you didn't pre-install that already, you are sorta screwed.

Start with

$ sudo apt purge linux-image-5.4.0-9*
$ sudo apt purge linux-image-5.4.0-10[0-6]*  linux-image-5.4.0-109-generic linux-image-5.4.0-110-generic
$ sudo apt autoremove purge

DO NOT REBOOT! It could go bad. That should leave just the linux-image-5.4.0-107-generic kernel, which appears to be what you are booted into. You can verify that with uname -r

$ sudo apt update

should install 1 new kernel, cleanly. If it doesn't, don't reboot and post back here the errors.

[Impavidus]

You have to remove the linux-image-..., the linux-modules-... and linux-modules-extra-... packages. Not the linux-image-extra-... package, which isn't installed. The way how kernels are packaged changed some time ago, so there may still be old guides around telling you to remove linux-image-extra-...

linux-image-... cannot be removed before linux-modules-extra-... (because the latter depends on the former) and linux-modules-... cannot be removed before linux-image-..., so if you want to remove all three of them separately, the order is linux-modules-extra-..., linux-image-..., linux-modules-..., but you can tell dpkg to remove all of them at the same time.

You currently run kernel 5.4.0-107, so I suggest you remove 5.4.0-99, -100 and -109, then use apt to fix things (which will fully install -110). And then a regular upgrade should also install -121. After rebooting, remove -107, but that should work the normal way. And remove the header packages.

sudo apt update
sudo dpkg --purge linux-modules-extra-5.4.0-99-generic linux-image-5.4.0-99-generic linux-modules-5.4.0-99-generic
# The same for -100 and -109
sudo apt install -f
sudo apt upgrade

[elpidiovaldez5]

Many thanks for your advice. I tried the purge and update ideas. There were a few hiccups, but I did free some space and manage to update the system. Now my system will boot 5.4.0-121-generic, but only in recovery mode. I kind of expected this, since some of the hiccups wer complaining about nvidia packages.

The current state of my system is:

paul@clio:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  2.1M  3.2G   1% /run
/dev/nvme0n1p3  468G  136G  309G  31% /
tmpfs            16G   48M   16G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs            16G  620K   16G   1% /tmp
/dev/nvme0n1p2  510M  449M   25M  95% /boot
/dev/nvme0n1p1  532M  5.3M  527M   1% /boot/efi
/dev/loop0      128K  128K     0 100% /snap/bare/5
/dev/loop2      163M  163M     0 100% /snap/gnome-3-28-1804/145
/dev/loop1       56M   56M     0 100% /snap/core18/2409
/dev/loop5       62M   62M     0 100% /snap/core20/1494
/dev/loop4      401M  401M     0 100% /snap/gnome-3-38-2004/112
/dev/loop3      165M  165M     0 100% /snap/gnome-3-28-1804/161
/dev/loop7      915M  915M     0 100% /snap/libreoffice/256
/dev/loop6       92M   92M     0 100% /snap/gtk-common-themes/1535
/dev/loop8       56M   56M     0 100% /snap/core18/2344
/dev/loop9       62M   62M     0 100% /snap/core20/1518
/dev/loop10      82M   82M     0 100% /snap/gtk-common-themes/1534
/dev/loop11     255M  255M     0 100% /snap/gnome-3-38-2004/106
/dev/loop12      47M   47M     0 100% /snap/snapd/16292
/dev/loop13     918M  918M     0 100% /snap/libreoffice/254
/dev/loop14      47M   47M     0 100% /snap/snapd/16010
tmpfs           3.2G   12K  3.2G   1% /run/user/1000
/dev/sdb1       1.9T 1008G  856G  55% /media/paul/Seagate Backup Plus Drive

Installed kernels are:

paul@clio:~$ dpkg -l | tail -n +6 | grep -E 'linux-image-[0-9]+'
ii  linux-image-5.4.0-107-generic                 5.4.0-107.121                        amd64        Signed kernel image generic
ii  linux-image-5.4.0-110-generic                 5.4.0-110.124                        amd64        Signed kernel image generic
ii  linux-image-5.4.0-121-generic                 5.4.0-121.137                        amd64        Signed kernel image generic

kernel 107 no longer boots.

kernel 121 will boot using nouveau drivers, but all the nvidia drivers complain of dependency errors.

pk-client-error-quark: The following packages have unmet dependencies:
  nvidia-driver-515: Depends: libnvidia-gl-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-extra-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-decode-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-encode-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: xserver-xorg-video-nvidia-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-cfg1-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-fbc1-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Recommends: libnvidia-compute-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
                     Recommends: libnvidia-decode-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
                     Recommends: libnvidia-encode-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
                     Recommends: libnvidia-fbc1-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
                     Recommends: libnvidia-gl-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
 (268)

I have minimal space left in /boot, and need to be really careful not to destroy the system now that kernel 107 no longer works.

What do you think I should do now ?

[elpidiovaldez5]

I also tried installing the nvidia package and got the following output:

paul@clio:~$ sudo apt install nvidia-driver-515
[sudo] password for paul: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies.
 nvidia-driver-515 : Depends: libnvidia-gl-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-extra-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-decode-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-encode-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: xserver-xorg-video-nvidia-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-cfg1-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Depends: libnvidia-fbc1-515 (= 515.48.07-0ubuntu0.20.04.2) but it is not going to be installed
                     Recommends: libnvidia-compute-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
                     Recommends: libnvidia-decode-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
                     Recommends: libnvidia-encode-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
                     Recommends: libnvidia-fbc1-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
                     Recommends: libnvidia-gl-515:i386 (= 515.48.07-0ubuntu0.20.04.2)
E: Unable to correct problems, you have held broken packages.

However no held packages are found by dpkg --get-selections | grep hold

[Impavidus]

If kernel -107 no longer works, you can just as well remove it. The linux-modules-extra-..., linux-image-... and linux-modules-... packages. You can also remove the linux-headers-... packages for -107, but those don't take any space in /boot.

Does the -110 kernel work properly? And the -121 kernel only using the nouveau driver?

I've got no experience with nvidia drivers, so I'll leave this to people who have (unless none appear).

[TheFu]

If the nvidia dkms isn't working, the quick fix is usually to let the ubuntu-drivers tool handle it.

sudo ubuntu-drivers autoinstall

For each new kernel, this might be necessary, until either the nvidia dkms is fixed or you fix it. I'm lazy. On my nvidia system, if my patch script sees that a new kernel was installed, then it just re-runs the command above. I haven't bothered looking into the "correct" solution and it has been over 2 yrs that this has been working.

BTW, you might want to remove all the old nvidia driver packages. If you are using nouveau, now is the perfect time.

A few other tidbits. When it comes to storage use, all the loop devices are useless. They aren't taking up any storage that isn't accounted for elsewhere. Same for all the tmpfs and devfs stuff. We never need to see those things. They just add cruft to your posts and confuse any real storage issue. This alias (add to your normal alias lists in your dotfiles). I've added another handy alias for storage stuff:

alias dft='df -hT -x squashfs -x tmpfs -x devtmpfs'
alias lsblkt='lsblk -e 7 -o name,size,type,fstype,mountpoint'

Clean, useful, information.

[ajgreeny]

BTW, you might want to remove all the old nvidia driver packages. If you are using nouveau, now is the perfect time.

I am fairly sure that it is essential to remove an older, non-working nvidia driver before attempting to install another so follow TheFu's idea and you may get things going again.