Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 26

Thread: Unable to clean/clear old kernels from /boot...

  1. #11
    Join Date
    Oct 2010
    Beans
    19

    Re: Unable to clean/clear old kernels from /boot...

    Quote Originally Posted by TheFu View Post
    Ouch. You are stuck. This problem has been there, 2-3 week, at least. Like a leaky roof, out of storage errors while patching don't get better over time.

    There are some things I'd try, but I have excellent backups, so I can put things back easily. If you are willing to go to extreme measures, let me know. Be certain you have everything you want to keep backed up. It would entail moving some files to make room, but create empty files in /boot/ so APT doesn't know any better. Then do the sudo apt-get -f install to get out of APT hell. Then come back and use some aggressive apt-get purge package globbing to remove all but 3 kernels. And lastly, run an autoremove. If all that works, you can wipe the kernel files that were copied somewhere else. Only move the kernel files that won't be used, for any needed reboots.

    Next time around, run sudo apt autoremove every 2 weeks or so. This will clean up all but 2-3 kernels. Under 200MB should be needed in /boot/.

    Holy crap. Okay. Believe it or not I'm actually relieved it wasn't something easy I'm missing. Yes, I do suck at autoremove, better believe I'll be better now.

    The good news is that this is a VM in ESXi 6.5 and I have a snapshot taken before I started any of this. So if things get rough, I can always restore the snapshot back to before I started.

    I would very, very much appreciate any instructions or advice you would be so kind to give!

  2. #12
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Unable to clean/clear old kernels from /boot...

    Be very careful, NOT to delete the kernel that is currently booting or any of those support files.

    There are 4 kernels and supporting files trying to be installed, but failing because there isn't any space left. I don't know how to skip those failed installs. We need to make room, get them installed, then remove them. Need to remove the oldest, largest, kernels from /boot/, but that will make APT unhappy.

    Any files that are deleted, need to be recreated, so the filenames exist, but they don't need to have any specific contents or size. I'd use touch to create zero-length files. I've used this technique before for other package that wouldn't install/remove, but never for kernel failures. I'm hoping this works.

    If not, you can always reinstall or clone the datastore and shift everything right, so you can double the size of /boot/.

    If this took me over 1 hr to resolve, I would have invoked my restore process, where I can resize any storage areas as needed. Takes about 45 minutes, at most, to have the system up and working exactly as before.

  3. #13
    Join Date
    Aug 2011
    Location
    51.8° N 5.8° E
    Beans
    4,720
    Distro
    Xubuntu 19.04 Disco Dingo

    Re: Unable to clean/clear old kernels from /boot...

    Usually dpkg can still remove old kernel packages even when apt no longer works. The problem can probably be solved with just a few commands. First identify which packages are installed and which kernel is currently running, than use dpkg to remove one or two, then fix apt and use autoremove to remove the other old kernels.

    Can you show the output of
    Code:
    dpkg --list | grep linux-
    uname -r
    df -h /boot

  4. #14
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Unable to clean/clear old kernels from /boot...

    Code:
    dpkg -l |egrep linux-image | grep '^ii'
    would be better to show just the currently installed linux kernels. without the 2nd grep, removed kernels will be displayed too. While I avoid using dpkg to install anything, getting information about installed packages is handy.

  5. #15
    Join Date
    Aug 2011
    Location
    51.8° N 5.8° E
    Beans
    4,720
    Distro
    Xubuntu 19.04 Disco Dingo

    Re: Unable to clean/clear old kernels from /boot...

    Not showing the rc (removed, remaining configuration) packages is a good idea, but I prefer to see the linux-generic, linux-headers* and linux-modules-*, as well as the partially installed packages, too. The fully removed packages won't show.

    You could make it
    Code:
    dpkg -l | grep linux- | grep -v '^rc'

  6. #16
    Join Date
    Mar 2007
    Beans
    471

    Re: Unable to clean/clear old kernels from /boot...

    Have you tried
    Code:
     sudo apt autoclean
    ? You might even try
    Code:
    sudo apt clean
    I don't see that mentioned. There may be a lot of trash hanging around from old upgrades that autoclean or clean can clear out.
    The space problem seems to be in
    Code:
     /var/cache/apt/archives/
    Also a
    Code:
    df -k
    might show which files systems are full and give some insight into your setup.

    Looking at the autoremove package list. Why can't he just remove the 5 oldest kernel versions - 4.4.4.0-130, 4.4.4.0-134, 4.4.4.0-137, 4.4.4.0-138, and 4.4.4.0-139?
    I'm asking not suggesting that but seeing how many kernel versions are installed it seems highly unlikely any that old are actually being used.
    I'm hoping someone can enlighten me.

  7. #17
    Join Date
    Aug 2011
    Location
    51.8° N 5.8° E
    Beans
    4,720
    Distro
    Xubuntu 19.04 Disco Dingo

    Re: Unable to clean/clear old kernels from /boot...

    Quote Originally Posted by rsteinmetz70112 View Post
    Looking at the autoremove package list. Why can't he just remove the 5 oldest kernel versions - 4.4.4.0-130, 4.4.4.0-134, 4.4.4.0-137, 4.4.4.0-138, and 4.4.4.0-139?
    I'm asking not suggesting that but seeing how many kernel versions are installed it seems highly unlikely any that old are actually being used.
    I'm hoping someone can enlighten me.
    The problem is the /boot partition, which is full. Systems with a separate /boot partition often suffer from this problem. The solution is indeed to remove some of the older kernels, but unfortunately the package management is now in an inconsistent state. The metapackage (linux-image or linux-image-generic or something like that) that pulls in the next kernel package (linux-image-w.x.y-z-generic) has been upgraded, but installation of the new kernel package has failed because of lacking space. Now apt first wants to make the package system consistent by completing installation of the new kernel package before it's willing to do anything else, like removing old kernel packages, but that can't be done before those old kernel packages have been removed. So you have to use lower level tools to get those old kernel packages removed.

  8. #18
    Join Date
    Mar 2007
    Beans
    471

    Re: Unable to clean/clear old kernels from /boot...

    So why can't those files in /boot simply be deleted? It will create an additional inconsistency which, given some space to work apt might be able to handle.

    Of course the packages might be able to be removed using dokg, which is what apt mostly uses to remove packages, but dpkg does a lot less checking, which in this case might be good. You could try something like
    Code:
    dpkg -r linux-image-extra-4.4.0-130-generic
    dpkg -r linux-image-4.4.0-130-generic
    Then work your way through the packages autoremove wants to remove.Until you get enough space in boot to run autoremove.
    Based on the message above it looks like you will need something like 200 megabytes.
    Code:
    After this operation, 201 MB of additional disk space will be used.
    According to the code posted earlier the packages you want to remove would be:
    linux-headers-4.4.0-150
    linux-headers-4.4.0-150-generic
    linux-headers-4.4.0-151
    linux-headers-4.4.0-151-generic
    linux-image-4.4.0-130-generic
    linux-image-4.4.0-134-generic
    linux-image-4.4.0-137-generic
    linux-image-4.4.0-138-generic
    linux-image-4.4.0-139-generic
    linux-image-4.4.0-141-generic
    linux-image-4.4.0-142-generic
    linux-image-4.4.0-143-generic
    linux-image-4.4.0-150-generic
    linux-image-4.4.0-151-generic
    linux-image-extra-4.4.0-130-generic
    linux-image-extra-4.4.0-134-generic
    linux-image-extra-4.4.0-137-generic
    linux-image-extra-4.4.0-138-generic
    linux-image-extra-4.4.0-139-generic
    linux-image-extra-4.4.0-141-generic
    linux-image-extra-4.4.0-142-generic
    linux-modules-4.4.0-143-generic
    linux-modules-4.4.0-150-generic
    linux-modules-4.4.0-151-generic
    linux-modules-extra-4.4.0-143-generic
    linux-modules-extra-4.4.0-150-generic
    linux-modules-extra-4.4.0-151-generic
    I noticed that all of the same packages are not installed for every kernel version.

    I also noticed that a lot of packages have not been updated:

    Code:
    0 upgraded, 
    6 newly installed, 
    0 to remove and 
    178 not upgraded.
    16 not fully installed or removed.

  9. #19
    Join Date
    Aug 2011
    Location
    51.8° N 5.8° E
    Beans
    4,720
    Distro
    Xubuntu 19.04 Disco Dingo

    Re: Unable to clean/clear old kernels from /boot...

    Quote Originally Posted by rsteinmetz70112 View Post
    So why can't those files in /boot simply be deleted? It will create an additional inconsistency which, given some space to work apt might be able to handle.

    Of course the packages might be able to be removed using dokg, which is what apt mostly uses to remove packages, but dpkg does a lot less checking, which in this case might be good. You could try something like ...
    Exactly, that's the plan. And as dpkg does less checking, we have to do that ourselves. That's why I first want a list of all installed and partially installed kernels and the currently running version. The list of files in /boot may give false information in case of partially installed packages and I don't want to create even more inconsistencies by manually removing files, which could confuse dpkg when it tries to remove a package that's not properly installed, when package management says it's fully installed. The removal script may return an error, which can only be solved by reinstalling the package, then removing it again, when there's no room for that.

    So let's wait until OP reports back.

  10. #20
    Join Date
    Mar 2007
    Beans
    471

    Re: Unable to clean/clear old kernels from /boot...

    What kernel is normally booting?
    Code:
    # uname -r
    Anything older than that should be relatively safe to remove.
    If you remove those older kernels with dpkg it shouldn't cause any real problems. I'd start with the oldest first and work my way forward until you have enough space in /boot.
    I have seen situations where removing the kernel does not update /boot even using more sophisticated tools like synaptic. In that case you should try to remove them with
    Code:
    # update-initramfs -d -k <Kernel version number>
    # example:
    # update-initramfs -d -k 4.4.0-130
    If that fails you will need need to rm the files manually.
    The update-initramfs command doesn't remove all of the files in /boot related to a specific kernel. Those have to be removed manually.

    Typically each kernel installed has at least the following files in /boot:

    abi-<kernel number>
    config-<kernel number>
    initrd-<kernel number>
    System.map-<kernel number>
    vmlinuz-<kernel number>
    On my machine each of these sets of files takes up around 50 megabytes so you would likely would need to remove 5 sets to get enough space according to the information previously posted. That would be 4.4.0-130, 4.4.0-134, 4.4.0-137, 4.4.0-138, and 4.4.0-139. It is possible there kernels aren't even still installed.

    You can probably remove these older files and still boot as long as you don't try to boot that kernel but it's highly not recommended. In any event as long as the kernel files are installed these files will probably get recreated.

    Once you get enough space you can try
    Code:
    $ sudo -i
    # apt autoremove
    # apt autoclean
    or
    Code:
    $ sudo apt install -f 
    $ sudo dpg --configure -a
    $ sudo apt autoremove
    $ sudo apt autoclean
    either should be able to fix it.

    After that I suggest you run:

    Code:
    # update-initramfs -u all
    # update-grub
    Although the apt install will typically update this stuff.
    Then make sure everything is working properly
    Last edited by rsteinmetz70112; 2 Weeks Ago at 07:20 PM. Reason: to correct typos

Page 2 of 3 FirstFirst 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •