Clean up installation

[Esokra]

Hello

I am a happy Ubuntu user since a few years. I have been using Ubuntu as my main operating system, as well in different Virtual machines etc. One issue I have run into very often is, running out of space, although I did not install any additional software (not even updates, as it was in a VM), nor did I save so many files etc.
To make an experiment, I set up Ubuntu in a VM, used it a few months heavily and tried to clean up the installation, completely.
I purged all additional software with

sudo apt-get --purge remove

removed all unneeded packages with

sudo apt-get --purge autoremove

and finally ran the following commands

sudo apt-get autoclean
sudo apt-get clean

Finally I added a new user and deleted the old one (I did not need the data anymore).
So I only had all packages left, which were installed by default (even less, as I had deleted libre office). Compared to a clean install (3.1 GB) there were 4.5 GB, so I am asking right now, where the 1.4GB come from? I even deleted all the track history under privacy in the system settings. Are this log files? I can warrant, that I have never saved anything in the root homefolder, or in the file system outsie home.
Are this 1.4 GB all log files?
What else can be done in order to clean the installation?(I know it does not make sense, but I want to keep my installs clear over a longer period of time)

Thank you in advance!

[mikewhatever]

Let's see how many kernels and headers there are:

dpkg -l| grep -e linux-image-[0-9] -e linux-headers-[0-9]

Quite a few of them can get stacked up in a few months time.

[oldsoundguy]

Many use the third party "ubuntu tweak" to do the clean up. I do, Run it about once a month. (get rid of those excess kernels) But, I store my pictures, videos & music off of my main hard drive, so never run out of space using a 250gb HD.(storing off drive makes doing a back up a lot easier .. only need to copy my ~/home folder.)

[Esokra]

Thank you for answering!
I wanted to mention it first, that there are no additional kernels, but then I decided to not write it explicitly, as I thought it would be a consequence of not updating, which, I have to admit, is not always true.
So there are no additional kernels, as the output confirms:

ii  linux-headers-3.2.0-23                 3.2.0-23.36                             Header files related to Linux kernel version 3.2.0
ii  linux-headers-3.2.0-23-generic         3.2.0-23.36                             Linux kernel headers for version 3.2.0 on 64 bit x86 SMP
ii  linux-image-3.2.0-23-generic           3.2.0-23.36                             Linux kernel image for version 3.2.0 on 64 bit x86 SMP

I cleaned with Ubuntu tweak too and uninstalled it aftwerwards, so everything is clean, according to the Janitor of Ubuntu Tweak.

[Esokra]

Today I found an interesting issue:

While passing the wrong option to a tmpfs Ramdisk (in etc/fstab), the system did not boot, so I corrected the entry from shell. When I rebootet, there was a noticeable disk usage difference of several megabytes. Does Ubuntu make backups of system files?

[mikewhatever]

How about running something like this:

du -h | sort -nr | head -n20

That will show the list of 20 largest files in you home folder.

[Esokra]

Hello

I could find the problem: There were many folders left from arkose sandbox and lxc, by deleting them, I freed up much space.
Now I have nearly a clan install, but there are a few questions left for me:

Although exactly all the same packages are installed (comparing a new clean install with the old cleaned one) there are a few differences (All folders despite usr, lib and etc are approximately (only a few kb difference) equal in terms of space) I do not understand:

usr is 500 MB larger on the cleaned up (old) install
lib is 4 MB larger on the cleaned up (old) install
etc is 2 MB larger on the cleaned install

(According to du)

So what makes this huge difference? (I really made sure to uninstall everything, which is not on a clean system (I did this with deborphan, apt-get --purge remove, autoremove and manually with find, for example: find | grep google-chrome | xargs rm -rf). Furthermore I cleaned up var (which made another 500 MB difference).
What makes the huge difference for /usr ? As far as I am concerned this folder contains user programs, but how is the difference possible, if I removed all additional software?

I would be really glad if you could help me to solve that problem!

Thanks for all that help!

[Esokra]

Has anybody an idea how to examine the difference (I do not want and I do not have the time to go through the content of these directories. ) I have already tried to save the output of ls -lh of these directories and use diff for the files, but still I would have to go manually trough the lines. How to find out the additional files and directories for comparing to directories?

Another idea would be, to use the package manager to examine which packages are installes (all) and comapre it (but still I would need an comfortable way to compare this information on two different system). I think this would be sensible, as I must have missed packages, since the usr directory only contains programmes.

[mikewhatever]

Run 'du -h /usr | sort -nr | head -n20' on the clean and cleaned install to see the dufference of the /usr directories. I don't know of any other way to troubleshoot it.

[Esokra]

For both systems I did:

dpkg -l | grep ^ii | sed 's_  _\t_g' | cut -f 2 > installed-pkgs

in order to get a list of installed packages.
Then I named one file o and the other n. (o is cleaned up install, n is new install)
After this I sorted both files with:

sort -o so o
sort -o sn n

After this operation I ran

comm -31 so sn >> in-sn-notin-so.txt 
comm -32 so sn >> in-so-notin-sn.txt

in order to sort out the packages which are installed on both systems.
"in-sn-notin-so.txt" contains the packages which are missing on the cleaned up install, but are part of a new install (I decided to install these)
"in-so-notin-sn.txt" contains the packages which are on the cleaned up install, but are not part of a new install. This packages are the ones we want to remove.

!!!! But we have to be careful, as I noticed, that the clean install had a different kernel version, so we have to modify both files and remove the lines beginning with "linux-".

As these are many packages and we want pack them in one command "sudo apt-get --purge remove ... "
we have to transform the files consisting of many lines in a file, containing all these packages in one line. This is done with:

tr '\n' ' ' < in-sn-notin-so.txt  > install.sh
tr '\n' ' ' < in-so-notin-sn.txt  > remove.sh

So finally I modifed remove.sh to:

#! /bin/bash
apt-get --purge remove (here comes the content of remove.sh)

and made it executable (chmod u+x remove.sh) . I did the same with "apt-get install" for the file install.sh.

After I ran both scripts finally both installs had the same size, so cleaning up the old system was sucessful!

After all I ask myself why all these packages were not removed (as these packages were not deliberately installed, so they were dependencies) by the "apt-get --purge remove" command? For example I have never installes "apache2-utils" etc.
An explanation would be, that I have installed packages from ppa (but not many) and apt-get can not find or uninstall all the dependencies?
What can I do in order to prevent these things in the future? It seems apt-get is not fully capable of removing all dependencies coming with one package, is there a better alternative?
What to do in order to remove all dependencies while removing an application.?
I could write a script similar to the way we cleaned up the install, which captures the difference of installed packages while installing and then removes all this, when I want to remove the package. But is there a better way?

Thank you for all your help and ideas!