12.04 Upgrade failure due to disk space: 3 possible solutions??

[jaspiel6654]

I am getting an upgrade failure and the reason seems to be a lack of disk space. After some investigation via Google, it seems that the problem is really that the INODE address capability is full (or nearly full). The solution (1) given is to increase the size of the appropriate partition (in this case /) using GPARTED. That is okay and I could do it but this is a three system PC (12.04, Debian, and Windows 7, so I am reluctant to start moving the partitions around. How about one of the following: 2) Could I set up a new partition and move /usr (this seems to be the directory that is the source of the most INODE usage) to it and mount it, thus moving the INODE requirement to a new parition or, 3) could I set up a new partition and set up a symlink to /usr and move the existing /usr to the new partition and leave the mount commands as they are? I like choices 2 or 3 better than 1 because it moves the problem area to a separate location that I can monitor and fix more easily in the future. BTW, I am just learning Linux at the command level, so if my "solutions" above just don't make any sense, that is why.

My current workaround is to just go the "prior linux" at the boot and use the last prior successful version.

Thanks for your help, Jim

[fantab]

Tell us what does the output of


df -h

says.

[jaspiel6654]

Thanks for the quick response. I have included a couple of other bits of information.

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 9.9G 7.7G 1.7G 83% /
udev 4.0G 4.0K 4.0G 1% /dev
tmpfs 1.6G 932K 1.6G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 4.0G 156K 4.0G 1% /run/shm
/dev/sda5 20G 14G 4.9G 75% /home

also df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 655360 649887 5473 100% /
udev 196185 608 195577 1% /dev
tmpfs 199838 552 199286 1% /run
none 199838 3 199835 1% /run/lock
none 199838 7 199831 1% /run/shm
/dev/sda5 1310720 17606 1293114 2% /home

and finally, ls --inode
393219 bin
524290 boot
156974 cdrom
1025 dev
393217 etc
2 home
2053 initrd.img
2122 initrd.img.old
524291 lib
11 lost+found
131073 media
524292 mnt
8193 opt
1 proc
16385 root
1228 run
24577 sbin
24578 selinux
32769 srv
1 sys
40961 tmp
40962 usr
131074 var
1062 vmlinuz
2572 vmlinuz.old

Somewhere I made a mistake in assuming that /var was the main user of inodes, there are several other directories that have higher usage but I think that my question about methods still is valid???
JIm

[Bashing-om]

jaspiel6654; Hi !

This:


also df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 655360 649887 5473 100% /
AND
and finally, ls --inode
393219 bin
524290 boot

Says there is a 4th way to handle this situation -> delete all those old kernel images and headers.

Confirm:


cd /
sudo du -sx * | sort -n

If you need to drill down further, use cd to move to a directory of interest then repeat the du command.
The results are in megabytes. ( like maybe "cd /boot" )

To see all those un-needed kernels:


dpkg -l | grep linux-image
dpkg -l | grep linux-headers


One MUST not remove the current kernel and it is advisable to keep at least 1 backup kernel;
The current booted kernel is known by:


uname -r


Once we know what the kernel versions are, we can craft a nifty command sequence to remove them.
IF my suspicions prove to be the source of the problem.

Cheers ->




it's ubuntu



many ways to one end

[jaspiel6654]

Bashing-om:
results of du -sx...
0 initrd.img
0 initrd.img.old
0 proc
0 sys
0 vmlinuz
0 vmlinuz.old
4 cdrom
4 dev
4 media
4 mnt
4 opt
4 selinux
4 srv
16 lost+found
40 root
48 tmp
932 run
8644 bin
8768 sbin
13960 etc
422176 boot
658792 var
2299736 lib
4467104 usr
14365760 home


BTW, /home is in its own partition.

I did the list of the images and headers and get about a dozen, maybe a little more of each. And there is one more than the one that comes up in uname -r, so it (they) are probably the one that failed during the install. Couple of questions 1) will removing the old kernels free up enough of the INODE space to complete the current install. Probably, but I guess this becomes an ongoing maintenance issue, right? Also, in terms of removing kernels, how do you like the synaptic solution outlined at
http://askubuntu.com/questions/2793/how-do-i-remove-or-hide-old-kernel-versions-to-clean-up-the-boot-menu
?
2) just for my edification, would my solutions 2 and 3 work?
and finally, where are the kernel hidden? I searched around using ls could not find them.
Thanks, Jim

[Bashing-om]

jaspiel6654;

Removing kernels with synaptic: Works great and is the preferred GUI method. -Mark for complete removal ! -
Then I still like to go back with the terminal and see what the package manager thinks, so I just do the removal from terminal anyway in a prescribed procedure to do so .. Removing the kernels otherwise will wreck havoc with the file management system.

Your solution #1, one could increase the partition size, but - risky to say the least, and is only a temporary solution, as eventually even that added space will be consumed.

Your solution #2, Most assuredly you can, and that "could" help bunches:
I do, and for reference, how I have my system disks set up:


sysop@1310mini:/$ sudo blkid
[sudo] password for sysop:
/dev/sda1: LABEL="1310mroot" UUID="3a47f1f1-ed1f-4134-b6aa-be101a7d97b4" TYPE="ext4"
/dev/sda2: LABEL="1310mhome" UUID="29a6fc4f-ff12-4cac-8eb1-e98e50f1107f" TYPE="ext4"
/dev/sda5: UUID="192a4783-56fa-4fd0-a62f-c45a14c08482" TYPE="swap"
/dev/sda6: LABEL="DATA" UUID="3ad091a1-5036-463b-ba4e-88e98e41b07a" TYPE="ext4"
/dev/sda7: LABEL="LUBU1310" UUID="4e6cd96d-49bd-47f0-9dfe-8eeebad4cf9d" TYPE="ext4"
/dev/sda8: LABEL="1310mvar" UUID="136af805-5758-4880-acc4-0e1d35e2c266" TYPE="ext4"
/dev/sdb1: LABEL="my_stuff" UUID="6a24777c-8191-4230-81f1-376f31b321e5" TYPE="ext4"
/dev/sdc1: LABEL="ubie1204" UUID="5bae8c40-b15d-42f8-9fc9-0cf087f338d4" TYPE="ext4"
sysop@1310mini:/$

However, that is above my skill set to set that up post install. With a separate /var directory, one has to pay closer attention to it, Watch it that the files (spamming also) do not consume and overrun.

Your solution #3, certainly doable, never having had to do it, I do not have the ready knowledge to implement it. I would be more than willing to explore that with you, oldfred has posted instructions to symlink directories.

All the above, however, - if the problem is elsewhere than /var - will not address the present issue. There is no substitute for removing files to free up inodes.
You will save approximately 120Mb per kernel and header files that are removed. Adds up to a substantial amount.
My /boot size with several old kernels still installed: "86548 boot".
Your /var is rather large, might take a look and make sure there is nothing running amuck and logging to the files. If needed one can manually delete the old files to get some operating head room.
Your /home is the huge culprit, suggest ya delete stuff that you no longer need, and/or move some stuff to an external "storage" medium.

The more direct thing to do presently is remove those old kernels and headers, that will help the most/quickest.
Hopefully, at this point the package manager is not going to scream and holler when we do so.

In the event that a better long term solution is required, well, you (re-)partition and in the partition set up there is a means to increase the inode allocation above what is the default.

We build bigger and bigger buildings to hold more and more stuff, sometimes the best thing is just throw the stuff out the back door !

But the good thing -> there are solutions,



so where's the problem

[jaspiel6654]

Bashing-om,

Thanks for your comments. I will go off and try out these solutions and get back with the results and documentation. It may be a couple of days. Thanks again.

Jim

[Bashing-om]

jaspiel6654; Roger that !

Study twice, act once.
We will be here and I will see you when you post and we will pick this back up.



real tough to put 6 pounds of sugar in a 5 pound sack

[jaspiel6654]

Bashing-om,
Sorry for the delay, but I've been away.

However, the following notes:
First, I tried deleting the unneeded versions of ubuntu and could ;not get beyond a variety of error messages. I tried using the various apt- functions but without success. Maybe because my package is screwed up due to the inability to complete the install?

Second, I did some work on a laptop I have for experimentation and was able to create a new /lib on a new partition. I followed the instructions in the following :: https://help.ubuntu.com/community/Partitioning/Home/Moving that is aimed at moving your /home directory to a new partition if you did not create a separate partition for /home when you installed ubuntu. It seems to have work fine. I do have a question. I ignored the last step of saving /home to /old-home before rebooting, since I made the assumption that by updating fstab to mount /lib on /dev/sdb8 it would leave /lib on the initial drive alone, but now I think that assumption is incorrect. Can there only be one /lib on the system? What happened to the /lib on the / deviced when the mount /lib on the /dev/sdb8 occurred?

I used /lib as an experiment since it wasn't too large but was large enough to use to test with and would free up enough inodes to get me back to a successful install of the lastest update of ubunty. Then I would like to go back and figure out how to get rid of the extra versions of linux.

Comments?
Jim

[Bashing-om]

jaspiel6654; UHHHMM,

My suggestion is still to put the effort into cleaning up /boot, purge those old kernels.
I have had good results with "dpkg" at a lower level in removing kernels.

As to the /lib(s). Did the symlinks get established ?
What returns from:


ls -la /
from the /home directory ? There can be but one directory active.

not a thing wrong in try'n

[jaspiel6654]

output of ls -la / >output

total 104
drwxr-xr-x 23 root root 4096 Jan 17 14:23 .
drwxr-xr-x 23 root root 4096 Jan 17 14:23 ..
drwxr-xr-x 2 root root 4096 Jan 10 15:08 bin
drwxr-xr-x 3 root root 4096 Jan 10 15:09 boot
drwxr-xr-x 2 root root 4096 Dec 13 16:08 cdrom
drwxr-xr-x 16 root root 4300 Jan 30 15:58 dev
drwxr-xr-x 139 root root 12288 Jan 30 15:58 etc
drwxr-xr-x 4 root root 4096 Dec 13 16:08 home
lrwxrwxrwx 1 root root 32 Jan 10 15:09 initrd.img -> boot/initrd.img-3.5.0-45-generic
lrwxrwxrwx 1 root root 33 Jan 10 15:08 initrd.img.old -> /boot/initrd.img-3.5.0-45-generic
lrwxrwxrwx 1 root root 14 Jan 17 14:23 jawdir -> /home/jim/jaw/
drwxr-xr-x 22 root root 4096 Dec 13 17:07 lib
drwx------ 2 root root 16384 Dec 13 16:03 lost+found
drwxr-xr-x 4 root root 4096 Jan 27 14:51 media
drwxr-xr-x 2 root root 4096 Apr 19 2012 mnt
drwxr-xr-x 2 root root 4096 Aug 17 2012 opt
dr-xr-xr-x 195 root root 0 Jan 30 10:58 proc
drwx------ 11 root root 4096 Dec 29 15:25 root
drwxr-xr-x 22 root root 780 Jan 30 15:58 run
drwxr-xr-x 2 root root 12288 Jan 10 15:08 sbin
drwxr-xr-x 2 root root 4096 Mar 5 2012 selinux
drwxr-xr-x 2 root root 4096 Aug 17 2012 srv
dr-xr-xr-x 13 root root 0 Jan 30 10:58 sys
drwxrwxrwt 9 root root 4096 Jan 30 17:17 tmp
drwxr-xr-x 11 root root 4096 Aug 17 2012 usr
drwxr-xr-x 14 root root 4096 Jan 29 19:38 var
lrwxrwxrwx 1 root root 29 Jan 10 15:09 vmlinuz -> boot/vmlinuz-3.5.0-45-generic
lrwxrwxrwx 1 root root 29 Jan 10 15:08 vmlinuz.old -> boot/vmlinuz-3.5.0-45-generic

I'll go back and look at dpkg and its variants and maybe send you some of my results.. Remember, the output directly above is from my laptop, not the system which is having trouble. The laptop is a test to see what actions I can take to at least get me to a running system.

Thanks.

[Bashing-om]

jaspiel6654; Well, well ;

as you can see the symlink for /lib was not established.

The easy way to remove those troublesome kernels/headers:


uname -r ##to KNOW what kernel is in use - DO NOT remove it !
dpkg -l | grep linux-image ##list of all kermel images installed
sudo dpkg -P linux-image-3.0.0-{12,13,14,15,16,17}-generic-pae ##substitute your kernel version, sequence numbers, and type (generic ?)
dpkg -l | grep linux-headers
sudo dpkg -P linux-headers-3.0.0-{12,13,14,15,16,17}-generic-pae ##same same as above for the header files.


If that is done, will look about a bit of cleanup of packages with remaining config files still on the system.

The easy way is



worth a shot

[jaspiel6654]

I did a --list with dpkg (see below) and I assume that these would be the kernels name we would be looking at, but this system, which is running 12.10 only keeps the current one and one back, unlike what 12.04 seems to do. But if I wanted to delete one of these, what would I do next - use the apt-get remove linux-image-3.2.0-29-generic-pae command?

Thanks

ii linux-headers-generic 3.5.0.45.61 i386 Generic Linux kernel headers
ii linux-headers-generic-pae 3.5.0.45.61 i386 Transitional package
ii linux-image-3.2.0-29-generic-pae 3.2.0-29.46 i386 Linux kernel image for version 3.2.0 on 32 bit x86 SMP
ii linux-image-3.5.0-44-generic 3.5.0-44.67 i386 Linux kernel image for version 3.5.0 on 32 bit x86 SMP
ii linux-image-3.5.0-45-generic 3.5.0-45.68 i386 Linux kernel image for version 3.5.0 on 32 bit x86 SMP
ii linux-image-extra-3.5.0-44-generic 3.5.0-44.67 i386 Linux kernel image for version 3.5.0 on 32 bit x86 SMP
ii linux-image-extra-3.5.0-45-generic 3.5.0-45.68 i386 Linux kernel image for version 3.5.0 on 32 bit x86 SMP
ii linux-image-generic 3.5.0.45.61 i386 Generic Linux kernel image
ii linux-image-generic-pae 3.5.0.45.61 i386 Transitional package
ii linux-libc-dev:i386 3.5.0-45.68 i386 Linux Kernel Headers for development

[Bashing-om]

jaspiel6654; Well,,,

You had advised earlier that "apt-get remove" did not give good results.
Now, until I know what version you are running, I am unable to give you explicit directions.
show:


uname -r
dpkg -l | grep linux-image

And yes, version 12.04 does not "autoremove" old kernels, must be done manually as we are doing.



one step at a time

[jaspiel6654]

ok, tomorrow will do the cleanup on the problem system in removing the unneeded images and headers. I assume from your comment about the symlink for /lib that the system still recognizes the original /lib, ie. the one on the same partition as /.

I had assumed from the output from mount (see below) that the new /lib was ineffect. Hmmmm. Much to learn I guess.
/dev/sdb5 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
/dev/sdb2 on /home type ext4 (rw)
/dev/sda1 on /usr type ext4 (rw)
/dev/sdb8 on /lib type ext3 (rw)
gvfsd-fuse on /run/user/jim/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,user=jim)

[Bashing-om]

jaspiel6654; Well;

It is like this, one never quits learning this system !

As to symlinks:
This shows a symlink, from your output:


lrwxrwxrwx 1 root root 14 Jan 17 14:23 jawdir -> /home/jim/jaw/
see the 'l' at the start of the permissions field "lrwxrwxrwx", the 'l' is for link; and the '->' where it is linked to.
as oppposed to:


drwxr-xr-x 22 root root 4096 Dec 13 17:07 lib

see:


man symlink


So yes, the system still recognizes the '/lib' on "/root". Now as to what changes (removed/added files) are. I hope none; but only you know at this time.

When you are ready for my help to proceed removing kernels, provide the outputs requested.



learning too is a process

[jaspiel6654]

Bashing-om:
Okay okay. I am on the broken system now, and saved outputs from df -i, df -h, and du -sx in case we needed any of this for comparison. Then I did a modest deconstruction of your elegant command to remove all of the images/headers that were not necessary and I did it by hand to remove the headers and images for 45, 48, and 49. That reduced the IUse% from 100% to 87%. I then decided that would be enough for a test and I brought up Update Manager and ran it in Repair mode, which was successful and the IUse% went up slightly to 89%. This is so easy. Why didn't you recommend it in the first place? Oh yeah, you did. Well what can I say.

I am running on -57 but -58 has been loaded. I think I will keep at least -56 around just in case.

I still don't quite understand why you don't think my move of /lib to a new partition on /dev/sdb8 did not work. It looks like it did in the mount command -- it looks exactily like /usr which I set up on its own partition when I built the system. You did correctly pickup on my symlink jawdir which I had done as a test but did not actually use it for anything. I think /lib points to the new partition. How to test this?

Thanks for your patience and help on all of this. You did mention some additional cleanup actions beyond just the image and header files? And I suppose I will need to keep an eye on this, at least until I move to a later version of Ubuntu, maybe 14.4 when it comes out.
Jim

[Bashing-om]

jaspiel6654; Hey ;

Making good progress. You now have some operating head room to facilitate moving things around.

Let's make sure things are cleaned up:


sudo apt-get autoclean # only removes files that cannot be downloaded anymore (obsolete)
sudo apt-get autoremove
sudo apt-get clean
dpkg -l | awk '/^rc/{print $2}' | sudo xargs dpkg -P # Removes the Configs (rc) files from obsolete packages.
sudo apt-get update
sudo apt-get upgrade

What returns now for inode usage ? - crucial ! -


df -i


Mounting; Uhh, just because something is mounted, does not mean it is used, just that it is available to be used.

As you are rightly concerned that moving the /lib directory could have adverse effects, the thing to do is copy that entire directory off to an external medium. Then one will be able to reverse any damage done.

At this time IF you are going to move the /lib directory. I would undo all I have done to this point, and begin from the start in the tutorial, and follow all the way through ( ya got the back up, what ever may happen bad, can then be fixed)

Keep this in mind:
The great thing about ubuntu, if you keep good backups, is that one can (RE-)install from scratch and be back up and running in less than an hour ! I have in my time resorted to (RE-)INstalling on at least 3 occasions. Happens to the best of us.

Learning this operating system is a life long affair. I have seen it said more than once that it takes 10 years to become comfortable at large with this operating system. Not one of us were born knowing what we know. Time and effort !



it is




all in the process

[jaspiel6654]

looks like I have my system back. Shall I try a reboot?

jim@jim-desktop:/$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 655360 310724 344636 48% /
udev 196185 612 195573 1% /dev
tmpfs 199838 555 199283 1% /run
none 199838 3 199835 1% /run/lock
none 199838 7 199831 1% /run/shm
/dev/sda5 1310720 23243 1287477 2% /home

[Bashing-om]

jaspiel6654; Yes ! Looks good.

Go ahead and reboot and let's see what happens.



all systems go

[jaspiel6654]

reboot went fine.
jim@jim-desktop:~$ uname -r
3.2.0-58-generic-pae
Now using the lastest version.
Checked the Update Manager and the number of updates has dropped from 56 to 4.

Everything looks good.
Thanks for your help. Did you suffer from the recent storm that hit the South and froze Atlanta?
Anything more to be done or are we done at this point?
Jim

[Bashing-om]

Jim; Yeah !

This thread should now be closed, only you may do so; 1st post -> thread tools, An then only if you are satisfied that there is resolution to the original issue.

Other issues, well, open as many threads as needed to address them !
We are here to help.

Off Topic: storms that laid the south low;

Nope, I am a bit to the north of the worst of it. Just the cold here. (Throw another log on the fire !)

We are far enough north that we have learned to deal with a little ice and snow. Our transportation departments are prepared to cope with it is the big factor.


nothing to it;



if you know