PDA

View Full Version : [SOLVED] Server won't boot (help!)



mattlach
September 15th, 2010, 03:31 AM
Hey all,

I would really appreciate your help!

I needed to re-route the power cables to my server today, so I shut it down via SSH. The shut down message appeared and I lost the connection (as expected)

The server - however - never completely shut down, so I forced it by holding the power switch.

Just my luck, afterwards when everything is moved, the server is no longer reachable via SSH.

So I grab a monitor and hook it up to try to diagnose. Turns out its not booting due to the partition being corrupt, and it needs fsck run manually.

I boot up from the rescue disk and manually run fsck and fix the partition. It is now reported as "clean".

The server still refuses to boot.

When I attempt a boot, all I see on the screen is as follows:



fsck from util-linux-ng 2.17.2
/dev/sda1: clean, 62143/935424 files, 337874/1870592 blocks.


and then nothing.

I can still mount the partition from the live disk and it works fine...

Figuring something may have messed up my booting I ran the handy bootscript thats floating around.

Results are as follows:




Boot Info Script 0.55 dated February 15th, 2010

============================= Boot Info Summary: ==============================

=> Grub 2 is installed in the MBR of /dev/sda and looks on the same drive in
partition #1 for /boot/grub.

sda1: __________________________________________________ _______________________

File system: ext2
Boot sector type: Unknown
Boot sector info:
Operating System: Ubuntu 10.04.1 LTS
Boot files/dirs: /boot/grub/grub.cfg /etc/fstab /boot/grub/core.img

sda2: __________________________________________________ _______________________

File system: Extended Partition
Boot sector type: Unknown
Boot sector info:

sda5: __________________________________________________ _______________________

File system: swap
Boot sector type: -
Boot sector info:

=========================== Drive/Partition Info: =============================

Drive: sda ___________________ __________________________________________________ ___

Disk /dev/sda: 8061 MB, 8061419520 bytes
255 heads, 63 sectors/track, 980 cylinders, total 15744960 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

Partition Boot Start End Size Id System

/dev/sda1 * 2,048 14,966,783 14,964,736 83 Linux
/dev/sda2 14,968,830 15,742,975 774,146 5 Extended
/dev/sda5 14,968,832 15,742,975 774,144 82 Linux swap / Solaris


blkid -c /dev/null: __________________________________________________ __________

Device UUID TYPE LABEL

/dev/loop0 squashfs
/dev/sda1 b3ee9792-99a3-4cf8-9879-73f2ee066382 ext2
/dev/sda2: PTTYPE="dos"
/dev/sda5 ada56e21-a5a1-40d3-adee-5ecb040de413 swap
/dev/sda: PTTYPE="dos"
error: /dev/sdb: No medium found

============================ "mount | grep ^/dev output: ===========================

Device Mount_Point Type Options

aufs / aufs (rw)
/dev/sr0 /cdrom iso9660 (ro,noatime)
/dev/loop0 /rofs squashfs (ro,noatime)


=========================== sda1/boot/grub/grub.cfg: ===========================

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by /usr/sbin/grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
load_env
fi
set default="0"
if [ ${prev_saved_entry} ]; then
set saved_entry=${prev_saved_entry}
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi

function savedefault {
if [ -z ${boot_once} ]; then
saved_entry=${chosen}
save_env saved_entry
fi
}

function recordfail {
set recordfail=1
if [ -n ${have_grubenv} ]; then if [ -z ${boot_once} ]; then save_env recordfail; fi; fi
}
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
if loadfont /usr/share/grub/unicode.pf2 ; then
set gfxmode=640x480
insmod gfxterm
insmod vbe
if terminal_output gfxterm ; then true ; else
# For backward compatibility with versions of terminal.mod that don't
# understand terminal_output
terminal gfxterm
fi
fi
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
set locale_dir=($root)/boot/grub/locale
set lang=en
insmod gettext
if [ ${recordfail} = 1 ]; then
set timeout=-1
else
set timeout=10
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Ubuntu, with Linux 2.6.32-24-server' --class ubuntu --class gnu-linux --class gnu --class os {
recordfail
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
linux /boot/vmlinuz-2.6.32-24-server root=UUID=b3ee9792-99a3-4cf8-9879-73f2ee066382 ro quiet
initrd /boot/initrd.img-2.6.32-24-server
}
menuentry 'Ubuntu, with Linux 2.6.32-24-server (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os {
recordfail
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
echo 'Loading Linux 2.6.32-24-server ...'
linux /boot/vmlinuz-2.6.32-24-server root=UUID=b3ee9792-99a3-4cf8-9879-73f2ee066382 ro single
echo 'Loading initial ramdisk ...'
initrd /boot/initrd.img-2.6.32-24-server
}
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry "Memory test (memtest86+)" {
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
linux16 /boot/memtest86+.bin
}
menuentry "Memory test (memtest86+, serial console 115200)" {
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
linux16 /boot/memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
if [ ${timeout} != -1 ]; then
if keystatus; then
if keystatus --shift; then
set timeout=-1
else
set timeout=0
fi
else
if sleep --interruptible 3 ; then
set timeout=0
fi
fi
fi
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

=============================== sda1/etc/fstab: ===============================

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda1 during installation
UUID=b3ee9792-99a3-4cf8-9879-73f2ee066382 / ext2 errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=ada56e21-a5a1-40d3-adee-5ecb040de413 none swap sw 0 0
# drobo was on /dev/sdb1 when added via USB (eSATA not working)
UUID=db7d26da-73c5-4258-979d-29e91388d752 /home ext3 auto,users,rw,relatime 0 0

=================== sda1: Location of files loaded by Grub: ===================


6.6GB: boot/grub/core.img
6.6GB: boot/grub/grub.cfg
6.6GB: boot/initrd.img-2.6.32-24-server
6.6GB: boot/vmlinuz-2.6.32-24-server
6.6GB: initrd.img
6.6GB: vmlinuz
=========================== Unknown MBRs/Boot Sectors/etc =======================

Unknown BootLoader on sda1

00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
00000200

Unknown BootLoader on sda2

00000000 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
000001b0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff 00 c3 |................|
000001c0 e1 a3 82 f3 e0 d3 02 00 00 00 00 d0 0b 00 00 00 |................|
000001d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200


=======Devices which don't seem to have a corresponding hard drive==============

sdb

In other words, nothing that really stands out. (sdb is my external storage array that is disconnected during the troubleshooting)

I would really appreciate any help! I really don't want to have to set up the server the way I like it all over again :(

CharlesA
September 15th, 2010, 03:40 AM
Try hitting "s" to see if it'll say anything. Otherwise, Enable the splash screen since that'll show you any errors.

Also try to boot into recovery mode, or turn off "quiet" before booting into normal mode, so you can see any error messages.

What version of Ubuntu are you running?

mattlach
September 15th, 2010, 04:14 AM
Try hitting "s" to see if it'll say anything. Otherwise, Enable the splash screen since that'll show you any errors.

Also try to boot into recovery mode, or turn off "quiet" before booting into normal mode, so you can see any error messages.

What version of Ubuntu are you running?

Interesting, I have a new theory.

I think it was running a runaway and s skipped it.

Either way it booteb back up again.

Thanks for the help.

(I'm running Ubuntu Server Edition 10.4 x64, by the way)

mattlach
September 15th, 2010, 05:04 AM
...and man do I feel stupid...


(Though I can't for the life of me figure out why its taking so long. An hour later and it's only progressed by 5 files and 100 blocks... This stuff ought to be very fast, as its booting from an SSD!)

CharlesA
September 15th, 2010, 05:34 AM
What file system is on the SSD?

mattlach
September 15th, 2010, 05:55 PM
What file system is on the SSD?

File system is ext2. It's a low end small 8GB supertalent SSD, so I didn't want a journal adding unnecessary write cycles.

Fsck has run all night at this point with no change to the files or blocks. I'm suspecting fsck is somehow ceasing to function properly.

I can skip it and make the server boot up, but I don't want to do that every time...

Any suggestions?

CharlesA
September 15th, 2010, 07:01 PM
Kinda figured it would be ext2, since it has no journaling.

Try booting off a livecd and run fsck from there.

mattlach
September 15th, 2010, 10:35 PM
Kinda figured it would be ext2, since it has no journaling.

Try booting off a livecd and run fsck from there.


I ran e2fsck from a live CD which repaired the file system.

Now when I boot it up from a live cd the file system checks out as clean, but when I boot up from the SSD again it goes into the eternal fsck at boot...

CharlesA
September 15th, 2010, 11:51 PM
I ran e2fsck from a live CD which repaired the file system.

Now when I boot it up from a live cd the file system checks out as clean, but when I boot up from the SSD again it goes into the eternal fsck at boot...
Ugh. It seems like it still thinks the SSD has a corrupt file system, but I don't know how to tell it that it's clean.

Does running fsck when it boots do anything?

How are your partitions set up? Do you have a seperate /home partition/drive?

mattlach
September 16th, 2010, 01:04 AM
Ugh. It seems like it still thinks the SSD has a corrupt file system, but I don't know how to tell it that it's clean.

Does running fsck when it boots do anything?

I tried real quick when I was at home, but I have to remount my root read only, and I didn't have time ti figure it out. I'll do that when I get home and try again



How are your partitions set up? Do you have a seperate /home partition/drive?

/boot and root are on the same partition

My /home is on my external storage array in my closet (though it is temporarily disconnected)

Thank you very much for help!

CharlesA
September 16th, 2010, 02:22 AM
Ok, that's good. At least if you do need to reinstall, you won't lose yer data. :)

But yeah, give fsck a shot and see what happens.

mattlach
September 16th, 2010, 03:42 AM
Ok, that's good. At least if you do need to reinstall, you won't lose yer data. :)

But yeah, give fsck a shot and see what happens.

I figured it out, and it's silly...

The root partition was genuinely damaged during the shutdown, and I fixed it with e2fsck on the live CD.

What was causing it to freeze during fsck on boot was the fact that it could not find my external storage array (as it was not plugged in) and fsck was trying to check it.

Can't check something that isn't there, so you freeze :p

CharlesA
September 16th, 2010, 03:44 AM
D'oh! Glad it's working now.

What a pain.

mattlach
September 16th, 2010, 04:25 AM
D'oh! Glad it's working now.

What a pain.

Thank you :)