Server won't boot (help!)

[mattlach]

Hey all,

I would really appreciate your help!

I needed to re-route the power cables to my server today, so I shut it down via SSH. The shut down message appeared and I lost the connection (as expected)

The server - however - never completely shut down, so I forced it by holding the power switch.

Just my luck, afterwards when everything is moved, the server is no longer reachable via SSH.

So I grab a monitor and hook it up to try to diagnose. Turns out its not booting due to the partition being corrupt, and it needs fsck run manually.

I boot up from the rescue disk and manually run fsck and fix the partition. It is now reported as "clean".

The server still refuses to boot.

When I attempt a boot, all I see on the screen is as follows:

fsck from util-linux-ng 2.17.2
/dev/sda1: clean, 62143/935424 files, 337874/1870592 blocks.

and then nothing.

I can still mount the partition from the live disk and it works fine...

Figuring something may have messed up my booting I ran the handy bootscript thats floating around.

Results are as follows:

                                       
                Boot Info Script 0.55    dated February 15th, 2010                    

============================= Boot Info Summary: ==============================

 => Grub 2 is installed in the MBR of /dev/sda and looks on the same drive in 
    partition #1 for /boot/grub.

sda1: _________________________________________________________________________

    File system:       ext2
    Boot sector type:  Unknown
    Boot sector info:  
    Operating System:  Ubuntu 10.04.1 LTS
    Boot files/dirs:   /boot/grub/grub.cfg /etc/fstab /boot/grub/core.img

sda2: _________________________________________________________________________

    File system:       Extended Partition
    Boot sector type:  Unknown
    Boot sector info:  

sda5: _________________________________________________________________________

    File system:       swap
    Boot sector type:  -
    Boot sector info:  

=========================== Drive/Partition Info: =============================

Drive: sda ___________________ _____________________________________________________

Disk /dev/sda: 8061 MB, 8061419520 bytes
255 heads, 63 sectors/track, 980 cylinders, total 15744960 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

Partition  Boot         Start           End          Size  Id System

/dev/sda1    *          2,048    14,966,783    14,964,736  83 Linux
/dev/sda2          14,968,830    15,742,975       774,146   5 Extended
/dev/sda5          14,968,832    15,742,975       774,144  82 Linux swap / Solaris


blkid -c /dev/null: ____________________________________________________________

Device           UUID                                   TYPE       LABEL                         

/dev/loop0                                              squashfs                                 
/dev/sda1        b3ee9792-99a3-4cf8-9879-73f2ee066382   ext2                                     
/dev/sda2: PTTYPE="dos" 
/dev/sda5        ada56e21-a5a1-40d3-adee-5ecb040de413   swap                                     
/dev/sda: PTTYPE="dos" 
error: /dev/sdb: No medium found

============================ "mount | grep ^/dev  output: ===========================

Device           Mount_Point              Type       Options

aufs             /                        aufs       (rw)
/dev/sr0         /cdrom                   iso9660    (ro,noatime)
/dev/loop0       /rofs                    squashfs   (ro,noatime)


=========================== sda1/boot/grub/grub.cfg: ===========================

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by /usr/sbin/grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  load_env
fi
set default="0"
if [ ${prev_saved_entry} ]; then
  set saved_entry=${prev_saved_entry}
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z ${boot_once} ]; then
    saved_entry=${chosen}
    save_env saved_entry
  fi
}

function recordfail {
  set recordfail=1
  if [ -n ${have_grubenv} ]; then if [ -z ${boot_once} ]; then save_env recordfail; fi; fi
}
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
if loadfont /usr/share/grub/unicode.pf2 ; then
  set gfxmode=640x480
  insmod gfxterm
  insmod vbe
  if terminal_output gfxterm ; then true ; else
    # For backward compatibility with versions of terminal.mod that don't
    # understand terminal_output
    terminal gfxterm
  fi
fi
insmod ext2
set root='(hd0,1)'
search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
set locale_dir=($root)/boot/grub/locale
set lang=en
insmod gettext
if [ ${recordfail} = 1 ]; then
  set timeout=-1
else
  set timeout=10
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Ubuntu, with Linux 2.6.32-24-server' --class ubuntu --class gnu-linux --class gnu --class os {
	recordfail
	insmod ext2
	set root='(hd0,1)'
	search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
	linux	/boot/vmlinuz-2.6.32-24-server root=UUID=b3ee9792-99a3-4cf8-9879-73f2ee066382 ro   quiet
	initrd	/boot/initrd.img-2.6.32-24-server
}
menuentry 'Ubuntu, with Linux 2.6.32-24-server (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os {
	recordfail
	insmod ext2
	set root='(hd0,1)'
	search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
	echo	'Loading Linux 2.6.32-24-server ...'
	linux	/boot/vmlinuz-2.6.32-24-server root=UUID=b3ee9792-99a3-4cf8-9879-73f2ee066382 ro single 
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-2.6.32-24-server
}
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry "Memory test (memtest86+)" {
	insmod ext2
	set root='(hd0,1)'
	search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
	linux16	/boot/memtest86+.bin
}
menuentry "Memory test (memtest86+, serial console 115200)" {
	insmod ext2
	set root='(hd0,1)'
	search --no-floppy --fs-uuid --set b3ee9792-99a3-4cf8-9879-73f2ee066382
	linux16	/boot/memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
if [ ${timeout} != -1 ]; then
  if keystatus; then
    if keystatus --shift; then
      set timeout=-1
    else
      set timeout=0
    fi
  else
    if sleep --interruptible 3 ; then
      set timeout=0
    fi
  fi
fi
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

=============================== sda1/etc/fstab: ===============================

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=b3ee9792-99a3-4cf8-9879-73f2ee066382 /               ext2    errors=remount-ro 0       1
# swap was on /dev/sda5 during installation
UUID=ada56e21-a5a1-40d3-adee-5ecb040de413 none            swap    sw              0       0
# drobo was on /dev/sdb1 when added via USB (eSATA not working)
UUID=db7d26da-73c5-4258-979d-29e91388d752	/home	ext3    auto,users,rw,relatime 0 0

=================== sda1: Location of files loaded by Grub: ===================


   6.6GB: boot/grub/core.img
   6.6GB: boot/grub/grub.cfg
   6.6GB: boot/initrd.img-2.6.32-24-server
   6.6GB: boot/vmlinuz-2.6.32-24-server
   6.6GB: initrd.img
   6.6GB: vmlinuz
=========================== Unknown MBRs/Boot Sectors/etc =======================

Unknown BootLoader  on sda1

00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00000200

Unknown BootLoader  on sda2

00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
000001b0  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff 00 c3  |................|
000001c0  e1 a3 82 f3 e0 d3 02 00  00 00 00 d0 0b 00 00 00  |................|
000001d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200


=======Devices which don't seem to have a corresponding hard drive==============

sdb

In other words, nothing that really stands out. (sdb is my external storage array that is disconnected during the troubleshooting)

I would really appreciate any help! I really don't want to have to set up the server the way I like it all over again

[CharlesA]

Try hitting "s" to see if it'll say anything. Otherwise, Enable the splash screen since that'll show you any errors.

Also try to boot into recovery mode, or turn off "quiet" before booting into normal mode, so you can see any error messages.

What version of Ubuntu are you running?

[mattlach]

Try hitting "s" to see if it'll say anything. Otherwise, Enable the splash screen since that'll show you any errors.

Also try to boot into recovery mode, or turn off "quiet" before booting into normal mode, so you can see any error messages.

What version of Ubuntu are you running?

Interesting, I have a new theory.

I think it was running a runaway and s skipped it.

Either way it booteb back up again.

Thanks for the help.

(I'm running Ubuntu Server Edition 10.4 x64, by the way)

[mattlach]

...and man do I feel stupid...


(Though I can't for the life of me figure out why its taking so long. An hour later and it's only progressed by 5 files and 100 blocks... This stuff ought to be very fast, as its booting from an SSD!)

[CharlesA]

What file system is on the SSD?

[mattlach]

What file system is on the SSD?

File system is ext2. It's a low end small 8GB supertalent SSD, so I didn't want a journal adding unnecessary write cycles.

Fsck has run all night at this point with no change to the files or blocks. I'm suspecting fsck is somehow ceasing to function properly.

I can skip it and make the server boot up, but I don't want to do that every time...

Any suggestions?

[CharlesA]

Kinda figured it would be ext2, since it has no journaling.

Try booting off a livecd and run fsck from there.

[mattlach]

Kinda figured it would be ext2, since it has no journaling.

Try booting off a livecd and run fsck from there.

I ran e2fsck from a live CD which repaired the file system.

Now when I boot it up from a live cd the file system checks out as clean, but when I boot up from the SSD again it goes into the eternal fsck at boot...

[CharlesA]

I ran e2fsck from a live CD which repaired the file system.

Now when I boot it up from a live cd the file system checks out as clean, but when I boot up from the SSD again it goes into the eternal fsck at boot...

Ugh. It seems like it still thinks the SSD has a corrupt file system, but I don't know how to tell it that it's clean.

Does running fsck when it boots do anything?

How are your partitions set up? Do you have a seperate /home partition/drive?

[mattlach]

Ugh. It seems like it still thinks the SSD has a corrupt file system, but I don't know how to tell it that it's clean.

Does running fsck when it boots do anything?

I tried real quick when I was at home, but I have to remount my root read only, and I didn't have time ti figure it out. I'll do that when I get home and try again

How are your partitions set up? Do you have a seperate /home partition/drive?

/boot and root are on the same partition

My /home is on my external storage array in my closet (though it is temporarily disconnected)

Thank you very much for help!