thecabinet
November 15th, 2010, 03:14 AM
For the last few days I've been trying to setup a new server running 64-bit desktop Ubuntu 10.10. Every time the computer is rebooted, the hard disk paths (/dev/sda, /dev/sdb, etc.) change to one of two assignments. I've twiddled various BIOS settings, none of which make any difference that I can tell. It seems to be random which of the two assignments is chosen, but there are consistently only two.
The box has 7 hard drives. There are 4x nearly identical 400GB drives (2x PATA and 2x SATA), and 3x identical 2TB drives. The motherboard is an ASUS M4A785-M. All the drives are using the motherboard's onboard SATA ports.
One set of assignments looks like:
root@server:~# fdisk -cul | grep -i -e disk -e dev -e '^$'
Disk /dev/sda: 400.1 GB, 400088457216 bytes
Disk identifier: 0x000b556d
Device Boot Start End Blocks Id System
Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0x56870040
Device Boot Start End Blocks Id System
/dev/sdc1 2048 3907029167 1953513560 83 Linux
Disk /dev/sdb: 400.1 GB, 400088457216 bytes
Disk identifier: 0x000769cf
Device Boot Start End Blocks Id System
Disk /dev/sdd: 400.1 GB, 400088457216 bytes
Disk identifier: 0x00038e47
Device Boot Start End Blocks Id System
/dev/sdd1 * 2048 999423 498688 83 Linux
/dev/sdd2 999424 765421567 382211072 83 Linux
/dev/sdd3 765421568 781422591 8000512 82 Linux swap / Solaris
Disk /dev/sde: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0x1f3a7538
Device Boot Start End Blocks Id System
/dev/sde1 2048 3907029167 1953513560 83 Linux
Disk /dev/sdf: 400.1 GB, 400088457216 bytes
Disk identifier: 0x0001d258
Device Boot Start End Blocks Id System
Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0xdb9a2603
Device Boot Start End Blocks Id System
/dev/sdg1 2048 3907029167 1953513560 83 Linux
And the other looks like:
root@server:~# fdisk -cul | grep -i -e disk -e dev -e '^$'
Disk /dev/sda: 400.1 GB, 400088457216 bytes
Disk identifier: 0x00038e47
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 999423 498688 83 Linux
/dev/sda2 999424 765421567 382211072 83 Linux
/dev/sda3 765421568 781422591 8000512 82 Linux swap / Solaris
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0x1f3a7538
Device Boot Start End Blocks Id System
/dev/sdb1 2048 3907029167 1953513560 83 Linux
Disk /dev/sdc: 400.1 GB, 400088457216 bytes
Disk identifier: 0x0001d258
Device Boot Start End Blocks Id System
Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0xdb9a2603
Device Boot Start End Blocks Id System
/dev/sdd1 2048 3907029167 1953513560 83 Linux
Disk /dev/sde: 400.1 GB, 400088457216 bytes
Disk identifier: 0x000b556d
Device Boot Start End Blocks Id System
Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0x56870040
Device Boot Start End Blocks Id System
/dev/sdg1 2048 3907029167 1953513560 83 Linux
Disk /dev/sdf: 400.1 GB, 400088457216 bytes
Disk identifier: 0x000769cf
Device Boot Start End Blocks Id System
I noticed that the devices aren't listed in order (sdf comes after sdg) but I'm not sure if that's unusual or not.
Here's the output from blkid, if that's relevant (in the first configuration).
/dev/sda: UUID="5757bc13-01cf-c15b-f213-6d9a31f9d279" TYPE="linux_raid_member"
/dev/sdc1: UUID="8afe42ae-2bde-47a3-97c1-f45e3e4e348a" UUID_SUB="c0fe6ef0-3e0e-4b82-989f-c4ea0fe4b74f" TYPE="btrfs"
/dev/sdb: TYPE="zfs"
/dev/sdd1: UUID="47ed0671-ea5c-48d2-b0c5-b0d3ea601c41" TYPE="ext2"
/dev/sdd2: UUID="384cb7b8-5c59-4d39-9231-32b3a4d8fdb6" TYPE="ext4"
/dev/sdd3: UUID="df60fb6c-8d47-4f1c-968b-1de8260832fd" TYPE="swap"
/dev/sde1: UUID="8afe42ae-2bde-47a3-97c1-f45e3e4e348a" UUID_SUB="f5f2ea00-66a3-44e9-ac02-675d932c8f17" TYPE="btrfs"
/dev/sdf: TYPE="zfs"
/dev/sdg1: UUID="8afe42ae-2bde-47a3-97c1-f45e3e4e348a" UUID_SUB="38f72d0c-e7b2-49c3-8763-43a7c98755ad" TYPE="btrfs"
I was finally able to get the box to boot reliably once I specified /boot using the UUID instead of the device:
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda2 during installation
UUID=384cb7b8-5c59-4d39-9231-32b3a4d8fdb6 / ext4 errors=remount-ro 0 1
# As created by Ubuntu 10.10 installer:
#/dev/sda1 /boot ext2 defaults 0 2
UUID=47ed0671-ea5c-48d2-b0c5-b0d3ea601c41 /boot ext2 defaults 0 2
# swap was on /dev/sda3 during installation
UUID=df60fb6c-8d47-4f1c-968b-1de8260832fd none swap sw 0 0
I don't know if it's related or not, but the 2TB drives are part of a btrfs pool (or whatever it's called). I always have to mount that manually because (a) the mount command only succeeds with one of the three drives (not sure which one) and (b) trying to mount by UUID never chooses the right one.
FWIW, I think this is a Linux problem (rather than a BIOS problem) because the box does reliably try to boot (off the disk specified in the BIOS), and it's during the Linux boot process it will fail because /boot is /dev/sdd1 instead of /dev/sda1.
Update: I did some experimenting with the 32-bit desktop ISO, and it exhibits the same problem of two sets of hard-disk device paths. So it seems to not be an issue with the 64-bit build, although that doesn't really help explain the craziness.
The box has 7 hard drives. There are 4x nearly identical 400GB drives (2x PATA and 2x SATA), and 3x identical 2TB drives. The motherboard is an ASUS M4A785-M. All the drives are using the motherboard's onboard SATA ports.
One set of assignments looks like:
root@server:~# fdisk -cul | grep -i -e disk -e dev -e '^$'
Disk /dev/sda: 400.1 GB, 400088457216 bytes
Disk identifier: 0x000b556d
Device Boot Start End Blocks Id System
Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0x56870040
Device Boot Start End Blocks Id System
/dev/sdc1 2048 3907029167 1953513560 83 Linux
Disk /dev/sdb: 400.1 GB, 400088457216 bytes
Disk identifier: 0x000769cf
Device Boot Start End Blocks Id System
Disk /dev/sdd: 400.1 GB, 400088457216 bytes
Disk identifier: 0x00038e47
Device Boot Start End Blocks Id System
/dev/sdd1 * 2048 999423 498688 83 Linux
/dev/sdd2 999424 765421567 382211072 83 Linux
/dev/sdd3 765421568 781422591 8000512 82 Linux swap / Solaris
Disk /dev/sde: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0x1f3a7538
Device Boot Start End Blocks Id System
/dev/sde1 2048 3907029167 1953513560 83 Linux
Disk /dev/sdf: 400.1 GB, 400088457216 bytes
Disk identifier: 0x0001d258
Device Boot Start End Blocks Id System
Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0xdb9a2603
Device Boot Start End Blocks Id System
/dev/sdg1 2048 3907029167 1953513560 83 Linux
And the other looks like:
root@server:~# fdisk -cul | grep -i -e disk -e dev -e '^$'
Disk /dev/sda: 400.1 GB, 400088457216 bytes
Disk identifier: 0x00038e47
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 999423 498688 83 Linux
/dev/sda2 999424 765421567 382211072 83 Linux
/dev/sda3 765421568 781422591 8000512 82 Linux swap / Solaris
Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0x1f3a7538
Device Boot Start End Blocks Id System
/dev/sdb1 2048 3907029167 1953513560 83 Linux
Disk /dev/sdc: 400.1 GB, 400088457216 bytes
Disk identifier: 0x0001d258
Device Boot Start End Blocks Id System
Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0xdb9a2603
Device Boot Start End Blocks Id System
/dev/sdd1 2048 3907029167 1953513560 83 Linux
Disk /dev/sde: 400.1 GB, 400088457216 bytes
Disk identifier: 0x000b556d
Device Boot Start End Blocks Id System
Disk /dev/sdg: 2000.4 GB, 2000398934016 bytes
Disk identifier: 0x56870040
Device Boot Start End Blocks Id System
/dev/sdg1 2048 3907029167 1953513560 83 Linux
Disk /dev/sdf: 400.1 GB, 400088457216 bytes
Disk identifier: 0x000769cf
Device Boot Start End Blocks Id System
I noticed that the devices aren't listed in order (sdf comes after sdg) but I'm not sure if that's unusual or not.
Here's the output from blkid, if that's relevant (in the first configuration).
/dev/sda: UUID="5757bc13-01cf-c15b-f213-6d9a31f9d279" TYPE="linux_raid_member"
/dev/sdc1: UUID="8afe42ae-2bde-47a3-97c1-f45e3e4e348a" UUID_SUB="c0fe6ef0-3e0e-4b82-989f-c4ea0fe4b74f" TYPE="btrfs"
/dev/sdb: TYPE="zfs"
/dev/sdd1: UUID="47ed0671-ea5c-48d2-b0c5-b0d3ea601c41" TYPE="ext2"
/dev/sdd2: UUID="384cb7b8-5c59-4d39-9231-32b3a4d8fdb6" TYPE="ext4"
/dev/sdd3: UUID="df60fb6c-8d47-4f1c-968b-1de8260832fd" TYPE="swap"
/dev/sde1: UUID="8afe42ae-2bde-47a3-97c1-f45e3e4e348a" UUID_SUB="f5f2ea00-66a3-44e9-ac02-675d932c8f17" TYPE="btrfs"
/dev/sdf: TYPE="zfs"
/dev/sdg1: UUID="8afe42ae-2bde-47a3-97c1-f45e3e4e348a" UUID_SUB="38f72d0c-e7b2-49c3-8763-43a7c98755ad" TYPE="btrfs"
I was finally able to get the box to boot reliably once I specified /boot using the UUID instead of the device:
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda2 during installation
UUID=384cb7b8-5c59-4d39-9231-32b3a4d8fdb6 / ext4 errors=remount-ro 0 1
# As created by Ubuntu 10.10 installer:
#/dev/sda1 /boot ext2 defaults 0 2
UUID=47ed0671-ea5c-48d2-b0c5-b0d3ea601c41 /boot ext2 defaults 0 2
# swap was on /dev/sda3 during installation
UUID=df60fb6c-8d47-4f1c-968b-1de8260832fd none swap sw 0 0
I don't know if it's related or not, but the 2TB drives are part of a btrfs pool (or whatever it's called). I always have to mount that manually because (a) the mount command only succeeds with one of the three drives (not sure which one) and (b) trying to mount by UUID never chooses the right one.
FWIW, I think this is a Linux problem (rather than a BIOS problem) because the box does reliably try to boot (off the disk specified in the BIOS), and it's during the Linux boot process it will fail because /boot is /dev/sdd1 instead of /dev/sda1.
Update: I did some experimenting with the 32-bit desktop ISO, and it exhibits the same problem of two sets of hard-disk device paths. So it seems to not be an issue with the 64-bit build, although that doesn't really help explain the craziness.