8.04 raid 5 will not start after reboot

[SpaceBas]

Hey folks,
I've got a raid 5 array that I'm having some issues with. After a recent reboot the array failed to come up properly. Mdadm is being its typically quiet self, but it seems to think 2 of the disks have failed. Spinrite reports them as very clean and I'm frankly inclined to agree ... I dont think I'd have two drives fail because of a reboot (of course, anything can happen)...

So to that end, I'd love some help troublshooting this thing...anyone have any ideas?

administrator@telluride:~$ sudo mdadm -A /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.
administrator@telluride:~$ sudo mdadm -v -A /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: added /dev/sdb1 to /dev/md0 as 0
mdadm: added /dev/sdc1 to /dev/md0 as 1
mdadm: added /dev/sde1 to /dev/md0 as 3
mdadm: added /dev/sdd1 to /dev/md0 as 2
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.

administrator@telluride:~$ sudo mdadm --examine /dev/sd[b-e]1
/dev/sdb1:
Magic : a92b4efc
Version : 00.90.00
UUID : 3077899a:55e36b23:81cca090:cc945d8e
Creation Time : Fri Jun 6 09:31:08 2008
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 1465151808 (1397.28 GiB 1500.32 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0

Update Time : Sun Jul 13 19:36:54 2008
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 2bc836f1 - correct
Events : 0.22

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 0 8 17 0 active sync /dev/sdb1

0 0 8 17 0 active sync /dev/sdb1
1 1 8 33 1 active sync /dev/sdc1
2 2 8 49 2 active sync /dev/sdd1
3 3 8 65 3 active sync /dev/sde1
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.00
UUID : 3077899a:55e36b23:81cca090:cc945d8e
Creation Time : Fri Jun 6 09:31:08 2008
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 1465151808 (1397.28 GiB 1500.32 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0

Update Time : Sun Jul 13 19:36:54 2008
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 2bc83703 - correct
Events : 0.22

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 1 8 33 1 active sync /dev/sdc1

0 0 8 17 0 active sync /dev/sdb1
1 1 8 33 1 active sync /dev/sdc1
2 2 8 49 2 active sync /dev/sdd1
3 3 8 65 3 active sync /dev/sde1
/dev/sdd1:
Magic : a92b4efc
Version : 00.90.00
UUID : 3077899a:55e36b23:81cca090:cc945d8e
Creation Time : Fri Jun 6 09:31:08 2008
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 1465151808 (1397.28 GiB 1500.32 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0

Update Time : Sun Jul 13 20:15:19 2008
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0
Checksum : 2bc84038 - correct
Events : 0.28

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 2 8 49 2 active sync /dev/sdd1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 49 2 active sync /dev/sdd1
3 3 8 65 3 active sync /dev/sde1
/dev/sde1:
Magic : a92b4efc
Version : 00.90.00
UUID : 3077899a:55e36b23:81cca090:cc945d8e
Creation Time : Fri Jun 6 09:31:08 2008
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 1465151808 (1397.28 GiB 1500.32 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0

Update Time : Sun Jul 13 20:15:19 2008
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0
Checksum : 2bc8404a - correct
Events : 0.28

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDevice State
this 3 8 65 3 active sync /dev/sde1

0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 49 2 active sync /dev/sdd1
3 3 8 65 3 active sync /dev/sde1
administrator@telluride:~$

[fjgaude]

You might try a -f to force the assemble:

sudo --assemble -f --scan

and see what happens.

You might check to see if the UUID in the mdadm.conf file is the same as the UUID you get for the individual drives.

Let us know how you are doing.

[SpaceBas]

You might try a -f to force the assemble:

sudo --assemble -f --scan

and see what happens.

You might check to see if the UUID in the mdadm.conf file is the same as the UUID you get for the individual drives.

Let us know how you are doing.

you sir are my hero - that did it!
What do I need to do to insure the integrity of the array and prevent this from happening again?

Great learning experience - thanks again!

[fjgaude]

I can't say... I really have little or no issues with software raid5, moving from one machine, upgrades, booting, so forth.

I can't say what caused you to have this issue either.

Good luck and be careful of the commands you use, knowing that some can cause really bad things to happen.