Dear all,
Last week the mstat deamon sent me an email message that my RAID5 array had failed:
I have been trying to recover the array but I got lost in mdadm giving me inconsistent information
the mdstat output was:
Code:
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10]
md0 : active raid5 sdb1[3] sdc1[0] sde1[2] sdd1[4](F)
5860540224 blocks level 5, 64k chunk, algorithm 2 [4/3] [U_UU]
Getting the info from mdadm:
Code:
root@Perseus:~# mdadm -D /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Fri Aug 27 22:06:39 2010
Raid Level : raid5
Array Size : 5860540224 (5589.05 GiB 6001.19 GB)
Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Sun Sep 2 12:01:40 2012
State : clean, degraded
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : e8fdb9c9:a21ad335:145f0e20:b7304898
Events : 0.23996
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 0 0 1 removed
2 8 65 2 active sync /dev/sde1
3 8 17 3 active sync /dev/sdb1
So here it still seems that sdd1 is lost. So, I examine the four drives in my array:
SDB1
Code:
root@Perseus:~# mdadm -E /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 00.90.00
UUID : e8fdb9c9:a21ad335:145f0e20:b7304898
Creation Time : Fri Aug 27 22:06:39 2010
Raid Level : raid5
Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
Array Size : 5860540224 (5589.05 GiB 6001.19 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sun Sep 2 12:19:53 2012
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 11007cd1 - correct
Events : 23998
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 17 3 active sync /dev/sdb1
0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 17 3 active sync /dev/sdb1
SDC1
Code:
root@Perseus:~# mdadm -E /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.00
UUID : e8fdb9c9:a21ad335:145f0e20:b7304898
Creation Time : Fri Aug 27 22:06:39 2010
Raid Level : raid5
Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
Array Size : 5860540224 (5589.05 GiB 6001.19 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sun Sep 2 12:19:53 2012
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 11007cdb - correct
Events : 23998
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 33 0 active sync /dev/sdc1
0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 17 3 active sync /dev/sdb1
SDD1
Code:
root@Perseus:~# mdadm -E /dev/sdd1
/dev/sdd1:
Magic : a92b4efc
Version : 00.90.00
UUID : e8fdb9c9:a21ad335:145f0e20:b7304898
Creation Time : Fri Aug 27 22:06:39 2010
Raid Level : raid5
Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
Array Size : 5860540224 (5589.05 GiB 6001.19 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Fri Aug 31 11:51:04 2012
State : active
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Checksum : 10fd7354 - expected 10fe138b
Events : 23405
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 49 1 active sync /dev/sdd1
0 0 8 33 0 active sync /dev/sdc1
1 1 8 49 1 active sync /dev/sdd1
2 2 8 65 2 active sync /dev/sde1
3 3 8 17 3 active sync /dev/sdb1
SDE1
Code:
root@Perseus:~# mdadm -E /dev/sde1
/dev/sde1:
Magic : a92b4efc
Version : 00.90.00
UUID : e8fdb9c9:a21ad335:145f0e20:b7304898
Creation Time : Fri Aug 27 22:06:39 2010
Raid Level : raid5
Used Dev Size : 1953513408 (1863.02 GiB 2000.40 GB)
Array Size : 5860540224 (5589.05 GiB 6001.19 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Update Time : Sun Sep 2 12:19:53 2012
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Checksum : 11007cff - correct
Events : 23998
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 65 2 active sync /dev/sde1
0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 8 65 2 active sync /dev/sde1
3 3 8 17 3 active sync /dev/sdb1
So, for some reason mdstat thinks SDD1 is bad, but checking the device itself it is pronounced healthy except for the fact that the checksum is wrong.
I have tried to re-assemble the array but here I find myself lost:
Code:
root@Perseus:~# umount /dev/md0
root@Perseus:~# mdadm --assemble --scan --verbose
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/block/9:0
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: /dev/sdd1 is not built for host Perseus.
mdadm: no recogniseable superblock on /dev/sdd
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy
So what to do next? Any help is appreciated.
Bookmarks