Page 1 of 2 12 LastLast
Results 1 to 10 of 16

Thread: RAID drives suddenly become spare drives

  1. #1
    Join Date
    Oct 2011
    Beans
    9

    Angry RAID drives suddenly become spare drives

    Hi,
    i have a serious raid problem here. I'm running here a RAID-5/LVM configuration consisting of 3 physical harddrives of 2TB each. Each drive has 8 partitions of about 250gigs. Only the last partition has less because 2TB/8 is less than 250gig which is compensated here.

    RAID
    The first 7 sets of partitions (sdaX,sdbX and sdcX) is configured as a RAID5 providing 500gigs per set. The 8th set is configured as a RAID1 for fault tolerance.

    LVM
    The LVM configuration combines the 7 raids to one logical volume.

    I chose this setup to minimize the recovery time if one raid set has problems and needs recovery.

    The system is a Lucid (10.4 LTS) server installation.

    Here's my problem

    For some time one drive is behaving a little bit strange and is failing sometimes on startup of the system. A restart fixed it until now. Sometimes this means that the raid has to regenerate some sets or the whole raid. Yesterdays fail caused a full regeneration over all 7 raids.
    Usually this takes one night and the system is back again.

    The output of more /proc/mdstat is something like this
    Code:
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid1 sda8[0] sdc8[1] sdb8[3]
    116390840 blocks super 1.2 [3/2] [_UU]
    resync=DELAYED
    
    md1 : active raid5 sda1[0] sdc1[1] sdb1[3]
    524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [_UU]
    resync=DELAYED
    
    md7 : active raid5 sda7[4] sdc7[1] sdb7[3]
    524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [_UU]
    resync=DELAYED
    
    md6 : active raid5 sda6[0] sdc6[1] sdb6[3]
    524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [_UU]
    [>....................] recovery = 0.0% (85120/262142784) finish=564.3min speed=7738K/sec
    
    md5 : active raid5 sda5[0] sdc5[1] sdb5[3]
    524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [_UU]
    resync=DELAYED
    
    md4 : active raid5 sda4[0] sdc4[1] sdb4[3]
    524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [_UU]
    resync=DELAYED
    
    md3 : active raid5 sda3[4] sdc3[1] sdb3[3]
    524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [_UU]
    resync=DELAYED
    
    md2 : active raid5 sda2[0] sdc2[4] sdb2[3]
    524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [_UU]
    resync=DELAYED
    But suddenly in the morning some partitions of healthy parts of some raids became spare drives. As a result the /proc/mdadm looks now like this
    Code:
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid1 sda8[0] sdc8[1](F) sdb8[3]
          116390840 blocks super 1.2 [3/2] [U_U]
    
    md1 : active raid5 sda1[0](S) sdc1[1](F) sdb1[3]
          524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/1] [__U]
    
    md7 : active raid5 sda7[4] sdc7[1](F) sdb7[3]
          524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [U_U]
    
    md6 : active raid5 sda6[0] sdc6[1](F) sdb6[3]
          524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [U_U]
    
    md5 : active raid5 sda5[0](S) sdc5[1](F) sdb5[3]
          524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/1] [__U]
    
    md4 : active raid5 sda4[0](S) sdc4[1](F) sdb4[3]
          524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/1] [__U]
    
    md3 : active raid5 sda3[4] sdc3[1](F) sdb3[3]
          524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [U_U]
    
    md2 : active raid5 sda2[0] sdc2[4](F) sdb2[3]
          524285568 blocks super 1.2 level 5, 64k chunk, algorithm 2 [3/2] [U_U]
    
    unused devices: <none>
    To fix the failed drive sdc the system had to be rebooted but i don't know how to deal with the "sudden" spares. Therefore i don't dare to do the reboot now.

    Can please someone help me and provide ideas what to do?

    Thanks very much in advance

    dolittle

  2. #2
    Join Date
    Oct 2011
    Beans
    9

    Re: RAID drives suddenly become spare drives

    Does nobody have an idea?

    Please help me. This is really bad for me.

    Thanks in advance

    dolittle

  3. #3
    Join Date
    Nov 2009
    Location
    Segur De Calafell, Spain
    Beans
    11,660
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: RAID drives suddenly become spare drives

    I'm not mdadm expert, and I understand you want to know ASAP how to fix this, but you will have to be patient someone to jump in.

    Meanwhile, you can get more details about md1 for example, which has one spare and one failed partition with:
    sudo mdadm -E /dev/sd[abc]1

    That will list mdadm superblock details for all three partitions. I would watch out for the Events counter. They should match on all partitions, or at least on two.

    You can start with posting those details.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 12.04 LTS 64bit & Windows 7 Ultimate 64bit

  4. #4
    Join Date
    Oct 2011
    Beans
    9

    Re: RAID drives suddenly become spare drives

    Thanks for the reply. Drive sdc is completely offline right now. But it will be up again after the next boot. I don't worry about it. It will have to be replaced but thats a different story.

    Due to this i can post only for sdaX and sdbX

    mdadm -E output for md1
    Code:
    root@mediacenter:~# mdadm -E /dev/sda1
    /dev/sda1:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : b62960c9:9c2ff240:6ce247d2:89823d8a
               Name : ubuntu:1
      Creation Time : Wed Jun 22 00:36:45 2011
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 524285952 (250.00 GiB 268.43 GB)
         Array Size : 1048571136 (500.00 GiB 536.87 GB)
      Used Dev Size : 524285568 (250.00 GiB 268.43 GB)
        Data Offset : 2048 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 67c323c2:ebe508d2:24c05758:0b0dc597
    
        Update Time : Fri Nov 16 13:09:14 2012
           Checksum : 1067e5d2 - correct
             Events : 18532
    
             Layout : left-symmetric
         Chunk Size : 64K
    
        Array Slot : 0 (empty, failed, failed, 2, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
       Array State : __u 382 failed
    root@mediacenter:~# mdadm -E /dev/sdb1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : b62960c9:9c2ff240:6ce247d2:89823d8a
               Name : ubuntu:1
      Creation Time : Wed Jun 22 00:36:45 2011
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 524287728 (250.00 GiB 268.44 GB)
         Array Size : 1048571136 (500.00 GiB 536.87 GB)
      Used Dev Size : 524285568 (250.00 GiB 268.43 GB)
        Data Offset : 272 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : ee43ca75:0e6d9d6e:970e905c:e94053a9
    
        Update Time : Fri Nov 16 13:09:14 2012
           Checksum : 76696fcf - correct
             Events : 18532
    
             Layout : left-symmetric
         Chunk Size : 64K
    
        Array Slot : 3 (empty, failed, failed, 2, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
       Array State : __U 382 failed
    mdadm -E output for md4
    Code:
    root@mediacenter:~# mdadm -E /dev/sda4
    /dev/sda4:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 6cace689:3fa697ee:d1d9b0fd:a9f13de2
               Name : ubuntu:4
      Creation Time : Wed Jun 22 00:37:35 2011
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 524285952 (250.00 GiB 268.43 GB)
         Array Size : 1048571136 (500.00 GiB 536.87 GB)
      Used Dev Size : 524285568 (250.00 GiB 268.43 GB)
        Data Offset : 2048 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 24f7210c:a33ddc95:f30cc951:393be419
    
        Update Time : Fri Nov 16 05:43:48 2012
           Checksum : 8e5f394f - correct
             Events : 40390
    
             Layout : left-symmetric
         Chunk Size : 64K
    
        Array Slot : 0 (empty, failed, failed, 2, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
       Array State : __u 382 failed
    root@mediacenter:~# mdadm -E /dev/sdb4
    /dev/sdb4:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 6cace689:3fa697ee:d1d9b0fd:a9f13de2
               Name : ubuntu:4
      Creation Time : Wed Jun 22 00:37:35 2011
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 524287728 (250.00 GiB 268.44 GB)
         Array Size : 1048571136 (500.00 GiB 536.87 GB)
      Used Dev Size : 524285568 (250.00 GiB 268.43 GB)
        Data Offset : 272 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : aa86866e:ad5cb9e0:b30b61ea:2d18bde6
    
        Update Time : Fri Nov 16 05:43:48 2012
           Checksum : a111c398 - correct
             Events : 40390
    
             Layout : left-symmetric
         Chunk Size : 64K
    
        Array Slot : 3 (empty, failed, failed, 2, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
       Array State : __U 382 failed
    mdadm -E output for md5
    Code:
    root@mediacenter:~# mdadm -E /dev/sda5
    /dev/sda5:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 99891c18:4900aaee:7c7d140a:9e965a91
               Name : ubuntu:5
      Creation Time : Wed Jun 22 00:37:45 2011
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 524285952 (250.00 GiB 268.43 GB)
         Array Size : 1048571136 (500.00 GiB 536.87 GB)
      Used Dev Size : 524285568 (250.00 GiB 268.43 GB)
        Data Offset : 2048 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 1ab1ee6a:bfb2e8cd:3f8df469:e7164bae
    
        Update Time : Fri Nov 16 05:43:48 2012
           Checksum : 1c959d48 - correct
             Events : 194258
    
             Layout : left-symmetric
         Chunk Size : 64K
    
        Array Slot : 0 (empty, failed, failed, 2, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
       Array State : __u 382 failed
    root@mediacenter:~# mdadm -E /dev/sdb5
    /dev/sdb5:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 99891c18:4900aaee:7c7d140a:9e965a91
               Name : ubuntu:5
      Creation Time : Wed Jun 22 00:37:45 2011
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 524287728 (250.00 GiB 268.44 GB)
         Array Size : 1048571136 (500.00 GiB 536.87 GB)
      Used Dev Size : 524285568 (250.00 GiB 268.43 GB)
        Data Offset : 272 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : f6c5ea70:78ce33ae:55cdb935:ed4fc916
    
        Update Time : Fri Nov 16 05:43:48 2012
           Checksum : 372046fb - correct
             Events : 194258
    
             Layout : left-symmetric
         Chunk Size : 64K
    
        Array Slot : 3 (empty, failed, failed, 2, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
       Array State : __U 382 failed
    The events counters match on each pair. I hope there's an easy way just to "switch" the Spare partitions back to normal, reboot and wait for the rebuild.

    Hope someone has an idea.

    Thanks

  5. #5
    Join Date
    Aug 2005
    Location
    Adelaide, SA, Australia
    Beans
    549
    Distro
    Ubuntu 13.10 Saucy Salamander

    Re: RAID drives suddenly become spare drives

    Don't use RAID5 ever.

  6. #6
    Join Date
    Nov 2009
    Location
    Segur De Calafell, Spain
    Beans
    11,660
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: RAID drives suddenly become spare drives

    The events counters give some hope, but since this is a raid5 array and one disk was already failing, I am worried is it still good with partitions from a second disk marked as spare.

    I'll try to contact one guy that knows much more about mdadm, but lets see if he can join the discussion.

    PS. This is probably not what you want to hear, but with one disk failing regularly you decided to keep running a raid5 array which tolerates only one disk failure. You were asking for trouble if something happens to a second disk, and it finally did.
    Last edited by darkod; November 16th, 2012 at 11:39 PM.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 12.04 LTS 64bit & Windows 7 Ultimate 64bit

  7. #7
    Join Date
    Oct 2011
    Beans
    9

    Re: RAID drives suddenly become spare drives

    Thanks. You're very nice.

  8. #8
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,120
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: RAID drives suddenly become spare drives

    Well, this is the most convoluted mdadm setup I've ever seen. This is just asking for breakage, and I'm honestly surprised it's worked well for you for some time. First question, do you have a backup of this data?

    The first step to try to remedy this is to reboot, and get /dev/sdc online. The next step is to run smartmontools on each disk and verify their health.

    Code:
    apt-get install smartmontools
    smartctl -a /dev/sda
    smartctl -a /dev/sdb
    smartctl -a /dev/sdc
    If all disks pass and SMART test, and don't have a bunch of Reallocated or Pending Sectors, then you can continue, if not, post back for further instructions.

    You are going to need to force assemble the arrays, that have disks marked as spares to get them back online.
    Code:
    mdadm --assemble --force /dev/md1 /dev/sd[abc]1
    mdadm --assemble --force /dev/md4 /dev/sd[abc]4
    If you can get this restored, I'd really suggest rebuilding / reconfiguring your arrays to greatly simplify this, and to add another disk and migrate to a RAID6.

  9. #9
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: RAID drives suddenly become spare drives

    @OP: I have never seen a RAID 5 set up like that.

    Quote Originally Posted by madverb View Post
    Don't use RAID5 ever.
    Got a reason for that?

    Oh yeah, +1 to Ruby - he is very good with mdadm.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  10. #10
    Join Date
    Oct 2011
    Beans
    9

    Re: RAID drives suddenly become spare drives

    @rubylaser: Thanks for the information. I'll report back as soon as I'm finished. It may take some hours since I try to salvage everthing readable from the raid.

    Quote Originally Posted by CharlesA View Post
    @OP: I have never seen a RAID 5 set up like that.
    Some of you seen to have ideas for improvement of my raid setup I'd be happy if you'd share them with me. It would be a perfect time to execute them.

    Thanks for your help

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •