Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: Unable to Rebuild Raid 1

  1. #1
    Join Date
    Mar 2010
    Beans
    8

    Red face Unable to Rebuild Raid 1

    Hello,

    I am running Ubuntu Server 12.04. I installed with a software raid1 array. To test the array I pulled one of the drives and made sure it would boot. I then tried the same with the other drive. Both booted as expected.

    When I try to rebuild the array I am running into issues. Some background info:

    Code:
    ~# cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid1 sdb3[1]
          479008632 blocks super 1.2 [2/1] [_U]
    
    md1 : active raid1 sdb1[1]
          975860 blocks super 1.2 [2/1] [_U]
    
    unused devices: <none>
    Code:
    ~# mdadm --query --detail /dev/md0
    /dev/md0:
            Version : 1.2
      Creation Time : Thu May 24 11:46:44 2012
         Raid Level : raid1
         Array Size : 479008632 (456.82 GiB 490.50 GB)
      Used Dev Size : 479008632 (456.82 GiB 490.50 GB)
       Raid Devices : 2
      Total Devices : 1
        Persistence : Superblock is persistent
    
        Update Time : Tue Jun 26 15:52:31 2012
              State : clean, degraded
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 0
      Spare Devices : 0
    
               Name : DunnAppraisals:0
               UUID : d154ced9:cd0fcc08:0f4f2986:cae8dc98
             Events : 14729
    
        Number   Major   Minor   RaidDevice State
           0       0        0        0      removed
           1       8       19        1      active sync   /dev/sdb3
    Code:
    ~# mdadm --query --detail /dev/md1
    /dev/md1:
            Version : 1.2
      Creation Time : Thu May 24 11:49:04 2012
         Raid Level : raid1
         Array Size : 975860 (953.15 MiB 999.28 MB)
      Used Dev Size : 975860 (953.15 MiB 999.28 MB)
       Raid Devices : 2
      Total Devices : 1
        Persistence : Superblock is persistent
    
        Update Time : Tue Jun 26 15:51:38 2012
              State : clean, degraded
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 0
      Spare Devices : 0
    
               Name : DunnAppraisals:1
               UUID : 741ee215:7c42dd3a:f7ec0cff:5a5b9ae9
             Events : 117
    
        Number   Major   Minor   RaidDevice State
           0       0        0        0      removed
           1       8       17        1      active sync   /dev/sdb1
    To me this looks like md0 is missing sda3 and md1 is missing sda1, so I tried to add them:

    Code:
    ~# mdadm --manage /dev/md0 --add /dev/sda3
    mdadm: /dev/sda3 reports being an active member for /dev/md0, but a --re-add fails.
    mdadm: not performing --add as that would convert /dev/sda3 in to a spare.
    mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sda3" first.
    I get the same result when trying to add sda1 to md1. I also tried --force. How is sda3 an active member of md0? Nowhere in /proc/mdstat or the mdadm query does it hint that. Thinking I am crazy or am getting something backwards, I tried to add sdb3 to md0:

    Code:
    ~# mdadm --manage /dev/md0 --add /dev/sdb3
    mdadm: Cannot open /dev/sdb3: Device or resource busy
    Again I get the same result when adding sdb1 to md1. Last thing that I tried was to add nodmraid to /etc/default/grub ("GRUB_CMDLINE_LINUX_DEFAULT="nodmraid quiet splash"), update-grub, reboot and try everything again. Nada. (As a side note, I tried entering nodmraid straight into the grub command line and it said it wasn't a valid command)

    Thanks in advance for reading this and taking the time to respond. This is my first time managing a raid in linux, be gentle.

  2. #2
    Join Date
    Nov 2009
    Location
    Segur De Calafell, Spain
    Beans
    11,660
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Unable to Rebuild Raid 1

    If I am not mistaken when reinserting the same disk it's better to use the --re-add command. So try something like:
    mdadm --manage /dev/md0 --re-add /dev/sda3
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 12.04 LTS 64bit & Windows 7 Ultimate 64bit

  3. #3
    Join Date
    Mar 2010
    Beans
    8

    Re: Unable to Rebuild Raid 1

    Thanks for the reply darkod. No luck though...
    Code:
    ~# mdadm --manage /dev/md0 --re-add /dev/sda3
    mdadm: --re-add for /dev/sda3 to /dev/md0 is not possible

  4. #4
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,106
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Unable to Rebuild Raid 1

    Quote Originally Posted by darkod View Post
    If I am not mistaken when reinserting the same disk it's better to use the --re-add command. So try something like:
    mdadm --manage /dev/md0 --re-add /dev/sda3
    This only works it there is an intent bitmap, and cat /proc/mdstat doesn't reflect one, so I'm sure one doesn't exist. What's the state of the superblock on each disk?

    Code:
    mdadm -E /dev/sd[ab]1
    mdadm -E /dev/sd[ab]3
    If the event counters are similar, it's probably easiest to boot from the LiveCD, install mdadm, and then reassemble each array. Once they are in sync, you can reboot, and all should be well.

  5. #5
    Join Date
    Mar 2010
    Beans
    8

    Re: Unable to Rebuild Raid 1

    State of all superblocks is clean. Events:

    sda1 - 77
    sdb1 - 127
    sda3 - 177
    sdb3 - 22011

    Based on that, should I install mdadm from the LiveCD? Seems like sdb3 is a little out of whack...

    Both disks are identical at this point (for all practical purposes). Would another option be to wipe the disk, re-partition and then build the raid? How could I do this safely?

  6. #6
    Join Date
    Nov 2009
    Location
    Segur De Calafell, Spain
    Beans
    11,660
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Unable to Rebuild Raid 1

    Quote Originally Posted by nclucid View Post
    State of all superblocks is clean. Events:

    sda1 - 77
    sdb1 - 127
    sda3 - 177
    sdb3 - 22011

    Based on that, should I install mdadm from the LiveCD? Seems like sdb3 is a little out of whack...

    Both disks are identical at this point (for all practical purposes). Would another option be to wipe the disk, re-partition and then build the raid? How could I do this safely?
    rubylaser mentioned that option if the counters are similar. They look very different especially for md0.

    Since this is only raid1, assuming the sdb disk has the full data, you could zero the superblock of sda1 and sda3 for example and try to add the disk as if it was new.

    But lets wait what rubylaser says, he's the expert for raid.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 12.04 LTS 64bit & Windows 7 Ultimate 64bit

  7. #7
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,106
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Unable to Rebuild Raid 1

    Quote Originally Posted by darkod View Post
    rubylaser mentioned that option if the counters are similar. They look very different especially for md0.

    Since this is only raid1, assuming the sdb disk has the full data, you could zero the superblock of sda1 and sda3 for example and try to add the disk as if it was new.

    But lets wait what rubylaser says, he's the expert for raid.
    +1. I would zero the superblock on both of those sda partitions (sda1 and sda3), and then add them back to their appropriate arrays just as darkod has said. Being slightly different on the counters isn't typically a being deal for putting an array back together, but being off by over 20,000 is

  8. #8
    Join Date
    Mar 2010
    Beans
    8

    Re: Unable to Rebuild Raid 1

    Quote Originally Posted by darkod View Post
    Since this is only raid1, assuming the sdb disk has the full data, you could zero the superblock of sda1 and sda3 for example and try to add the disk as if it was new.
    Worked like a charm!

    Code:
    ~# mdadm --zero-superblock /dev/sda1
    ~# mdadm --zero-superblock /dev/sda3
    ~# mdadm --manage /dev/md0 --add /dev/sda3
    mdadm: added /dev/sda3
    ~# mdadm --manage /dev/md1 --add /dev/sda1
    mdadm: added /dev/sda1
    Code:
    ~# watch -n1 cat /proc/mdstat
    Every 1.0s: cat /proc/mdstat                                                                                                                                                   Wed Jun 27 08:56:12 2012
    
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid1 sda3[2] sdb3[1]
          479008632 blocks super 1.2 [2/1] [_U]
          [=>...................]  recovery =  7.8% (37384832/479008632) finish=61.8min speed=119022K/sec
    
    md1 : active raid1 sda1[2] sdb1[1]
          975860 blocks super 1.2 [2/1] [_U]
            resync=DELAYED
    
    unused devices: <none>
    Looks like it's rebuilding as expected. Thank you Darko and rubylaser for the quick and extremely helpful responses

  9. #9
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,106
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Unable to Rebuild Raid 1

    Glad you got it working

  10. #10
    Join Date
    Apr 2012
    Beans
    5,466

    Re: Unable to Rebuild Raid 1

    pmji but would one of you good folks be able to give a short explanation of what the RAID event counters are counting? how do they relate (if at all) to the various SMART disk counters? tia

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •