Unable to Rebuild Raid 1

[nclucid]

Hello,

I am running Ubuntu Server 12.04. I installed with a software raid1 array. To test the array I pulled one of the drives and made sure it would boot. I then tried the same with the other drive. Both booted as expected.

When I try to rebuild the array I am running into issues. Some background info:

~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb3[1]
      479008632 blocks super 1.2 [2/1] [_U]

md1 : active raid1 sdb1[1]
      975860 blocks super 1.2 [2/1] [_U]

unused devices: <none>

~# mdadm --query --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Thu May 24 11:46:44 2012
     Raid Level : raid1
     Array Size : 479008632 (456.82 GiB 490.50 GB)
  Used Dev Size : 479008632 (456.82 GiB 490.50 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Tue Jun 26 15:52:31 2012
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : DunnAppraisals:0
           UUID : d154ced9:cd0fcc08:0f4f2986:cae8dc98
         Events : 14729

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       19        1      active sync   /dev/sdb3

~# mdadm --query --detail /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Thu May 24 11:49:04 2012
     Raid Level : raid1
     Array Size : 975860 (953.15 MiB 999.28 MB)
  Used Dev Size : 975860 (953.15 MiB 999.28 MB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Tue Jun 26 15:51:38 2012
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : DunnAppraisals:1
           UUID : 741ee215:7c42dd3a:f7ec0cff:5a5b9ae9
         Events : 117

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       17        1      active sync   /dev/sdb1

To me this looks like md0 is missing sda3 and md1 is missing sda1, so I tried to add them:

~# mdadm --manage /dev/md0 --add /dev/sda3
mdadm: /dev/sda3 reports being an active member for /dev/md0, but a --re-add fails.
mdadm: not performing --add as that would convert /dev/sda3 in to a spare.
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sda3" first.

I get the same result when trying to add sda1 to md1. I also tried --force. How is sda3 an active member of md0? Nowhere in /proc/mdstat or the mdadm query does it hint that. Thinking I am crazy or am getting something backwards, I tried to add sdb3 to md0:

~# mdadm --manage /dev/md0 --add /dev/sdb3
mdadm: Cannot open /dev/sdb3: Device or resource busy

Again I get the same result when adding sdb1 to md1. Last thing that I tried was to add nodmraid to /etc/default/grub ("GRUB_CMDLINE_LINUX_DEFAULT="nodmraid quiet splash"), update-grub, reboot and try everything again. Nada. (As a side note, I tried entering nodmraid straight into the grub command line and it said it wasn't a valid command)

Thanks in advance for reading this and taking the time to respond. This is my first time managing a raid in linux, be gentle.

[darkod]

If I am not mistaken when reinserting the same disk it's better to use the --re-add command. So try something like:
mdadm --manage /dev/md0 --re-add /dev/sda3

[nclucid]

Thanks for the reply darkod. No luck though...

~# mdadm --manage /dev/md0 --re-add /dev/sda3
mdadm: --re-add for /dev/sda3 to /dev/md0 is not possible

[rubylaser]

If I am not mistaken when reinserting the same disk it's better to use the --re-add command. So try something like:
mdadm --manage /dev/md0 --re-add /dev/sda3

This only works it there is an intent bitmap, and cat /proc/mdstat doesn't reflect one, so I'm sure one doesn't exist. What's the state of the superblock on each disk?

mdadm -E /dev/sd[ab]1
mdadm -E /dev/sd[ab]3

If the event counters are similar, it's probably easiest to boot from the LiveCD, install mdadm, and then reassemble each array. Once they are in sync, you can reboot, and all should be well.

[nclucid]

State of all superblocks is clean. Events:

sda1 - 77
sdb1 - 127
sda3 - 177
sdb3 - 22011

Based on that, should I install mdadm from the LiveCD? Seems like sdb3 is a little out of whack...

Both disks are identical at this point (for all practical purposes). Would another option be to wipe the disk, re-partition and then build the raid? How could I do this safely?

[darkod]

State of all superblocks is clean. Events:

sda1 - 77
sdb1 - 127
sda3 - 177
sdb3 - 22011

Based on that, should I install mdadm from the LiveCD? Seems like sdb3 is a little out of whack...

Both disks are identical at this point (for all practical purposes). Would another option be to wipe the disk, re-partition and then build the raid? How could I do this safely?

rubylaser mentioned that option if the counters are similar. They look very different especially for md0.

Since this is only raid1, assuming the sdb disk has the full data, you could zero the superblock of sda1 and sda3 for example and try to add the disk as if it was new.

But lets wait what rubylaser says, he's the expert for raid.

[rubylaser]

rubylaser mentioned that option if the counters are similar. They look very different especially for md0.

Since this is only raid1, assuming the sdb disk has the full data, you could zero the superblock of sda1 and sda3 for example and try to add the disk as if it was new.

But lets wait what rubylaser says, he's the expert for raid.

+1. I would zero the superblock on both of those sda partitions (sda1 and sda3), and then add them back to their appropriate arrays just as darkod has said. Being slightly different on the counters isn't typically a being deal for putting an array back together, but being off by over 20,000 is

[nclucid]

Since this is only raid1, assuming the sdb disk has the full data, you could zero the superblock of sda1 and sda3 for example and try to add the disk as if it was new.

Worked like a charm!

~# mdadm --zero-superblock /dev/sda1
~# mdadm --zero-superblock /dev/sda3
~# mdadm --manage /dev/md0 --add /dev/sda3
mdadm: added /dev/sda3
~# mdadm --manage /dev/md1 --add /dev/sda1
mdadm: added /dev/sda1

~# watch -n1 cat /proc/mdstat
Every 1.0s: cat /proc/mdstat                                                                                                                                                   Wed Jun 27 08:56:12 2012

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda3[2] sdb3[1]
      479008632 blocks super 1.2 [2/1] [_U]
      [=>...................]  recovery =  7.8% (37384832/479008632) finish=61.8min speed=119022K/sec

md1 : active raid1 sda1[2] sdb1[1]
      975860 blocks super 1.2 [2/1] [_U]
        resync=DELAYED

unused devices: <none>

Looks like it's rebuilding as expected. Thank you Darko and rubylaser for the quick and extremely helpful responses

[rubylaser]

Glad you got it working

[steeldriver]

pmji but would one of you good folks be able to give a short explanation of what the RAID event counters are counting? how do they relate (if at all) to the various SMART disk counters? tia