Missing Superblocks on mdadm Raid5

[Kljuka]

Yes, it's been taking fsck already a couple of hours doing the job.

Do you think this might be a sign that I assembled the array with the drives in a wrong order? Is there a way to figure out the order of disks by assembling the array and afterwards doing some superblock check?

p.s.: I really wanted to understand how mdadm actually assembles the array (logic and mathematics used in mdadm) but I could only find a lot of forums full of raid disasters and partial (sometimes inaccurate) solutions. Seems to me there's no official structured and more in depth documentation available.

[tealio]

*** EDIT *** : using --zero-superblock should not be used in this case. --assume-clean is better for this application. It worked for me, but apparently was a bad idea.


if fsck is taking that long, you've probably already trashed the filesystem. I know, because i assembled one in the wrong order just yesterday. It was late, and instead of sleeping and dealing with it in the morning, i decided to go ahead and destroy my filesystem. I had a backup.

It's probably too late for you, but here's what worked for me:

First, check the existing superblock too see what order the devices really go in:

root@AMD-Media:~# mdadm -E /dev/sd[abd] |egrep 'sd|Chunk|Role|Version'
/dev/sda:
        Version : 1.2
     Chunk Size : 128K
   Device Role : Active device 2
/dev/sdb:
        Version : 1.2
     Chunk Size : 128K
   Device Role : Active device 0
/dev/sdd:
        Version : 1.2
     Chunk Size : 128K
   Device Role : Active device 1

I'm using whole disks instead of partitions, and they are attached in a weird order, but your info should look similar.
Note the Device Role. They should be reassembled in the order 0 1 2. If there's a number missing (Let's say yours are numbered 0 1 3) they still just go in ascending order. In my case it's /dev/sdb /dev/sdd /dev/sda

Next, do something scary:

root@AMD-Media:~# mdadm --stop /dev/md0
root@AMD-Media:~# mdadm --remove /dev/md0
root@AMD-Media:~# mdadm --zero-superblock /dev/sd[abd]

This clears the superblock on each device so that mdadm doesn't give you any problems about assembling devices (partitions, whatever) that have different event numbers, modify dates, etc.

Next, use the info from the mdadm -E to recreate the array. In my case:

root@AMD-Media:~# mdadm --create /dev/md0 --bitmap=/etc/mdadm/md0.bitmap --chunk=128 --metadata=1.2 --level=5 --raid-devices=3 --force /dev/sdb /dev/sdd /dev/sda

I had to use force because my bitmap file already existed. Also, don't be tempted to use bracket expansion on the drive order. bracket expansion doesn't respect the order you put things in, and as far as i can tell it alphabetizes them, so

/dev/sd[bda]

isn't equal to

/dev/sdb /dev/sdd /dev/sda

I'd love to show you the output from that, but it's rebuilding right now, and i didn't cut and paste it when i did it...

It does, however come up and mention that there is evidence of an existing filesystem/partition table... If you're 100% SURE that you've got the metadata version, chunk size and order correct then you can allow the array to be created. It will do a resync, but your data should be immediately available to mount and use.

Again, this is what worked for ME and MY SITUATION! no guarantees!

[darkod]

Isn't a --create with --assume-clean a better option to try first? Without zeroing the superblocks. That should force it even if the events counters are not the same. Of course, you still have to use the same original order and use 'missing' for any missing members.

[tealio]

That is absolutely true. My case was a weird one though, because i was trying to upgrade disks.
I'm going on memory here, but i think i ended up with the 2 original disks and the new disk all having a newer creation date. Then i had one of the old disks fail. So i ended up trying to rebuild an array from 2 good disks with different creation dates, and a third disk that I'm not so sure about... I'm currently trying to rebuild the original array so i can start over. this has been going on since Wednesday.

If this rebuild fails at 99.9% like the others, then im going to run the array degraded, and put my new disk in a single disk RAID 1. Then I'll have to copy over the data, then mirror that disk, then convert to RAID5 and add the third disk... PITA

anyway, to answer your question, --assume-clean would probably still have been a better option.

I don't wanna get too far off here, but if --assume-clean is a better option with similar results to --zero-superblock, then what is the case in which --zero-superblock would be used?
Is it only for when you're really removing a drive from an array, but it's staying in your system? to keep mdadm from scanning it and trying to add it to an array?

[darkod]

Yes, as far as I understand it, you would use --zero-superblock to completely delete any traces of mdadm to use the disk as separate or add it later but as "new" disk, otherwise if it detects a superblock it might try to re-add it as disk that was earlier in the array and mess things for you.

For assembling existing array you would start with mdadm --assemble --scan, but that would work only if there are no major issues. If the events don't match I think it will fail. The next is --create --assume-clean or --create --run, with correct level and number of devices and their order.

If all of that fails, you need to look at drastic measures/attempts.

[tealio]

ok... so i just skipped straight to the big leagues... oops...

[chamsters]

Hey guys,

I'm having the identical issue and was hoping you guys could help me troubleshoot some?

root@server:~# dumpe2fs /dev/md3dumpe2fs 1.41.3 (12-Oct-2008)
Filesystem volume name: /var/lib/vz
Last mounted on: <not available>
Filesystem UUID: e4881189-a1aa-4dbd-9776-8df39c2f6039
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags: signed_directory_hash
Default mount options: (none)
Filesystem state: clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 179109888
Block count: 716422752
Reserved block count: 35821137
Free blocks: 236978281
Free inodes: 78393903
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 853
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8192
Inode blocks per group: 256
Filesystem created: Tue Aug 24 18:34:53 2010
Last mount time: Wed Feb 29 07:37:51 2012
Last write time: Wed May 8 16:02:53 2013
Mount count: 4
Maximum mount count: 24
Last checked: Thu Feb 16 10:44:25 2012
Check interval: 15552000 (6 months)
Next check after: Tue Aug 14 11:44:25 2012
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
First orphan inode: 45294652
Default directory hash: half_md4
Directory Hash Seed: da833f00-139d-4f21-90e2-38830d884e00
Journal backup: inode blocks
dumpe2fs: A block group is missing an inode table while reading journal inode

root@server:~# cat /proc/mdstatPersonalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md3 : active raid5 sdd3[2] sdc3[1] sda3[0]
1910460672 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]


md1 : active raid1 sda1[0] sdb1[1]
20480896 blocks [4/2] [UU__]


unused devices: <none>

root@server:~# mount -a /dev/md3mount: wrong fs type, bad option, bad superblock on /dev/md3,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so

[darkod]

What identical issue? The OP had the arrays not working, yours are active.

If /dev/md3 complains about mounting, did you try fsck first?

sudo fsck /dev/md3

Don't try to mount it before running fsck, it needs to be unmounted.

[chamsters]

What identical issue? The OP had the arrays not working, yours are active.

If /dev/md3 complains about mounting, did you try fsck first?

sudo fsck /dev/md3

Don't try to mount it before running fsck, it needs to be unmounted.

root@londonpower:~# fsck /dev/md3fsck 1.41.3 (12-Oct-2008)
e2fsck 1.41.3 (12-Oct-2008)
fsck.ext3: Group descriptors look bad... trying backup blocks...
fsck.ext3: Bad magic number in super-block while trying to open /dev/md3


The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193 <device>

that's what I get....

[darkod]

What do you have on md3? Can you post:

sudo parted -l