Page 1 of 3 123 LastLast
Results 1 to 10 of 24

Thread: Missing Superblocks on mdadm Raid5

  1. #1
    Join Date
    Apr 2010
    Beans
    81

    Missing Superblocks on mdadm Raid5

    My Mdadm Raid5 array of 4 drives failed during synchronization after replacement of one failed drive. It threw me to RamFS where after a while I couldn't mount the array any more (don't know what I did... ).



    I inserted Ubuntu live CD and this is where I am now :

    • fdisk finds all the partitions on all 4 drives correctly (/dev/sda - /dev/sdd)


    • I can assemble array using
      Code:
      mdadm --create /dev/md3 --assume-clean --level=5 --raid-devices=4 missing /dev/sdb3 /dev/sda3 /dev/sdc3
      
      cat /proc/mdstat
      Personalities : [raid6] [raid5] [raid4]
      md3 : active raid5 sda3[3] sda1[2] sda2[1]
      2923643904 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]


    • I can't mount the array (/dev/md3) using:
      Code:
      mount -t ext3 /dev/md3 /mnt/
      mount: wrong fs type, bad option, bad superblock on /dev/md3,
      missing codepage or helper program, or other error
      In some cases useful info is found in syslog - try
      dmesg | tail or so


    • checking filesystem with e2fsck fails
      Code:
      e2fsck /dev/md3
      e2fsck 1.42.5 (29-Jul-2012)
      ext2fs_open2: Bad magic number in super-block
      e2fsck: Superblock invalid, trying backup blocks...
      e2fsck: Bad magic number in super-block while trying to open /dev/md3
      
      The superblock could not be read or does not describe a correct ext2
      filesystem. If the device is valid and it really contains an ext2
      filesystem (and not swap or ufs or something else), then the superblock
      is corrupt, and you might try running e2fsck with an alternate superblock:
      e2fsck -b 8193


    • using dumpe2fs /dev/md3 won't find any Superblocks:
      Code:
      dumpe2fs /dev/md3 dumpe2fs 1.42.5 (29-Jul-2012)
      dumpe2fs: Bad magic number in super-block while trying to open /dev/md3
      Couldn't find valid filesystem superblock.


    It seems to me that all the Superblocks of ext3 filesystem on the raid array are missing.

    1. Is there a way to recreate the superblock on the array? Or am I doomed?
    2. Am I missing something? Did I forget to use some commands?
    3. If not: is there a way to recover all the files/directories?
    4. And in the worst possible case: how do I recover particular files (eg.: some bash scripts in my /home/ directory)? Which tools do you suggest?


    I will appreciate a lot your help because I'm completely in the dark here .


    p.s.: dumping the content of /dev/md3 and using "grep" shows that there's still data on it
    p.p.s.: I have a backup of important data. Only after disk failure do you figure out that some "unimportant" data was actually quite important.

  2. #2
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,132
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Missing Superblocks on mdadm Raid5

    When you re-create the array with the assume clean option, the metadata version has to be the same, and the disk order exactly the same. Are you positive that both of those conditions are met?

  3. #3
    Join Date
    Apr 2010
    Beans
    81

    Re: Missing Superblocks on mdadm Raid5

    I have no idea about the metadata version. How do I check that? When I created the array I was using Ubuntu 8.04 (I guess) so quite a long time ago (year 2008).

    And I'm 90% sure about the disk order (I'm working on a "dd copy" of all partitions not to damage the data).

  4. #4
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,132
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Missing Superblocks on mdadm Raid5

    If your array is that old, then your metadata version should be 0.90. You can redo the create --assume-clean again, but this time pass the --metadata=0.90 in. Also, your chunk size should have been 64K, so pass this in as well --chunk=64

  5. #5
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,310
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Missing Superblocks on mdadm Raid5

    Also note in your --create command you posted above, the order of the devices is "missing /dev/sdb3 /dev/sda3 /dev/sdc3".

    As rubylaser said, the exact same order is very important. When you created the array as new, wouldn't the logical disk order be like "sda3 sdb3 sdc3 sdd3"?

    And after you did the --create, look at the /proc/mdstat. It actually says md3 is assembled with "sda3 sda1 sda2". All are partitions on the same disk, not different disks.

    Any thoughts on that rubylaser?
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  6. #6
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,132
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Missing Superblocks on mdadm Raid5

    Quote Originally Posted by darkod View Post
    Also note in your --create command you posted above, the order of the devices is "missing /dev/sdb3 /dev/sda3 /dev/sdc3".

    As rubylaser said, the exact same order is very important. When you created the array as new, wouldn't the logical disk order be like "sda3 sdb3 sdc3 sdd3"?

    And after you did the --create, look at the /proc/mdstat. It actually says md3 is assembled with "sda3 sda1 sda2". All are partitions on the same disk, not different disks.

    Any thoughts on that rubylaser?
    I hope that's just a copy / paste error, because I can't explain that unless that was a failed auto assemble. I would stop /dev/md3 and try again with the create command.

    Code:
    mdadm --stop /dev/md3

  7. #7
    Join Date
    Apr 2010
    Beans
    81

    Re: Missing Superblocks on mdadm Raid5

    Wow rubylaser. Thanks. This definitely helped. After checking my old "mdadm --detail /dev/md3" it was indeed "metadata version 0.9" and "chunk size 64K".

    I assembled as you said:
    Code:
    mdadm --create /dev/md3 --assume-clean --metadata=0.90 --level=5  --chunk=64 --raid-devices=4 missing /dev/sdb2 /dev/sdb1 /dev/sdb3
    mdadm: /dev/sdb2 appears to contain an ext2fs file system
    size=-1102493376K mtime=Tue Nov 21 08:31:46 2028
    mdadm: /dev/sdb2 appears to be part of a raid array:
    level=raid5 devices=4 ctime=Sun Feb 24 03:57:57 2013
    mdadm: /dev/sdb1 appears to be part of a raid array:
    level=raid5 devices=4 ctime=Sun Feb 24 03:57:57 2013
    mdadm: /dev/sdb3 appears to be part of a raid array:
    level=raid5 devices=4 ctime=Sun Feb 24 04:54:27 2013
    Continue creating array? y
    mdadm: array /dev/md3 started.
    Why does it say /dev/sdb2 cointains ext2 while the others not?


    Now I can finally see some information back from dumpe2fs. So this is what it outputs:
    Code:
    dumpe2fs /dev/md3
    dumpe2fs 1.42 (29-Nov-2011)
    Filesystem volume name: storage
    Last mounted on: <not available>
    Filesystem UUID: a748cb0a-427a-43c8-956d-773c11d5eead
    Filesystem magic number: 0xEF53
    Filesystem revision #: 1 (dynamic)
    Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
    Filesystem flags: signed_directory_hash
    Default mount options: (none)
    Filesystem state: clean
    Errors behavior: Continue
    Filesystem OS type: Linux
    Inode count: 182755328
    Block count: 731009616
    Reserved block count: 36550480
    Free blocks: 39511366
    Free inodes: 181309822
    First block: 0
    Block size: 4096
    Fragment size: 4096
    Reserved GDT blocks: 849
    Blocks per group: 32768
    Fragments per group: 32768
    Inodes per group: 8192
    Inode blocks per group: 256
    Filesystem created: Sun Aug 3 23:06:31 2008
    Last mount time: Fri Feb 1 18:54:42 2013
    Last write time: Sat Jan 19 19:06:50 2013
    Mount count: 16
    Maximum mount count: 25
    Last checked: Sat Jan 19 19:06:50 2013
    Check interval: 15552000 (6 months)
    Next check after: Thu Jul 18 20:06:50 2013
    Reserved blocks uid: 0 (user root)
    Reserved blocks gid: 0 (group root)
    First inode: 11
    Inode size: 128
    Journal inode: 8
    Default directory hash: tea
    Directory Hash Seed: ad1bab00-ae50-4870-952f-05bbf2413152
    Journal backup: inode blocks
    dumpe2fs: A block group is missing an inode table while reading journal inode
    I guess the superblock is back. But trying to mount it still fails:
    Code:
    mount -t ext3 -o ro /dev/md3 /mnt/
    mount: wrong fs type, bad option, bad superblock on /dev/md3,
    missing codepage or helper program, or other error
    In some cases useful info is found in syslog - try
    dmesg | tail or so
    So I guess there are errors that have to be corrected. I ran:
    Code:
    e2fsck /dev/md3
    e2fsck 1.42 (29-Nov-2011)
    e2fsck: Group descriptors look bad... trying backup blocks...
    Superblock has an invalid journal (inode 8).
    Clear<y>?
    What should I do? Say yes to clear?



    And after you did the --create, look at the /proc/mdstat. It actually says md3 is assembled with "sda3 sda1 sda2". All are partitions on the same disk, not different disks."
    This was a copy/paste error on my part. The original disks were /dev/sd[a-d]3 but I dd copied the partitions to new disk /dev/sda[1-3] (and /dev/sdd3 missing). The order of disks changed through time because of reboots and disk replacements.

  8. #8
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,310
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Missing Superblocks on mdadm Raid5

    Quote Originally Posted by Kljuka View Post
    This was a copy/paste error on my part. The original disks were /dev/sd[a-d]3 but I dd copied the partitions to new disk /dev/sda[1-3] (and /dev/sdd3 missing). The order of disks changed through time because of reboots and disk replacements.
    Did you do this only as data access procedure? I hope you are not planning to run a raid array with all partitions on the same disk.

    Even for a short time, that might be too intense for the disk, imagine it has to race all over the disk since you made it have raid5 array with three partitions on the same disk. I don't understand. Even if one disk failed, you still had three left, right? Why dd-ing to a single disk?
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  9. #9
    Join Date
    Apr 2010
    Beans
    81

    Re: Missing Superblocks on mdadm Raid5

    I wouldn't run raid5 from 1 disk. This is only a "Save the data procedure". I don't want to test Data Recovery on original data disks (not to mess things up even further).
    Last edited by Kljuka; February 24th, 2013 at 09:26 PM.

  10. #10
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,132
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Missing Superblocks on mdadm Raid5

    It does look like it recognizes the filesystem, but it's messed up. If you run the fsck like this it's very likely you'll end up with almost the whole filesystem in your lost+found folder. If this is just a test, and you still have the original disks, it's worth a shot at this point.

Page 1 of 3 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •