Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 24

Thread: Missing Superblocks on mdadm Raid5

  1. #11
    Join Date
    Apr 2010
    Beans
    84

    Re: Missing Superblocks on mdadm Raid5

    Yes, it's been taking fsck already a couple of hours doing the job.

    Do you think this might be a sign that I assembled the array with the drives in a wrong order? Is there a way to figure out the order of disks by assembling the array and afterwards doing some superblock check?

    p.s.: I really wanted to understand how mdadm actually assembles the array (logic and mathematics used in mdadm) but I could only find a lot of forums full of raid disasters and partial (sometimes inaccurate) solutions. Seems to me there's no official structured and more in depth documentation available.

  2. #12
    Join Date
    Nov 2008
    Location
    Atlanta, GA
    Beans
    43
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Missing Superblocks on mdadm Raid5

    *** EDIT *** : using --zero-superblock should not be used in this case. --assume-clean is better for this application. It worked for me, but apparently was a bad idea.


    if fsck is taking that long, you've probably already trashed the filesystem. I know, because i assembled one in the wrong order just yesterday. It was late, and instead of sleeping and dealing with it in the morning, i decided to go ahead and destroy my filesystem. I had a backup.

    It's probably too late for you, but here's what worked for me:

    First, check the existing superblock too see what order the devices really go in:
    Code:
    root@AMD-Media:~# mdadm -E /dev/sd[abd] |egrep 'sd|Chunk|Role|Version'
    /dev/sda:
            Version : 1.2
         Chunk Size : 128K
       Device Role : Active device 2
    /dev/sdb:
            Version : 1.2
         Chunk Size : 128K
       Device Role : Active device 0
    /dev/sdd:
            Version : 1.2
         Chunk Size : 128K
       Device Role : Active device 1
    I'm using whole disks instead of partitions, and they are attached in a weird order, but your info should look similar.
    Note the Device Role. They should be reassembled in the order 0 1 2. If there's a number missing (Let's say yours are numbered 0 1 3) they still just go in ascending order. In my case it's /dev/sdb /dev/sdd /dev/sda

    Next, do something scary:
    Code:
    root@AMD-Media:~# mdadm --stop /dev/md0
    root@AMD-Media:~# mdadm --remove /dev/md0
    root@AMD-Media:~# mdadm --zero-superblock /dev/sd[abd]
    This clears the superblock on each device so that mdadm doesn't give you any problems about assembling devices (partitions, whatever) that have different event numbers, modify dates, etc.

    Next, use the info from the mdadm -E to recreate the array. In my case:
    Code:
    root@AMD-Media:~# mdadm --create /dev/md0 --bitmap=/etc/mdadm/md0.bitmap --chunk=128 --metadata=1.2 --level=5 --raid-devices=3 --force /dev/sdb /dev/sdd /dev/sda
    I had to use force because my bitmap file already existed. Also, don't be tempted to use bracket expansion on the drive order. bracket expansion doesn't respect the order you put things in, and as far as i can tell it alphabetizes them, so
    Code:
    /dev/sd[bda]
    isn't equal to
    Code:
    /dev/sdb /dev/sdd /dev/sda
    I'd love to show you the output from that, but it's rebuilding right now, and i didn't cut and paste it when i did it...

    It does, however come up and mention that there is evidence of an existing filesystem/partition table... If you're 100% SURE that you've got the metadata version, chunk size and order correct then you can allow the array to be created. It will do a resync, but your data should be immediately available to mount and use.

    Again, this is what worked for ME and MY SITUATION! no guarantees!
    Last edited by tealio; March 3rd, 2013 at 08:52 PM.

  3. #13
    Join Date
    Nov 2009
    Location
    Segur De Calafell, Spain
    Beans
    11,907
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Missing Superblocks on mdadm Raid5

    Isn't a --create with --assume-clean a better option to try first? Without zeroing the superblocks. That should force it even if the events counters are not the same. Of course, you still have to use the same original order and use 'missing' for any missing members.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 7 Ultimate 64bit

  4. #14
    Join Date
    Nov 2008
    Location
    Atlanta, GA
    Beans
    43
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Missing Superblocks on mdadm Raid5

    That is absolutely true. My case was a weird one though, because i was trying to upgrade disks.
    I'm going on memory here, but i think i ended up with the 2 original disks and the new disk all having a newer creation date. Then i had one of the old disks fail. So i ended up trying to rebuild an array from 2 good disks with different creation dates, and a third disk that I'm not so sure about... I'm currently trying to rebuild the original array so i can start over. this has been going on since Wednesday.

    If this rebuild fails at 99.9% like the others, then im going to run the array degraded, and put my new disk in a single disk RAID 1. Then I'll have to copy over the data, then mirror that disk, then convert to RAID5 and add the third disk... PITA

    anyway, to answer your question, --assume-clean would probably still have been a better option.

    I don't wanna get too far off here, but if --assume-clean is a better option with similar results to --zero-superblock, then what is the case in which --zero-superblock would be used?
    Is it only for when you're really removing a drive from an array, but it's staying in your system? to keep mdadm from scanning it and trying to add it to an array?

  5. #15
    Join Date
    Nov 2009
    Location
    Segur De Calafell, Spain
    Beans
    11,907
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Missing Superblocks on mdadm Raid5

    Yes, as far as I understand it, you would use --zero-superblock to completely delete any traces of mdadm to use the disk as separate or add it later but as "new" disk, otherwise if it detects a superblock it might try to re-add it as disk that was earlier in the array and mess things for you.

    For assembling existing array you would start with mdadm --assemble --scan, but that would work only if there are no major issues. If the events don't match I think it will fail. The next is --create --assume-clean or --create --run, with correct level and number of devices and their order.

    If all of that fails, you need to look at drastic measures/attempts.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 7 Ultimate 64bit

  6. #16
    Join Date
    Nov 2008
    Location
    Atlanta, GA
    Beans
    43
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Missing Superblocks on mdadm Raid5

    ok... so i just skipped straight to the big leagues... oops...

  7. #17
    Join Date
    Sep 2009
    Beans
    4

    Re: Missing Superblocks on mdadm Raid5

    Hey guys,

    I'm having the identical issue and was hoping you guys could help me troubleshoot some?

    root@server:~# dumpe2fs /dev/md3dumpe2fs 1.41.3 (12-Oct-2008)
    Filesystem volume name: /var/lib/vz
    Last mounted on: <not available>
    Filesystem UUID: e4881189-a1aa-4dbd-9776-8df39c2f6039
    Filesystem magic number: 0xEF53
    Filesystem revision #: 1 (dynamic)
    Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
    Filesystem flags: signed_directory_hash
    Default mount options: (none)
    Filesystem state: clean with errors
    Errors behavior: Continue
    Filesystem OS type: Linux
    Inode count: 179109888
    Block count: 716422752
    Reserved block count: 35821137
    Free blocks: 236978281
    Free inodes: 78393903
    First block: 0
    Block size: 4096
    Fragment size: 4096
    Reserved GDT blocks: 853
    Blocks per group: 32768
    Fragments per group: 32768
    Inodes per group: 8192
    Inode blocks per group: 256
    Filesystem created: Tue Aug 24 18:34:53 2010
    Last mount time: Wed Feb 29 07:37:51 2012
    Last write time: Wed May 8 16:02:53 2013
    Mount count: 4
    Maximum mount count: 24
    Last checked: Thu Feb 16 10:44:25 2012
    Check interval: 15552000 (6 months)
    Next check after: Tue Aug 14 11:44:25 2012
    Reserved blocks uid: 0 (user root)
    Reserved blocks gid: 0 (group root)
    First inode: 11
    Inode size: 128
    Journal inode: 8
    First orphan inode: 45294652
    Default directory hash: half_md4
    Directory Hash Seed: da833f00-139d-4f21-90e2-38830d884e00
    Journal backup: inode blocks
    dumpe2fs: A block group is missing an inode table while reading journal inode


    root@server:~# cat /proc/mdstatPersonalities : [raid1] [raid0] [raid6] [raid5] [raid4]
    md3 : active raid5 sdd3[2] sdc3[1] sda3[0]
    1910460672 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]


    md1 : active raid1 sda1[0] sdb1[1]
    20480896 blocks [4/2] [UU__]


    unused devices: <none>



    root@server:~# mount -a /dev/md3mount: wrong fs type, bad option, bad superblock on /dev/md3,
    missing codepage or helper program, or other error
    In some cases useful info is found in syslog - try
    dmesg | tail or so


  8. #18
    Join Date
    Nov 2009
    Location
    Segur De Calafell, Spain
    Beans
    11,907
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Missing Superblocks on mdadm Raid5

    What identical issue? The OP had the arrays not working, yours are active.

    If /dev/md3 complains about mounting, did you try fsck first?
    Code:
    sudo fsck /dev/md3
    Don't try to mount it before running fsck, it needs to be unmounted.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 7 Ultimate 64bit

  9. #19
    Join Date
    Sep 2009
    Beans
    4

    Re: Missing Superblocks on mdadm Raid5

    Quote Originally Posted by darkod View Post
    What identical issue? The OP had the arrays not working, yours are active.

    If /dev/md3 complains about mounting, did you try fsck first?
    Code:
    sudo fsck /dev/md3
    Don't try to mount it before running fsck, it needs to be unmounted.

    root@londonpower:~# fsck /dev/md3fsck 1.41.3 (12-Oct-2008)
    e2fsck 1.41.3 (12-Oct-2008)
    fsck.ext3: Group descriptors look bad... trying backup blocks...
    fsck.ext3: Bad magic number in super-block while trying to open /dev/md3


    The superblock could not be read or does not describe a correct ext2
    filesystem. If the device is valid and it really contains an ext2
    filesystem (and not swap or ufs or something else), then the superblock
    is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

    that's what I get....

  10. #20
    Join Date
    Nov 2009
    Location
    Segur De Calafell, Spain
    Beans
    11,907
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Missing Superblocks on mdadm Raid5

    What do you have on md3? Can you post:
    Code:
    sudo parted -l
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 7 Ultimate 64bit

Page 2 of 3 FirstFirst 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •