Mdadm issue, unable to re-mount array

[djh82uk]

Hi All

I have a big issue im trying to solve. I have an array (md0) that is currently resyncing, however I cannot mount md0 with the following command:

mount -t ext4 /dev/md0 /Data


It gives:
mount: wrong fs type, bad option, bad superblock on /dev/md0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog - try
dmesg | tail or so



dmesg gives:


[ 722.340720] md0: unknown partition table
[ 722.387711] md: could not open unknown-block(8,161).
[ 722.387759] md: md_import_device returned -16
[ 722.387769] md: could not open unknown-block(8,161).
[ 722.387807] md: md_import_device returned -16
[ 739.259850] EXT4-fs (md0): VFS: Can't find ext4 filesystem
[ 1141.465699] EXT4-fs (md0): VFS: Can't find ext4 filesystem




fsck.ext4 /dev/md0 says that there is no superblock


Trying to recover the backup blocks gives the same error.


Can anyone please help me? I have 12tb of data on this array, the backup server has been down for a few days to an issue also with mdadm (two drives failed).

So now I have no backup due to very unfortunate timing.

Can anyone thing what else I can try?

Thanks

DJH

[djh82uk]

Hmm, managed to get the backup server up and running, but it is having the exact same issue, both are re-syncing but cannot mount

[darkod]

First start by confirming the mdadm status:

cat /proc/mdstat
sudo mdadm --detail /dev/md0

[djh82uk]

Ok so


Cat /proc/mdstat

 cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdk1[8] sdj1[6] sdi1[5] sdh1[4] sdf1[3] sdd1[2] sdc1[1] sdb1[0]
      13674590720 blocks super 1.2 level 5, 512k chunk, algorithm 2 [8/7] [UUUUUUU_]
      [============>........]  recovery = 62.1% (1214521984/1953512960) finish=161.6min speed=76201K/sec

md1 : active raid1 sdg1[2] sde1[0]
      976758841 blocks super 1.2 [2/2] [UU]

unused devices: <none>

sudo mdadm --detail /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Sun May 19 21:15:42 2013
     Raid Level : raid5
     Array Size : 13674590720 (13041.11 GiB 14002.78 GB)
  Used Dev Size : 1953512960 (1863.02 GiB 2000.40 GB)
   Raid Devices : 8
  Total Devices : 8
    Persistence : Superblock is persistent

    Update Time : Mon May 20 01:48:55 2013
          State : clean, degraded, recovering
 Active Devices : 7
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

 Rebuild Status : 62% complete

           Name : fs01:0  (local to host fs01)
           UUID : 5046e2e3:c9d80bcd:87803c89:5d1ec772
         Events : 67

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       81        3      active sync   /dev/sdf1
       4       8      113        4      active sync   /dev/sdh1
       5       8      129        5      active sync   /dev/sdi1
       6       8      145        6      active sync   /dev/sdj1
       8       8      161        7      spare rebuilding   /dev/sdk1

Thanks

DJH

[darkod]

It looks fine, only it's in a process of rebuilding. raid5 array should still mount while rebuilding, but let it finish and see if the fsck does something, and if you can mount it. Try fsck without specifying the filesystem, it will detect any extN on its own.

sudo fsck /dev/md0

In general it looks good on the mdadm side. After it rebuilds you can start checking the filesystem side.

[djh82uk]

Hiya

Array has finished syncing, still won't mount due to the same issue:


FSCK

fsck from util-linux-ng 2.17.2
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device&gt;

[djh82uk]

Trying to repair the superblock does not help either

djh@fs01:~$ sudo e2fsck -b 98304 /dev/md0
e2fsck 1.41.14 (22-Dec-2010)
e2fsck: Bad magic number in super-block while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device&gt;

[darkod]

Are you sure the filesystem was ext2/ext3/ext4?

In any case this looks like filesystem repair now, not like mdadm repair. I don't have much experience with filesystem repairs except running the standard fsck.

[djh82uk]

Yeh it was definatley formated as ext4:


sudo fdisk -l gives

Disk /dev/md1: 1000.2 GB, 1000201053184 bytes
2 heads, 4 sectors/track, 244189710 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/md0: 14002.8 GB, 14002780897280 bytes
2 heads, 4 sectors/track, -876319616 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 3670016 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

It says md0 does not have a valid partition table, but it says the same for md1 which is ext4, has loads of data and is fine.

Thanks

[djh82uk]

Parted correctly identifies md1 as ext4

but for md0 it says :

Error: /dev/md0: unrecognised disk label