Page 4 of 4 FirstFirst ... 234
Results 31 to 40 of 40

Thread: File Server mdadm raid5 has crashed

  1. #31
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,610
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: File Server mdadm raid5 has crashed

    I would say yes. The superblock already says it's raid6, all members are present and the event counters are the same on all disks.

    If you want, wait for confirmation from rubylaser, I'll PM him now and depending on time zone and free time, he might pop in.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  2. #32
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,133
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: File Server mdadm raid5 has crashed

    I would do a force assemble at this point. It appears that the metadata has been updated, so this should work.

  3. #33
    Join Date
    Feb 2008
    Beans
    101

    Re: File Server mdadm raid5 has crashed

    Can I do the force assemble on a live ISO?

  4. #34
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    372
    Distro
    Ubuntu 12.04 Precise Pangolin

  5. #35
    Join Date
    Feb 2008
    Beans
    101

    Re: File Server mdadm raid5 has crashed

    No dice.

    Code:
    ubuntu@ubuntu:~$ sudo mdadm --assemble --force --run --verbose /dev/md0 /dev/sd[abcdef]
    mdadm: looking for devices for /dev/md0
    mdadm: /dev/sda is identified as a member of /dev/md0, slot 4.
    mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
    mdadm: /dev/sdc is identified as a member of /dev/md0, slot 5.
    mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2.
    mdadm: /dev/sde is identified as a member of /dev/md0, slot 3.
    mdadm: /dev/sdf is identified as a member of /dev/md0, slot 1.
    mdadm:/dev/md0 has an active reshape - checking if critical section needs to be restored
    mdadm: Failed to find backup of critical section
    mdadm: Failed to restore critical section for reshape, sorry.
          Possibly you needed to specify the --backup-file
    So I talked to a data recovery company. To retrieve the backup file from my SSD, they charge $100 to even look at the drive and then $500-$700 if they can restore the file.
    Hypothetically, if the file is restored perfectly, what are the chances of the assemble succeeding with it? 90%? 50? 30?

  6. #36
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    372
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: File Server mdadm raid5 has crashed

    Personally I don't believe in recovery, at least after my experiences.

    But of course you want to do all you can to get the backup-file back. Note that the more you use the drive the /tmp is at, the more chance the file will be overwritten by something else.

    http://askubuntu.com/questions/21760...-files-on-ext4

  7. #37
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,610
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: File Server mdadm raid5 has crashed

    Are you sure the /tmp is overwritten? Is it on the same array? If you have / on the same array, then /tmp is not accessible until the array is assembled, which is a catch 22.

    It seems it's best to have the backup file on another disk, but only if you have one.

    I'm not sure what's best to do right now. If they can get the backup file back, the reshape might be able to continue. But on the other hand, it depends whether they will get the file 100% back and whether they will destroy any other data/settings on the disk while doing it.

    Having the backup file back helps nothing if they mess up the stopped reshape. You want the backup file for reshape continuation.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  8. #38
    Join Date
    Feb 2008
    Beans
    101

    Re: File Server mdadm raid5 has crashed

    I keep my OS and / mounted on a separate drive besides the array. So the filesystem /tmp was on, does not need to be assembled, just scoured for deleted data. After the initial crash/reboot I tried a few mdadm commands, attempted an assemble, ls'ed the /tmp folder, and then shut the machine down immediately and unplugged the OS drive. So I've done my best to increase the chances of recovery from the instant I knew the file was missing.

    Yeah, apparently ubuntu clears the /tmp folder each time you reboot by default. This was news to me. Bad bad news.

    http://ubuntu-tutorials.com/2008/01/...nup-frequency/

  9. #39
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    372
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: File Server mdadm raid5 has crashed

    I would try the ext4 undelete from the link I gave you, read into it yourself some more. If you fail I think the recovery company will fail also.

  10. #40
    Join Date
    Feb 2008
    Beans
    101

    Re: File Server mdadm raid5 has crashed

    This epic saga continues. First, thanks everyone that helped me in this thread, I would have been useless without all these hints. Now I have recovered most of the more important part of my data, but would like some advice on if I can recover any more.

    I was able to get the backup file recovered and re-assembled the array with the --invalid-backup param in case parts of it had been corrupted. I was able to get my array back up and running for a while. First I took one external drive and copied of the smallest and most immediatly important stuff. At this point I had what was originally a synced raid5, which was growing/reshaping to a raid6 onto a new drive. As it was happeneing atleast one of the drives was still making the beeping noises periodically and I knew it was a matter of time before it went out. Eventually /dev/sdd totally failed and got marked F on /proc/mdstat. The array was still operational, still showed 4 of 6 components active, where the fifth still being resynced. After getting the most important smallest stuff copied to one location, I started doing an rsync -avH of the entire array contents to another array. This was going while it was reshaping, since I didn't have faith the array would live long enough for the 5 day reshape to complete, which would have gotten me up to 5/6 components of the raid 6, at which point I would have added another drive for 6/6. The reason I didn't immediatly add the extra drive is because I really felt this array and its drives were a timebomb and I was more immediatly interested in copying the data off than trying to get the full raid 6 running (again if it was going to take 10 days and I expected another drive to fail within a few).

    Anyway, quite a bit of my data got rsynced off, this morning I woke up to find out another drive (/dev/sda) in the array had failed as I expected. I am really confused about what kind of state it is in, and I am hoping for some advice about what is going on and if it's finally time to give up and take my baby off lifesupport.

    First:
    Code:
    cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
    md0 : active raid6 sdb[0] sdc[7] sda[5](F) sde[4] sdd[2](F) sdf[6]
          7813531648 blocks super 1.2 level 6, 512k chunk, algorithm 18 [6/3] [UU_U__]
          
    unused devices: <none>
    So judging from the above, I would expect the array to be totally offline/broken. 3 out of 6 drives in a raid6? But my array is still "active" and I still have it mounted, can still see the folder structure and many files, just the certain folders give I/O errors when you try to ls them. How is this possible? Why do I have some files but not the others? Is it wrong to expect all or nothing of a filesystem on md0 (ext4)?

    My theory: The array had been 40% through the reshape onto the new drive (even with 4/5 of the 'old' components), then 1 more of the old components died, so I am at 3/5 of the old components PLUS a new drive with 40% of the stripes, so right now in essence I have 40% of the 4th drive needed for minimum raid6 operation, and thus can see 40% of the files. I don't know if mdadm is capable of that kind of magic, but I can't otherwise explain how my array is even assembled right now. Can anyone tell me if this is indeed possible/accurate? If it is something like this, why doen't mdstat atleast say something other than "active" like "degraded" etc?

    The real question. Right now Disk Utility says /dev/sda is "green" but with 1 bad sector (I don't really know what this means except it's bad). What I'm asking is what everyone is always asking in these threads, is there any way I can bring my array back to full functionality?

    Code:
    sudo mdadm --examine /dev/sd[a-f]
    /dev/sda:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x4
         Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
               Name : mainframe:vault  (local to host mainframe)
      Creation Time : Wed Aug 15 21:57:14 2012
         Raid Level : raid6
       Raid Devices : 6
    
     Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
         Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
      Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : active
        Device UUID : f18da9cc:27f5eee4:61ba900e:dd6ca8b9
    
      Reshape pos'n : 3277520896 (3125.69 GiB 3356.18 GB)
         New Layout : left-symmetric
    
        Update Time : Sun Apr 21 06:27:31 2013
           Checksum : 75147b14 - correct
             Events : 755496
    
             Layout : left-symmetric-6
         Chunk Size : 512K
    
       Device Role : Active device 4
       Array State : AA.AAA ('A' == active, '.' == missing)
    /dev/sdb:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x4
         Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
               Name : mainframe:vault  (local to host mainframe)
      Creation Time : Wed Aug 15 21:57:14 2012
         Raid Level : raid6
       Raid Devices : 6
    
     Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
         Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
      Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 004a89c7:bd03e0fe:b6ea3ab9:76e5e5e0
    
      Reshape pos'n : 3277520896 (3125.69 GiB 3356.18 GB)
         New Layout : left-symmetric
    
        Update Time : Sun Apr 21 13:41:12 2013
           Checksum : 5bc7638b - correct
             Events : 759402
    
             Layout : left-symmetric-6
         Chunk Size : 512K
    
       Device Role : Active device 0
       Array State : AA.A.A ('A' == active, '.' == missing)
    /dev/sdc:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x6
         Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
               Name : mainframe:vault  (local to host mainframe)
      Creation Time : Wed Aug 15 21:57:14 2012
         Raid Level : raid6
       Raid Devices : 6
    
     Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
         Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
      Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
    Recovery Offset : 1638760448 sectors
              State : clean
        Device UUID : 0d8ddf14:2601f343:0b7e182f:cc8358e9
    
      Reshape pos'n : 3277520896 (3125.69 GiB 3356.18 GB)
         New Layout : left-symmetric
    
        Update Time : Sun Apr 21 13:41:12 2013
           Checksum : ce2e55b3 - correct
             Events : 759402
    
             Layout : left-symmetric-6
         Chunk Size : 512K
    
       Device Role : Active device 5
       Array State : AA.A.A ('A' == active, '.' == missing)
    /dev/sde:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x4
         Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
               Name : mainframe:vault  (local to host mainframe)
      Creation Time : Wed Aug 15 21:57:14 2012
         Raid Level : raid6
       Raid Devices : 6
    
     Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
         Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
      Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 1df1fd17:592f431a:f3f05592:fbfccdcd
    
      Reshape pos'n : 3277520896 (3125.69 GiB 3356.18 GB)
         New Layout : left-symmetric
    
        Update Time : Sun Apr 21 13:41:12 2013
           Checksum : 8da25408 - correct
             Events : 759402
    
             Layout : left-symmetric-6
         Chunk Size : 512K
    
       Device Role : Active device 3
       Array State : AA.A.A ('A' == active, '.' == missing)
    /dev/sdf:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x4
         Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
               Name : mainframe:vault  (local to host mainframe)
      Creation Time : Wed Aug 15 21:57:14 2012
         Raid Level : raid6
       Raid Devices : 6
    
     Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
         Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
      Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 15dcad1e:3808a229:7409b3aa:4e03ae1b
    
      Reshape pos'n : 3277520896 (3125.69 GiB 3356.18 GB)
         New Layout : left-symmetric
    
        Update Time : Sun Apr 21 13:41:12 2013
           Checksum : 9ee36b5 - correct
             Events : 759402
    
             Layout : left-symmetric-6
         Chunk Size : 512K
    
       Device Role : Active device 1
       Array State : AA.A.A ('A' == active, '.' == missing)

Page 4 of 4 FirstFirst ... 234

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •