Page 3 of 4 FirstFirst 1234 LastLast
Results 21 to 30 of 40

Thread: File Server mdadm raid5 has crashed

  1. #21
    Join Date
    Feb 2008
    Beans
    101

    Re: File Server mdadm raid5 has crashed

    Hmmm, I did this already last night, but skipped the --manage parameter
    Code:
    root@mainframe:~# mdadm --add /dev/md0 /dev/sde
    mdadm: added /dev/sde
    root@mainframe:~# cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
    md0 : active raid5 sde[6] sdb[0] sda[5] sdd[4] sdc[2]
          7813531648 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [U_UUU]
          [>....................]  recovery =  0.0% (445588/1953382912) finish=584.3min speed=55698K/sec
          
    unused devices: <none>
    and then ~6 hours later
    Code:
    root@mainframe:~# cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
    md0 : active raid5 sde[6] sdb[0] sda[5] sdd[4] sdc[2]
          7813531648 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/5] [UUUUU]
          
    unused devices: <none>
    So I notice the indicies of the array still skip slot 1. Is that normal? When a component fails, its number is retired like a sports jersey?

    What was the deal with those counters being off? If that one drive had a weak electrical connection going on and off (multiple times per second), does that explain how the other got out of sync? I guess I'm wondering if I should expect the other drives to fail soon?

    My plan is to buy enough drives to bring this up to raid6, do a full backup, and then a second backup for the 20% that is most important. It won't be cheap but it's money I should have spent already anyway.
    Last edited by apokkalyps; February 3rd, 2013 at 10:47 PM. Reason: typo

  2. #22
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,955
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: File Server mdadm raid5 has crashed

    I'm not sure how the slots go because it's not that only slot 1 was skipped, slot 3 is also skipped.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  3. #23
    Join Date
    Feb 2008
    Beans
    101

    Angry Re: File Server mdadm raid5 has crashed

    So after all that, I think I've managed to lose my files anyway. As soon as I had it back up to RAID5 I went and spent 800 bucks on 12TB of backup plus another 2TB to replace the physically damaged drive. Then, before doing the backup, I tried growing it to a RAID6. I didn't know this would take ~100 hours, but I let it run until my server hung at 15% completion. Seriously hung, like no REISUB, and the reset button didn't work. I rebooted, of course the array won't come up. I tried to do a mdadm --assemble --scan, and it says something about "unable to reshape, maybe you forgot to specify the backup file". I think, oh thank god it forced me to make a backup file, and I go looking for it and it's gone. How did I not know that ubuntu clears the /tmp/ folder at reboot?? I use c:\temp in windows a lot... What a perfect storm of bad decisions, fate cutting me some slack, and then more bad decisions to seal the deal...

    I shut the machine off. Unless anyone knows a magic way to recover the array without the backup file, it seems like my only slim chance is to take the SSD with the os on it to data recovery people. If they can maybe (maybe?) find the backup file on the file system (marked for deletion and not yet overwritten), then I could maybe, maybe recover the array.

    Even if I had the backup file, what are the chances that my array would be recoverable?

  4. #24
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,955
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: File Server mdadm raid5 has crashed

    Wait for rubylaser if he has some ideas.

    Just for info, did you try reshaping the degraded array before having a "full" raid5 with 5 disks and synced?

    Or first you added a new disk to make it a 5 disk raid5 array again, and only then tried the reshape?
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  5. #25
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,955
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: File Server mdadm raid5 has crashed

    Sorry, I just noticed above you did have the array full with all 5 members.

    Did you try checking the current mdadm status with something like:
    sudo mdadm --examine /dev/sd[abcde]
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  6. #26
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,955
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: File Server mdadm raid5 has crashed

    This guy had a similar problem trying to grow 4 disk raid5 to 5 disk raid6. At the end it seems --assemble --force worked.

    But lets see what the examine command above shows first in your case, and wait until rubylaser confirms with his opinion.

    In theory, the reshape should continue after you rebooted, if I got it right you say that it didn't.

    Also, in the thread I linked now ignore most of the situation, I think the guy made en error joining a disk as spare first. You followed rubylaser tutorial for reshaping to raid6, stick to it. I think a forced assmeble might help you but lets wait for another opinion.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  7. #27
    Join Date
    Feb 2008
    Beans
    101

    Re: File Server mdadm raid5 has crashed

    I wonder if it puts the path of the backup file in metadata and then uses that to continue the reshape, but ubuntu cleared the tmp folder on reboot so it couldn't continue. I'm only speculating.

    I did not try a force assemble, I could still try that, it's what fixed me last time. But I was betting that since it FORCED me to make the backup file, that it would be needed. I could try it, but right now I'm keeping the machine off so that there is less chance of the lost backup file getting over written on the file system.

    Maybe if I try to do a force assemble i should do it off a live ISO, if that's possible, rather than my standard OS drive (unless it HAS to be from the same OS drive that started the reshape).

  8. #28
    Join Date
    Nov 2009
    Location
    Mataro, Spain
    Beans
    13,955
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: File Server mdadm raid5 has crashed

    Yeah, that was my idea with the --examine command. Boot into live mode, add the mdadm package (because it's not included), and without trying to assemble md0 or nothing, simply try reading what the superblock says on the first five disks with:
    sudo mdadm --examine /dev/sd[abcde]

    You might have superblock on more disks since you tried growing, but lets see the original first 5 first.

    In my opinion, this should be safe because you are not mounting md0, you are not even trying to assemble it.

    I don't know where the backup file is kept but the the --grow command examples I have seen, you put your own path. And I have never seen it as tmp in any example. Just put it directly in / for example.
    Darko.
    -----------------------------------------------------------------------
    Ubuntu 14.04 LTS 64bit & Windows 10 Pro 64bit

  9. #29
    Join Date
    Feb 2008
    Beans
    101

    Re: File Server mdadm raid5 has crashed

    I stupidly, manually specified /tmp/raid-backup.bak as the path for the backup file. Shooting myself in the foot. I did not realize that in ubuntu, that location is actually cleared on reboot. School of hard knocks.

    I'll do the examine when I get home from work.

  10. #30
    Join Date
    Feb 2008
    Beans
    101

    Re: File Server mdadm raid5 has crashed

    /dev/sdc is the new drive I bought to replace the drive with the molex damage, and was being --grow'n onto for the raid6 reshape.

    Code:
        sudo mdadm --examine /dev/sd[abcdef]
        /dev/sda:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x4
             Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
                   Name : mainframe:vault
          Creation Time : Thu Aug 16 02:57:14 2012
             Raid Level : raid6
           Raid Devices : 6
         
         Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
             Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
          Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : f18da9cc:27f5eee4:61ba900e:dd6ca8b9
         
          Reshape pos'n : 1181333504 (1126.61 GiB 1209.69 GB)
             New Layout : left-symmetric
         
            Update Time : Tue Feb  5 03:05:43 2013
               Checksum : 7ac690a9 - correct
                 Events : 295638
         
                 Layout : left-symmetric-6
             Chunk Size : 512K
         
           Device Role : Active device 4
           Array State : AAAAAA ('A' == active, '.' == missing)
         
         
        /dev/sdb:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x4
             Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
                   Name : mainframe:vault
          Creation Time : Thu Aug 16 02:57:14 2012
             Raid Level : raid6
           Raid Devices : 6
         
         Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
             Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
          Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 004a89c7:bd03e0fe:b6ea3ab9:76e5e5e0
         
          Reshape pos'n : 1181333504 (1126.61 GiB 1209.69 GB)
             New Layout : left-symmetric
         
            Update Time : Tue Feb  5 03:05:43 2013
               Checksum : 617f0438 - correct
                 Events : 295638
         
                 Layout : left-symmetric-6
             Chunk Size : 512K
         
           Device Role : Active device 0
           Array State : AAAAAA ('A' == active, '.' == missing)
         
         
        /dev/sdc:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x6
             Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
                   Name : mainframe:vault
          Creation Time : Thu Aug 16 02:57:14 2012
             Raid Level : raid6
           Raid Devices : 6
         
         Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
             Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
          Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
        Recovery Offset : 590666752 sectors
                  State : clean
            Device UUID : 0d8ddf14:2601f343:0b7e182f:cc8358e9
         
          Reshape pos'n : 1181333504 (1126.61 GiB 1209.69 GB)
             New Layout : left-symmetric
         
            Update Time : Tue Feb  5 03:05:43 2013
               Checksum : 956d5260 - correct
                 Events : 295638
         
                 Layout : left-symmetric-6
             Chunk Size : 512K
         
           Device Role : Active device 5
           Array State : AAAAAA ('A' == active, '.' == missing)
         
         
        /dev/sdd:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x4
             Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
                   Name : mainframe:vault
          Creation Time : Thu Aug 16 02:57:14 2012
             Raid Level : raid6
           Raid Devices : 6
         
         Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
             Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
          Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : a6ad29b7:35b546ae:4bc2a5af:e6bc8252
         
          Reshape pos'n : 1181333504 (1126.61 GiB 1209.69 GB)
             New Layout : left-symmetric
         
            Update Time : Tue Feb  5 03:05:43 2013
               Checksum : 688dc85c - correct
                 Events : 295638
         
                 Layout : left-symmetric-6
             Chunk Size : 512K
         
           Device Role : Active device 2
           Array State : AAAAAA ('A' == active, '.' == missing)
         
         
        /dev/sde:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x4
             Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
                   Name : mainframe:vault
          Creation Time : Thu Aug 16 02:57:14 2012
             Raid Level : raid6
           Raid Devices : 6
         
         Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
             Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
          Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 1df1fd17:592f431a:f3f05592:fbfccdcd
         
          Reshape pos'n : 1181333504 (1126.61 GiB 1209.69 GB)
             New Layout : left-symmetric
         
            Update Time : Tue Feb  5 03:05:43 2013
               Checksum : 9359f4b5 - correct
                 Events : 295638
         
                 Layout : left-symmetric-6
             Chunk Size : 512K
         
           Device Role : Active device 3
           Array State : AAAAAA ('A' == active, '.' == missing)
         
         
        /dev/sdf:
                  Magic : a92b4efc
                Version : 1.2
            Feature Map : 0x4
             Array UUID : 8bc78af0:d9a981e3:73549f21:2f76cd24
                   Name : mainframe:vault
          Creation Time : Thu Aug 16 02:57:14 2012
             Raid Level : raid6
           Raid Devices : 6
         
         Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
             Array Size : 7813531648 (7451.56 GiB 8001.06 GB)
          Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
            Data Offset : 262144 sectors
           Super Offset : 8 sectors
                  State : clean
            Device UUID : 15dcad1e:3808a229:7409b3aa:4e03ae1b
         
          Reshape pos'n : 1181333504 (1126.61 GiB 1209.69 GB)
             New Layout : left-symmetric
         
            Update Time : Tue Feb  5 03:05:43 2013
               Checksum : fa5d762 - correct
                 Events : 295638
         
                 Layout : left-symmetric-6
             Chunk Size : 512K
         
           Device Role : Active device 1
           Array State : AAAAAA ('A' == active, '.' == missing)
    Is it safe to try a mdadm --assemble --force?

Page 3 of 4 FirstFirst 1234 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •