Page 1 of 4 123 ... LastLast
Results 1 to 10 of 33

Thread: RAID failure

  1. #1
    Join Date
    Aug 2010
    Beans
    16

    RAID failure

    Okay, my basic command line knowledge and googling/searching the forums have resulted in no success, so now I ask for the collective wisdom of the forum to help repair my RAID.

    Basic background:

    I am running a media server with 4 HDDs.
    /dev/sda is 1TB drive with the linux install on it
    /dev/sdb, /sdc and /sdd are 2TB drives that were up until about 1 hour ago a raid 5 array for photos, videos and so on.
    This array was built with 3 new HDDs about 2 months ago, and it is run maybe for 4-5 hours every couple of days.

    I was copying some stuff from a portable HDD to the array, started copying fine, but partway through came up with a Read only error. Brought up the command line to check the health of the array and was greated with [U__]

    (Here is my first mistake) I rebooted the computer at this point and from this point on the array of course won't start. (Lesson learnt for next time)

    Got the computer dback up and was greated by this:

    Code:
    sudo mdadm --detail /dev/md127
    /dev/md127:
            Version : 1.2
      Creation Time : Wed Nov 13 16:38:59 2013
         Raid Level : raid5
      Used Dev Size : 1953381888 (1862.89 GiB 2000.26 GB)
       Raid Devices : 3
      Total Devices : 1
        Persistence : Superblock is persistent
    
        Update Time : Sun Dec 22 13:15:33 2013
              State : active, FAILED, Not Started 
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 0
      Spare Devices : 0
    
             Layout : left-symmetric
         Chunk Size : 512K
    
               Name : MediaServer:0  (local to host MediaServer)
               UUID : 8e5ee34c:1a0ff976:b8d63fee:79dcf685
             Events : 122
    
        Number   Major   Minor   RaidDevice State
           0       8       17        0      active sync   /dev/sdb1
           1       0        0        1      removed
           2       0        0        2      removed
    Code:
    sudo mdadm --examine /dev/sdb1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 8e5ee34c:1a0ff976:b8d63fee:79dcf685
               Name : MediaServer:0  (local to host MediaServer)
      Creation Time : Wed Nov 13 16:38:59 2013
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
         Array Size : 3906763776 (3725.78 GiB 4000.53 GB)
      Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : dd41d054:dbe41263:961d1702:91703ade
    
        Update Time : Sun Dec 22 13:15:33 2013
           Checksum : 6649f97 - correct
             Events : 122
    
             Layout : left-symmetric
         Chunk Size : 512K
    
       Device Role : Active device 0
       Array State : A.. ('A' == active, '.' == missing)
    Code:
    sudo mdadm --examine /dev/sdc1
    /dev/sdc1:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 8e5ee34c:1a0ff976:b8d63fee:79dcf685
               Name : MediaServer:0  (local to host MediaServer)
      Creation Time : Wed Nov 13 16:38:59 2013
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
         Array Size : 3906763776 (3725.78 GiB 4000.53 GB)
      Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : active
        Device UUID : bd31d8ac:3ab9d88b:3797e5f7:df0bc6a7
    
        Update Time : Sun Dec 22 13:03:42 2013
           Checksum : 469375dd - correct
             Events : 89
    
             Layout : left-symmetric
         Chunk Size : 512K
    
       Device Role : Active device 1
       Array State : AAA ('A' == active, '.' == missing)
    Code:
    sudo mdadm --examine /dev/sdd1
    /dev/sdd1:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 8e5ee34c:1a0ff976:b8d63fee:79dcf685
               Name : MediaServer:0  (local to host MediaServer)
      Creation Time : Wed Nov 13 16:38:59 2013
         Raid Level : raid5
       Raid Devices : 3
    
     Avail Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
         Array Size : 3906763776 (3725.78 GiB 4000.53 GB)
      Used Dev Size : 3906763776 (1862.89 GiB 2000.26 GB)
        Data Offset : 262144 sectors
       Super Offset : 8 sectors
              State : active
        Device UUID : 8eb5ff9c:6edf5f2c:d2e0271e:3e9d62a5
    
        Update Time : Sun Dec 22 13:06:05 2013
           Checksum : fb1dfb87 - correct
             Events : 116
    
             Layout : left-symmetric
         Chunk Size : 512K
    
       Device Role : Active device 2
       Array State : A.A ('A' == active, '.' == missing)
    fdisk -l gave me this:

    Code:
    Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
    255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk identifier: 0x000f384d
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1   *        2048  1936996351   968497152   83  Linux
    /dev/sda2      1936998398  1953523711     8262657    5  Extended
    Partition 2 does not start on a physical sector boundary.
    /dev/sda5      1936998400  1953523711     8262656   82  Linux swap / Solaris
    
    Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
    255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk identifier: 0x0007fd53
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdd1            2048  3907028991  1953513472   83  Linux
    
    Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
    255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk identifier: 0x000a504c
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdc1            2048  3907028991  1953513472   83  Linux
    
    Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
    255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk identifier: 0x0008dab5
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdb1            2048  3907028991  1953513472   83  Linux
    and running:
    Code:
    sudo mdadm --assemble --scan
    mdadm: /dev/md/0 assembled from 1 drive - not enough to start the array
    Achieved nothing.

    My gut feeling is that the data is still there and that it is unlikely that 2 drives have failed so soon after purchase. But I'm at a loss as to where to go next.

    Your assistance is very much appreciated, because I'm stuck

  2. #2
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: RAID failure

    Judging from the fdisk output, both those drives aren't even being seen by the OS. Have you booted into the BIOS and checked to see if they are being seen?

    Reseat the power and data cables for them and see if they are seen.

    Once you find out if those drives are being seen or not, we can go from there.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  3. #3
    Join Date
    Aug 2010
    Beans
    16

    Re: RAID failure

    Ran up the BIOS,

    Yep definitely getting 1x1TB drive and 3x2TB drives

  4. #4
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: RAID failure

    Boot into the OS again and see if they are being seen.

    Code:
    sudo fdisk -l
    Code:
    sudo blkid -c /dev/null
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  5. #5
    Join Date
    Aug 2010
    Beans
    16

    Re: RAID failure

    As requested,

    sudo fdisk -l

    Code:
    Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
    255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk identifier: 0x000f384d
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sda1   *        2048  1936996351   968497152   83  Linux
    /dev/sda2      1936998398  1953523711     8262657    5  Extended
    Partition 2 does not start on a physical sector boundary.
    /dev/sda5      1936998400  1953523711     8262656   82  Linux swap / Solaris
    
    Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
    255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk identifier: 0x0008dab5
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdb1            2048  3907028991  1953513472   83  Linux
    
    Disk /dev/sdc: 2000.4 GB, 2000398934016 bytes
    255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk identifier: 0x000a504c
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdc1            2048  3907028991  1953513472   83  Linux
    
    Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
    255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
    Units = sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 4096 bytes
    I/O size (minimum/optimal): 4096 bytes / 4096 bytes
    Disk identifier: 0x0007fd53
    
       Device Boot      Start         End      Blocks   Id  System
    /dev/sdd1            2048  3907028991  1953513472   83  Linux
    and of course:

    sudo blkid -c /dev/null

    Code:
    /dev/sda1: UUID="15a6baa5-9692-4f56-8654-e715d4d1d4d4" TYPE="ext4" 
    /dev/sda5: UUID="dea126ee-1593-4fb1-99bf-8a3aad94c1aa" TYPE="swap" 
    /dev/sdb1: UUID="8e5ee34c-1a0f-f976-b8d6-3fee79dcf685" UUID_SUB="dd41d054-dbe4-1263-961d-170291703ade" LABEL="MediaServer:0" TYPE="linux_raid_member" 
    /dev/sdc1: UUID="8e5ee34c-1a0f-f976-b8d6-3fee79dcf685" UUID_SUB="bd31d8ac-3ab9-d88b-3797-e5f7df0bc6a7" LABEL="MediaServer:0" TYPE="linux_raid_member" 
    /dev/sdd1: UUID="8e5ee34c-1a0f-f976-b8d6-3fee79dcf685" UUID_SUB="8eb5ff9c-6edf-5f2c-d2e0-271e3e9d62a5" LABEL="MediaServer:0" TYPE="linux_raid_member"
    thpughts? (I'm also interested in what you saw that made you think it wasn't seeing the HDDs to start with)

  6. #6
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: RAID failure

    The drives weren't listed in fdisk and mdadm showed them as being 'removed.'

    Is that still the case?
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  7. #7
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,133
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: RAID failure

    Have you tried to stop the array and force assemble?
    Code:
    sudo -i
    mdadm --stop /dev/md127
    mdadm --assemble --force /dev/md0 /dev/sd[bcd]1
    Note, this will set the array up as /dev/md0 instead of /dev/md127. What are the contents of your mdadm.conf file?
    Code:
    cat /etc/mdadm/mdadm.conf

  8. #8
    Join Date
    Aug 2010
    Beans
    16

    Re: RAID failure

    Sorry, with the lead up to Christmas I haven't been able to action any of the suggestions. In answer to rubylaser's question, no I haven't tried that. I will make an attempt to do that on Boxing Day, and I will also post the answers to the other two questions. Thankyou for your help thus far everyone.

  9. #9
    Join Date
    Aug 2010
    Beans
    16

    Re: RAID failure

    /etc/mdadm/mdadm.conf

    Code:
    MAILADDR root@localhost
    ARRAY /dev/md/0 metadata=1.2 name=MediaServer:0 UUID=8e5ee34c:1a0ff976:b8d63fee:79dcf685
    sudo mdadm --detail /dev/md127

    Code:
            Version : 1.2
      Creation Time : Wed Nov 13 16:38:59 2013
         Raid Level : raid5
      Used Dev Size : 1953381888 (1862.89 GiB 2000.26 GB)
       Raid Devices : 3
      Total Devices : 1
        Persistence : Superblock is persistent
    
        Update Time : Sun Dec 22 13:15:33 2013
              State : active, FAILED, Not Started 
     Active Devices : 1
    Working Devices : 1
     Failed Devices : 0
      Spare Devices : 0
    
             Layout : left-symmetric
         Chunk Size : 512K
    
               Name : MediaServer:0  (local to host MediaServer)
               UUID : 8e5ee34c:1a0ff976:b8d63fee:79dcf685
             Events : 122
    
        Number   Major   Minor   RaidDevice State
           0       8       17        0      active sync   /dev/sdb1
           1       0        0        1      removed
           2       0        0        2      removed
    I'll try doing the stop and force assemble now, I'll post the results in a moment

  10. #10
    Join Date
    Aug 2010
    Beans
    16

    Re: RAID failure

    doing the force assemble produced this result:

    Code:
    david@MediaServer:~$ sudo mdadm --stop /dev/md127
    mdadm: stopped /dev/md127
    david@MediaServer:~$ sudo mdadm --assemble --force /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1
    mdadm: forcing event count in /dev/sdd1(2) from 116 upto 122
    mdadm: clearing FAULTY flag for device 2 in /dev/md0 for /dev/sdd1
    mdadm: Marking array /dev/md0 as 'clean'
    mdadm: /dev/md0 has been started with 2 drives (out of 3).
    so I take it thats a good thing, now mdadm -- detail shows this:

    Code:
    david@MediaServer:~$ sudo mdadm --detail /dev/md0
    /dev/md0:
            Version : 1.2
      Creation Time : Wed Nov 13 16:38:59 2013
         Raid Level : raid5
         Array Size : 3906763776 (3725.78 GiB 4000.53 GB)
      Used Dev Size : 1953381888 (1862.89 GiB 2000.26 GB)
       Raid Devices : 3
      Total Devices : 2
        Persistence : Superblock is persistent
    
        Update Time : Sun Dec 22 13:15:33 2013
              State : clean, degraded 
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0
    
             Layout : left-symmetric
         Chunk Size : 512K
    
               Name : MediaServer:0  (local to host MediaServer)
               UUID : 8e5ee34c:1a0ff976:b8d63fee:79dcf685
             Events : 122
    
        Number   Major   Minor   RaidDevice State
           0       8       17        0      active sync   /dev/sdb1
           1       0        0        1      removed
           3       8       49        2      active sync   /dev/sdd1
    and for what its worth this is mdadm.conf post the force assemble

    Code:
    MAILADDR root@localhost
    ARRAY /dev/md/0 metadata=1.2 name=MediaServer:0 UUID=8e5ee34c:1a0ff976:b8d63fee:79dcf685
    I've also had the array show up as a mount point, and can access the files, so I assume my next step is to tell the array to assume sdc1 is degraded and rebuild that drive, or do I just force add it?

Page 1 of 4 123 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •