Page 1 of 3 123 LastLast
Results 1 to 10 of 24

Thread: When creating a RAID5 array, mdadm will automatically create a degraded array

  1. #1
    Join Date
    Dec 2005
    Location
    USA
    Beans
    134
    Distro
    Ubuntu Development Release

    When creating a RAID5 array, mdadm will automatically create a degraded array

    So i have created a 7 device raid5 device with
    Code:
    mdadm --create /dev/md0 --level=5 --raid-devices=7 /dev/sd[b-h]1
    From the mdadm man page.
    When creating a RAID5 array, mdadm will automatically create a degraded array with an extra spare drive. This is because build‐ing the spare into a degraded array is in general faster than resyncing the parity on a non-degraded, but not clean, array. This feature can be overridden with the --force option.
    Code:
    mdadm --detail /dev/md0 
    /dev/md0:
            Version : 1.2
      Creation Time : Tue Jul  9 20:27:36 2013
         Raid Level : raid5
         Array Size : 5859781632 (5588.32 GiB 6000.42 GB)
      Used Dev Size : 976630272 (931.39 GiB 1000.07 GB)
       Raid Devices : 7
      Total Devices : 7
        Persistence : Superblock is persistent
    
        Update Time : Tue Jul  9 20:27:36 2013
              State : clean, degraded 
     Active Devices : 6
    Working Devices : 7
     Failed Devices : 0
      Spare Devices : 1
    
             Layout : left-symmetric
         Chunk Size : 512K
    
               Name : dalserver:0  (local to host dalserver)
               UUID : cd798370:e2a8d9ff:8b3a2154:454b57be
             Events : 0
    
        Number   Major   Minor   RaidDevice State
           0       8       17        0      active sync   /dev/sdb1
           1       8       33        1      active sync   /dev/sdc1
           2       8       49        2      active sync   /dev/sdd1
           3       8       65        3      active sync   /dev/sde1
           4       8       81        4      active sync   /dev/sdf1
           5       8       97        5      active sync   /dev/sdg1
           6       0        0        6      removed
    
           7       8      113        -      spare   /dev/sdh1
    How about that. The man page is correct, it created a degraded raid for me. So now do i "build the spare into a degraded array"?

  2. #2
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,133
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    This is not normal behavior after a sync. You shouldn't have a spare device after the initial sync is complete unless something went wrong, or you passed in the --spare-devices=1 with your --create action (which you didn't).Your array status should say clean, resyncing until the sync is complete, and then just say clean when the sync is done.

    What does /dev/sdh1 show in it's metadata?
    Code:
    mdadm -E /dev/sdh1
    What does this show?
    Code:
    cat /etc/mdadm/mdadm.conf
    finally, what does the SMART info look like on that disk?
    Code:
    sudo apt-get install smartmontools
    smartctl -a /dev/sdh
    EXAMPLE
    Here is a smaller example I just built in Virtualbox in Ubuntu 12.04 Server and mdadm 3.2.5 to show you what it should do.

    Create a 3 disk RAID5 array.
    Code:
    root@mdadm-test:~# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1
    mdadm: layout defaults to left-symmetric
    mdadm: layout defaults to left-symmetric
    mdadm: chunk size defaults to 512K
    mdadm: size set to 1045504K
    mdadm: Defaulting to version 1.2 metadata
    mdadm: array /dev/md0 started.
    Watch until the array is completed syncing
    Code:
    root@mdadm-test:~# watch cat /proc/mdstat
    Look at the completed array.
    Code:
    root@mdadm-test:~# mdadm --detail /dev/md0
    /dev/md0:
            Version : 1.2
      Creation Time : Wed Jul 10 08:10:00 2013
         Raid Level : raid5
         Array Size : 2091008 (2042.34 MiB 2141.19 MB)
      Used Dev Size : 1045504 (1021.17 MiB 1070.60 MB)
       Raid Devices : 3
      Total Devices : 3
        Persistence : Superblock is persistent
    
    
        Update Time : Wed Jul 10 08:10:06 2013
              State : clean 
     Active Devices : 3
    Working Devices : 3
     Failed Devices : 0
      Spare Devices : 0
    
    
             Layout : left-symmetric
         Chunk Size : 512K
    
    
               Name : mdadm-test:0  (local to host mdadm-test)
               UUID : 06166bb5:400e9825:43d6ea10:34921304
             Events : 18
    
    
        Number   Major   Minor   RaidDevice State
           0       8       17        0      active sync   /dev/sdb1
           1       8       33        1      active sync   /dev/sdc1
           3       8       49        2      active sync   /dev/sdd1
    Check the metadata on an individual disk
    Code:
    root@mdadm-test:~# mdadm -E /dev/sdb1
    /dev/sdb1:
              Magic : a92b4efc
            Version : 1.2
        Feature Map : 0x0
         Array UUID : 06166bb5:400e9825:43d6ea10:34921304
               Name : mdadm-test:0  (local to host mdadm-test)
      Creation Time : Wed Jul 10 08:10:00 2013
         Raid Level : raid5
       Raid Devices : 3
    
    
     Avail Dev Size : 2092032 (1021.67 MiB 1071.12 MB)
         Array Size : 2091008 (2042.34 MiB 2141.19 MB)
      Used Dev Size : 2091008 (1021.17 MiB 1070.60 MB)
        Data Offset : 1024 sectors
       Super Offset : 8 sectors
              State : clean
        Device UUID : 6d46c227:9a210f35:f71a445d:561ae516
    
    
        Update Time : Wed Jul 10 08:10:06 2013
           Checksum : 7f7f9f9 - correct
             Events : 18
    
    
             Layout : left-symmetric
         Chunk Size : 512K
    
    
       Device Role : Active device 0
       Array State : AAA ('A' == active, '.' == missing)
    <sidenote>7 disks is a bit too many, if you value your data, to put into a RAID5 array imho. After 6 disks, I would really suggest moving to a RAID6, or looking at something like SnapRAID if this is just a large storage volume for movies, pictures, docs, and music at home.</sidenote>
    Last edited by rubylaser; July 10th, 2013 at 01:33 PM.

  3. #3
    Join Date
    Jun 2006
    Beans
    97

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    Quick Question. I have a 5 disk mdadm array (2 tb green drives)

    I started w/3 drives during the OS Server install on a USB Stick. Then I manually added 2 more drives (1 at a time) and grew the array. All is well but my question is this. when I do "sudo mdadm --detail /dev/md0" it says /dev/sda1, /dev/sdb1, /dev/sdd1 (dunno why it skipped C during server OS install), /dev/sde (note missing the 1), /dev/def (note again missing the 1)

    Is it ok for sde & sdf to be missing the "1" did I miss a step? I have ran ext4.fsck each time after the 20 hours grow, and no errors. everything seems to be working fine.

    My next trick is moving to a better case (using an old chieftec full tower case I had) it's getting full. cause I have 2 more 2tb drives to install and making the transition to raid6 on the next "grow".

  4. #4
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,133
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    Quote Originally Posted by grunge09 View Post
    Quick Question. I have a 5 disk mdadm array (2 tb green drives)

    I started w/3 drives during the OS Server install on a USB Stick. Then I manually added 2 more drives (1 at a time) and grew the array. All is well but my question is this. when I do "sudo mdadm --detail /dev/md0" it says /dev/sda1, /dev/sdb1, /dev/sdd1 (dunno why it skipped C during server OS install), /dev/sde (note missing the 1), /dev/def (note again missing the 1)

    Is it ok for sde & sdf to be missing the "1" did I miss a step? I have ran ext4.fsck each time after the 20 hours grow, and no errors. everything seems to be working fine.

    My next trick is moving to a better case (using an old chieftec full tower case I had) it's getting full. cause I have 2 more 2tb drives to install and making the transition to raid6 on the next "grow".
    Yes, this is fine. You just used the whole disk instead of partitions for the last two disks. I like to use partitions, so that I can ensure that all my disks are the same size, and I won't have a disk that is slightly smaller than the rest that won't work in my array. All that being said, a mix of partitions and whole disks will work perfectly fine.

    Make sure when you do the grow to RAID6 that you use the --backup-file option to backup the critical portion at the beginning of the grow and that you write it to a different disk.

  5. #5
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    Quote Originally Posted by rubylaser View Post
    <sidenote>7 disks is a bit too many, if you value your data, to put into a RAID5 array imho. After 6 disks, I would really suggest moving to a RAID6, or looking at something like SnapRAID if this is just a large storage volume for movies, pictures, docs, and music at home.</sidenote>
    Not to derail the thread too much, but I'm planning on running mdadm with a 3 disk RAID5 for a backup of my main server. Would SnapRAID be better for that?

    Thanks for the tutorial, btw, that should help greatly.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  6. #6
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,133
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    Quote Originally Posted by CharlesA View Post
    Not to derail the thread too much, but I'm planning on running mdadm with a 3 disk RAID5 for a backup of my main server. Would SnapRAID be better for that?

    Thanks for the tutorial, btw, that should help greatly.
    Charles, in your case you will be backing up virtual machines that are changing constantly, SnapRAID would not be the best solution. SnapRAID is really designed for home media servers that have big files that don't change constantly, things like: movies, tv shows, pictures, documents, etc.

    Frankly, for a backup location if you don't intend to add disks in the future, I would use ZFS (if you will expand it disk-by-disk, then mdadm is a no brainer over ZFS). That way you can keep versioned snapshots on your backup volume, and have the bit rot protection that ZFS offers. mdadm + rsnapshot will work well in this scenario too.
    Last edited by rubylaser; July 11th, 2013 at 01:16 AM.

  7. #7
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    Quote Originally Posted by rubylaser View Post
    Charles, in your case you will be backing up virtual machines that are changing constantly, SnapRAID would not be the best solution. SnapRAID is really designed for home media servers that have big files that don't change constantly, things like: movies, tv shows, pictures, documents, etc.

    Frankly, for a backup location if you don't intend to add disks in the future, I would use ZFS (if you will expand it disk-by-disk, then mdadm is a no brainer over ZFS). That way you can keep versioned snapshots on your backup volume, and have the bit rot protection that ZFS offers. mdadm will also work well in this scenario too.
    I'm not really sure what I am going to due with the backup array at this point. The general idea was to have the main server synced to the backup server weekly, so I don't lose data in the event the one of the backup drives goes down.

    I'll probably run with mdadm in that case, especially if I will need to expand the array in the future.

    Sidenote: The backup server is a Dual core pentium with 8GB RAM, so it should be able to handle anything I throw at it, especially over Gigabit ethernet.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  8. #8
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,133
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    Quote Originally Posted by CharlesA View Post
    I'm not really sure what I am going to due with the backup array at this point. The general idea was to have the main server synced to the backup server weekly, so I don't lose data in the event the one of the backup drives goes down.

    I'll probably run with mdadm in that case, especially if I will need to expand the array in the future.

    Sidenote: The backup server is a Dual core pentium with 8GB RAM, so it should be able to handle anything I throw at it, especially over Gigabit ethernet.
    Yeah, that is way overkill for mdadm or SnapRAID and would work fine even with ZFS Are you going to script it to etherwake your backup box, rsync, then shut it down after successful backup + email, or are you just going to leave the backup box on 24/7?

  9. #9
    Join Date
    Oct 2009
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    Quote Originally Posted by rubylaser View Post
    Yeah, that is way overkill for mdadm or SnapRAID and would work fine even with ZFS Are you going to script it to etherwake your backup box, rsync, then shut it down after successful backup + email, or are you just going to leave the backup box on 24/7?
    I'm thinking of running a script from my main box to wake the backup machine up, wait for it to come up all the way and then rsync it. Not quite sure how I'm going to shut it down after the backup, but I don't really want it to be running 24/7 cuz it's sitting at the foot of my bed at the moment

    I think I need a dedicated space for my servers, but such is life in an apartment.

    EDIT: @OP: With 7 drives I would recommend going with RAID6. I just recently moved from RAID5 to RAID 6 and had two of my drives blow up in my face - one was stuck rebuilding and the other got dropped out of the array. So glad I wasn't running RAID 5 as I would have lost all of the data on the array. I did have everything backed up, so I would not have lost data, but it would have been a chore to copy all the data back to the array from backups.
    Last edited by CharlesA; July 11th, 2013 at 01:55 AM. Reason: added advice to OP.
    Come to #ubuntuforums! We have cookies! | Basic Ubuntu Security Guide

    Tomorrow's an illusion and yesterday's a dream, today is a solution...

  10. #10
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,133
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: When creating a RAID5 array, mdadm will automatically create a degraded array

    Quote Originally Posted by CharlesA View Post
    I'm thinking of running a script from my main box to wake the backup machine up, wait for it to come up all the way and then rsync it. Not quite sure how I'm going to shut it down after the backup, but I don't really want it to be running 24/7 cuz it's sitting at the foot of my bed at the moment

    I think I need a dedicated space for my servers, but such is life in an apartment.
    All you would need to do is write a simple BASH script from your regular fileserver to etherwake the backup-box by MAC address, ping it until it responds, then test SSH to make sure that it's available, rsync from your fileserver to the backup-box (setup keyless SSH), once the rsync is done, duse ssh to the remote box to shut it down (ssh root@backup-box 'shutdown -h now'). This way you can schedule via cron to do the backup when you are off at work, or not trying to sleep, etc.

Page 1 of 3 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •