Page 1 of 2 12 LastLast
Results 1 to 10 of 20

Thread: Excruciatingly slow RAID rebuild (MDADM)

  1. #1
    Join Date
    Nov 2007
    Beans
    14

    Excruciatingly slow RAID rebuild (MDADM)

    Long is short:
    I "upgraded" my 6 disk raid6 array by swapping each drive one at a time with a larger one and rebuilding the array. Each rebuild went at about 25-30mb per second and took about 3 hours which wasn't fantastic, but I could live it it.
    I finally took the last magic step and "grew" the raid to the full size after the last drive and its rebuilding again, but this time at only 500kb/s. This causes a rebuild time of 7,000 hours.

    Anyone familiar with MDADM rebuilds give me pointers on how to speed this up? I have the minimum rebuild speed set to 1000k/s and the maximum at 200000k/s but the rebuild still proceeds at exactly 550k/s (with very little deviation, 530-560). The enclosure I use (port multiplier enabled) has LEDs for each device and the activity LEDs are intermittent, blinking in waves. During the original rebuilds these would get pegged solid on.

    I really dont want to have to wait 7000 hours again before this raid is usable (cant really expand the partition yet). Please Help.

  2. #2
    Join Date
    May 2007
    Beans
    2,337

    Re: Excruciatingly slow RAID rebuild (MDADM)

    Hi,

    Try

    sudo echo 100000 > /proc/sys/dev/raid/speed_limit_min
    sudo echo 400000 > /proc/sys/dev/raid/speed_limit_max

    That'll increase the maximum rebuild rate to about 40Mb sec. And increase the min. If anything is using the array (writing to the fs) mdadm slows down the rebuild to the minimum value. Forcing the min to a higher value will speed up the rebuild but slowdown the fs.


    Regards
    Ian Dobson
    Last edited by ian dobson; November 7th, 2010 at 10:41 AM.
    Walking on water and writing software to specification is easy if they're frozen.

    My corner of the internet http://www.planet-ian.com

  3. #3
    Join Date
    Nov 2007
    Beans
    14

    Re: Excruciatingly slow RAID rebuild (MDADM)

    $ sudo echo 100000 > /proc/sys/dev/raid/speed_limit_min
    -bash: /proc/sys/dev/raid/speed_limit_min: Permission denied
    $ cat /proc/sys/dev/raid/speed_limit_min
    1000


    Correction, its now at 6000 minuets not hours, which is a much shorter time. I mounted the array under rebuild and tried to read something, seems to work if I can just deal with a 200kb/s transfer rates (usually get 80+mb/s).
    I suspect that my last drive may have been bad. It was DOA and I RMA'd it and unlike my last WD RMA's I did NOT get a refurb back. I didn't check serial numbers. Its also one of those new "4k" drives while my other drives are not. I'll have to check if my partitioning could be causing this.
    Is it alright to fail a drive during rebuild for a RAID6 array?

  4. #4
    Join Date
    Nov 2007
    Beans
    14

    Re: Excruciatingly slow RAID rebuild (MDADM)

    No idea what the original problem was, but dmesg showed some issues with a drive (soft resets), so I waited for the raid to rebuild (several days) then failed that drive out and plastered it with "dd if=/dev/urandom" several times and then finally /dev/zero once. No more demesg errors, and no smart issues (allocated sectors, etc). Rebuilding now at 25mb/s with no issues yet.
    Chaulk it up to random fluke?

  5. #5
    Join Date
    May 2007
    Beans
    2,337

    Re: Excruciatingly slow RAID rebuild (MDADM)

    Hi,

    soft resets are usually caused by a dodgy cable/connection from my experiance.

    Regards
    Ian Dobson
    Walking on water and writing software to specification is easy if they're frozen.

    My corner of the internet http://www.planet-ian.com

  6. #6
    Join Date
    Jun 2009
    Beans
    196

    Question Re: Excruciatingly slow RAID rebuild (MDADM)

    Can you set the min and max to higher speeds? My software RAID5 reads at 130MB/sec and writes at 95MB/sec. Could I put the min at 40 and the max at 95?

    Is this command something you give while it is rebuilding or does it set these speeds permanently?

    Thanks,

    Coder68

  7. #7
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,134
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: Excruciatingly slow RAID rebuild (MDADM)

    How about turning the internal bitmap on for the rebuild? Bitmaps optimize rebuild time after a crash, or after removing and re-adding a device. Turn it on by typing the following command:
    Code:
    mdadm --grow --bitmap=internal /dev/md0
    Once array rebuild or fully synced, disable bitmaps:
    Code:
    mdadm --grow --bitmap=none /dev/md0
    Also, you give the min/max commands when it's rebuilding, it will reset to defaults after a reboot.

  8. #8
    Join Date
    Jun 2009
    Beans
    196

    Question Re: Excruciatingly slow RAID rebuild (MDADM)

    I have increased the min and max, which sped up the rebuild a little, but not that much. I have a quad core with 8 gigs of RAM so that is not the issue. I did notice that my CPU usage will float around 20-25% for a while and then drop to 0% for a short bit and then go back up. (See attached images.)

    When I try the bitmap trick, I get an error:
    PHP Code:
    sudo mdadm --grow --bitmap=internal /dev/md1
    mdadm
    failed to set internal bitmap
    I googled this but did not find anything that made sense to me.

    Here is my mdstat:

    PHP Code:
    cat /proc/mdstat
    Personalities 
    : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10
    md0 active raid1 sda1[0sdc1[1]
          
    149903296 blocks [2/2] [UU]
          
    md1 active raid5 sde[4sdf[2sdg[3sdd[1sdb[0]
          
    5860543488 blocks super 0.91 level 54k chunkalgorithm 2 [5/5] [UUUUU]
          [===========>.........]  
    reshape 56.2% (1099626508/1953514496finish=561.7min speed=25335K/sec
          
    unused devices
    : <none
    Any ideas why I can't run the bitmap command?

    Thanks,

    C68
    Attached Images Attached Images
    Last edited by Coder68; February 21st, 2011 at 04:23 PM. Reason: spelling

  9. #9
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,134
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: Excruciatingly slow RAID rebuild (MDADM)

    Can you paste in the output of
    Code:
    mdadm -D /dev/md0

  10. #10
    Join Date
    Jun 2009
    Beans
    196

    Re: Excruciatingly slow RAID rebuild (MDADM)

    It is my md1 that I am having issues with. Here is what you ask for.

    PHP Code:
    /dev/md1:
            
    Version 00.91
      Creation Time 
    Sat Feb 19 22:07:42 2011
         Raid Level 
    raid5
         
    Array Size 5860543488 (5589.05 GiB 6001.20 GB)
      
    Used Dev Size 1953514496 (1863.02 GiB 2000.40 GB)
       
    Raid Devices 5
      Total Devices 
    5
    Preferred Minor 
    1
        Persistence 
    Superblock is persistent

        Update Time 
    Mon Feb 21 13:50:42 2011
              State 
    cleanrecovering
     Active Devices 
    5
    Working Devices 
    5
     Failed Devices 
    0
      Spare Devices 
    0

             Layout 
    left-symmetric
         Chunk Size 
    4K

     Reshape Status 
    71complete
      Delta Devices 
    1, (4->5)

               
    UUID e509485d:97ef7ead:d14ee5ec:eb91e22c (local to host us104)
             
    Events 0.13548

        Number   Major   Minor   RaidDevice State
           0       8       16        0      active sync   
    /dev/sdb
           1       8       48        1      active sync   
    /dev/sdd
           2       8       80        2      active sync   
    /dev/sdf
           3       8       96        3      active sync   
    /dev/sdg
           4       8       64        4      active sync   
    /dev/sde 
    Thanks,

    C68

    Additional:
    It suddenly sped up... not sure why.

    PHP Code:
    md1 active raid5 sde[4sdf[2sdg[3sdd[1sdb[0]
          
    5860543488 blocks super 0.91 level 54k chunkalgorithm 2 [5/5] [UUUUU]
          [=================>...]  
    reshape 87.2% (1704715672/1953514496finish=51.8min speed=79979K/sec
          
    unused devices
    : <none
    I did not do anything. I did notice that md1_reshape is now using a lot more CPU then before.
    Last edited by Coder68; February 21st, 2011 at 09:49 PM.

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •