Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 23

Thread: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

  1. #11
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,078
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    Quote Originally Posted by sparticle2000 View Post
    Is converting down Raid levels possible. e.g. From Raid 6 to Raid 0?
    You can reshape back down to RAID5 -> RAID1 from RAID6, but not to RAID0. Here is an older post from Neil Brown, the mdadm developer than explains mdadm's reshaping abilities.

  2. #12
    Join Date
    Aug 2012
    Beans
    27

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    Quote Originally Posted by rubylaser View Post
    You can reshape back down to RAID5 -> RAID1 from RAID6, but not to RAID0. Here is an older post from Neil Brown, the mdadm developer than explains mdadm's reshaping abilities.
    Reading that post it seems that getting this right to start with is pretty critical. Raid 10 looks like an island with no return! But might still be the right answer for this host. I am still a little confused about the ability/methodology of increasiong the individual drive sizes in a Raid 6/10 array to increase usable space. In this instance we could be starting with a 4x2TB Raid 10 array and ending up with a 4x4TB Raid 10 array. No difference in usable space between Raid 6 & 10 in this host (max 4 drives), but it seems a massive difference in performance and rebuild time in the event of a problem and no reliance on parity. Given that we are only talking about 4 drives max, does your advice of using Raid 6 still stand?

    Cheers
    Spart

  3. #13
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,078
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    Quote Originally Posted by sparticle2000 View Post
    Reading that post it seems that getting this right to start with is pretty critical. Raid 10 looks like an island with no return! But might still be the right answer for this host. I am still a little confused about the ability/methodology of increasiong the individual drive sizes in a Raid 6/10 array to increase usable space. In this instance we could be starting with a 4x2TB Raid 10 array and ending up with a 4x4TB Raid 10 array. No difference in usable space between Raid 6 & 10 in this host (max 4 drives), but it seems a massive difference in performance and rebuild time in the event of a problem and no reliance on parity. Given that we are only talking about 4 drives max, does your advice of using Raid 6 still stand?

    Cheers
    Spart
    The rebuild time will be much larger between RAID6 and RAID10 due to the lack of parity. But, you can answer your question of RAID6 vs. RAID10 yourself. Is data reliability more important to you or is overall speed? RAID6 always survives 2 disk failures RAID10 does not. RAID6 does have the ability to grow, RAID10 does not. What if in the future, you decide to move this array into a new chassis that will hold more than 4 disks? Personally, for home use, I always use RAID6. It shouldn't have a problem saturating a gigabit connection so for most home use cases, it's practically just as fast as the RAID10 is for it's "usable" speed.

    Honestly, for most home users, I've been suggesting a combination of SnapRAID and AUFS over mdadm just because each disk stands alone, and you can still get enough speed to saturate gigabit. In your case, you are already using and invested in mdadm, so I would just go with RAID6, but ultimately, the choice is yours

  4. #14
    Join Date
    Aug 2012
    Beans
    27

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    Zack,

    Thank you for being patient and taking the time to demystify some of the Raid issues we all deal with. You are right, I need to choose. I think on balance reliability is most important as this host will have most of our most precious data. And yes, there is a possibility of building a larger more 'enterprise' class array and consolidating a number of data stores, so the ability to take forward this array would be very useful. I pushed the button on the Raid 6 build a short while ago so we'll see how it goes. I will post back the metrics once it is complete and checked out.

    Thanks again.

    Spart

  5. #15
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,078
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    Great! I look forward to seeing your results

  6. #16
    Join Date
    Aug 2012
    Beans
    27

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    This could quite possibly be the worst thing I have ever done! It is still showing 19,000 hrs to complete and is running at c. 1500K/Sec. So we are looking at almost 2 weeks to convert 3TB! Is it maybe the USB drive that I am using for the backup file that is the issue or is it just this slow!

    Is there any way to stop the conversion and revert back to Raid 5?

    Cheers
    Spart

    A couple more observations. Processor usage is virtually nothing system is approx 95-97% idle whilst reshaping so doesn't seem to be anything to do with horsepower!

    Also, the longer it goes on the slower it seems to get. We are now down to c.1300K/Sec and 24000mins. At this rate of slow down it might never be synced! The backup file seesm to be about 40MB and does not seem to be growing or shrinking.

    Is this behaviour normal?
    Last edited by sparticle2000; December 23rd, 2013 at 08:57 PM.

  7. #17
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,078
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    No, this is not normal and could be pointing to a failing disk. Have you tried tuning your array, or setting the min_sync_speed options?

  8. #18
    Join Date
    Aug 2012
    Beans
    27

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    Disks are clean in SMART.

    Code:
    SDA
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   173   172   021    Pre-fail  Always       -       4350
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       18
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       306
     10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       18
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       9
    194 Temperature_Celsius     0x0022   124   111   000    Old_age   Always       -       23
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0
    
    SMART Error Log Version: 1
    No Errors Logged
    
    SDB
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   176   175   021    Pre-fail  Always       -       4175
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       18
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       306
     10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       18
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       9
    194 Temperature_Celsius     0x0022   125   110   000    Old_age   Always       -       22
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SDC
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   174   174   021    Pre-fail  Always       -       4258
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       18
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       306
     10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       18
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       9
    194 Temperature_Celsius     0x0022   125   110   000    Old_age   Always       -       22
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0
    
    SMART Error Log Version: 1
    No Errors Logged
    
    SDD
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   176   175   021    Pre-fail  Always       -       4158
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       61
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       7148
     10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       60
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       35
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       25
    194 Temperature_Celsius     0x0022   125   106   000    Old_age   Always       -       22
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0
    
    SMART Error Log Version: 1
    No Errors Logged
    And I check it frequently. The Min and Max sync speed are set high. If you set min to say 200000 and the system is not capable what happens?

    I used this post http://www.cyberciti.biz/tips/linux-...ild-speed.html to tune whatever i could whilst the reshape wsa happening. I could not set the bitmap or the stripe_cache.
    Is there a way to stop the reshape to Raid 6 and go back to Raid 5?

    Code:
    Every 10.0s: cat /proc/mdstat                                                   Tue Dec 24 10:00:55 2013
    
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid6 sdd1[3] sdc1[2] sdb1[1] sda1[0]
          3906763776 blocks super 1.2 level 6, 512k chunk, algorithm 18 [4/3] [UUU_]
          [=>...................]  reshape =  5.4% (105652224/1953381888) finish=22209.5min speed=1386K/sec
    
    unused devices: <none>
    
    $ cat /proc/sys/dev/raid/speed_limit_min
    200000
    $ cat /proc/sys/dev/raid/speed_limit_max
    500000
    $ 
    Every 10.0s: iostat -k 1 2                                                      Tue Dec 24 10:22:31 2013
    
    Linux 3.8.0-34-generic (TWHGNAS)        24/12/13        _x86_64_        (2 CPU)
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               1.43    0.01    0.78    4.97    0.00   92.80
    
    Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
    sda               3.12       366.22       650.96  330233310  586992337
    sdb               3.12       366.62       650.77  330591629  586822697
    sdc               3.14       366.59       650.95  330564533  586986861
    md0               4.24       341.94      1061.52  308337345  957217468
    sdd               0.45         0.19       153.79     168830  138675829
    sde               2.07         0.00       238.51       4365  215076524
    
    avg-cpu:  %user   %nice %system %iowait  %steal   %idle
               0.00    0.00    2.01   94.97    0.00    3.02
    
    Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
    sda              17.00      4096.00      4097.00       4096       4097
    sdb              17.00      4096.00      4096.50       4096       4096
    sdc              17.00      4096.00      4096.50       4096       4096
    md0               0.00         0.00         0.00          0          0
    sdd               9.00         0.00      4097.00          0       4097
    sde               3.00         0.00        80.00          0         80
    As you can see it is gradually grinding slower. started off around 1500K/Sec.

    Any help appreciated.

    Cheers
    Spart
    Last edited by sparticle2000; December 24th, 2013 at 11:23 AM.

  9. #19
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,078
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    I agree, the disks look good. How do you have them connected to your computer (USB, PCI card, to the Motherboard SATA head, etc. )? If you set it higher than it can support, it will go as fast as it can. You "should" be able to safely stop the array, reboot and restore with the backup-file, but I wouldn't consider this without a backup of all of your data first. The reshape is rewriting all of the parity info and the superblock info on the disks, so a backup is really needed first.

    Something is still not right though. I have done numerous reshapes over the years and never seen this level of slowness without an underlying issue. For example. I did a reshape recently on a RAID5 array with (8) 1TB disks that had 6TB of data on it. I converted it to a (10) disk RAID6, and the reshape was complete in a less than 2 days to reshape on an old AMD 4400+ X2 with 4GB of RAM.

    If it was me, I would buy a 4TB external disk, and rsync the data to it. Then, I would start from scratch with a new RAID array. That way, you can test the build speeds of both RAID10 and RAID6.

  10. #20
    Join Date
    Aug 2012
    Beans
    27

    Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

    Zack,

    Thanks for the response. All a bit moot now. I have the benefit of a full backup of the data elsewhere on another data store. It appears that the USB stick was duff and the backup file stopped working and the array stopped working with a system hang I could only reboot. I could not recover the backup file to copy it off.

    Anyway started, from scratch with a clean install of 12.04.3 and the array is now building as a fresh array. I am also copying data to it whilst it is syncing but when not copying data it is running at about 100,000K/Sec when copying data it is about 18,000K/Sec

    Lets see where I end up overnight when all the data is restored.

    I will run the script you pointed to before any reshape/resync in the future.

    OAN is there a way to get the mdadm drivers for the array loaded early in the boot as it did not ask me the question when installing it.

    Cheers
    Spart

Page 2 of 3 FirstFirst 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •