12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

**rubylaser** · December 21st, 2013

Originally Posted by sparticle2000

Is converting down Raid levels possible. e.g. From Raid 6 to Raid 0?

You can reshape back down to RAID5 -> RAID1 from RAID6, but not to RAID0. Here is an older post from Neil Brown, the mdadm developer than explains mdadm's reshaping abilities.

**sparticle2000** · December 21st, 2013

Originally Posted by rubylaser

You can reshape back down to RAID5 -> RAID1 from RAID6, but not to RAID0. Here is an older post from Neil Brown, the mdadm developer than explains mdadm's reshaping abilities.

Reading that post it seems that getting this right to start with is pretty critical. Raid 10 looks like an island with no return! But might still be the right answer for this host. I am still a little confused about the ability/methodology of increasiong the individual drive sizes in a Raid 6/10 array to increase usable space. In this instance we could be starting with a 4x2TB Raid 10 array and ending up with a 4x4TB Raid 10 array. No difference in usable space between Raid 6 & 10 in this host (max 4 drives), but it seems a massive difference in performance and rebuild time in the event of a problem and no reliance on parity. Given that we are only talking about 4 drives max, does your advice of using Raid 6 still stand?

Cheers
Spart

**rubylaser** · December 21st, 2013

Originally Posted by sparticle2000

Reading that post it seems that getting this right to start with is pretty critical. Raid 10 looks like an island with no return! But might still be the right answer for this host. I am still a little confused about the ability/methodology of increasiong the individual drive sizes in a Raid 6/10 array to increase usable space. In this instance we could be starting with a 4x2TB Raid 10 array and ending up with a 4x4TB Raid 10 array. No difference in usable space between Raid 6 & 10 in this host (max 4 drives), but it seems a massive difference in performance and rebuild time in the event of a problem and no reliance on parity. Given that we are only talking about 4 drives max, does your advice of using Raid 6 still stand?

Cheers
Spart

The rebuild time will be much larger between RAID6 and RAID10 due to the lack of parity. But, you can answer your question of RAID6 vs. RAID10 yourself. Is data reliability more important to you or is overall speed? RAID6 always survives 2 disk failures RAID10 does not. RAID6 does have the ability to grow, RAID10 does not. What if in the future, you decide to move this array into a new chassis that will hold more than 4 disks? Personally, for home use, I always use RAID6. It shouldn't have a problem saturating a gigabit connection so for most home use cases, it's practically just as fast as the RAID10 is for it's "usable" speed.

Honestly, for most home users, I've been suggesting a combination of SnapRAID and AUFS over mdadm just because each disk stands alone, and you can still get enough speed to saturate gigabit. In your case, you are already using and invested in mdadm, so I would just go with RAID6, but ultimately, the choice is yours

**sparticle2000** · December 22nd, 2013

Zack,

Thank you for being patient and taking the time to demystify some of the Raid issues we all deal with. You are right, I need to choose. I think on balance reliability is most important as this host will have most of our most precious data. And yes, there is a possibility of building a larger more 'enterprise' class array and consolidating a number of data stores, so the ability to take forward this array would be very useful. I pushed the button on the Raid 6 build a short while ago so we'll see how it goes. I will post back the metrics once it is complete and checked out.

Thanks again.

Spart

**rubylaser** · December 22nd, 2013

Great! I look forward to seeing your results

**sparticle2000** · December 23rd, 2013

This could quite possibly be the worst thing I have ever done! It is still showing 19,000 hrs to complete and is running at c. 1500K/Sec. So we are looking at almost 2 weeks to convert 3TB! Is it maybe the USB drive that I am using for the backup file that is the issue or is it just this slow!

Is there any way to stop the conversion and revert back to Raid 5?

Cheers
Spart

A couple more observations. Processor usage is virtually nothing system is approx 95-97% idle whilst reshaping so doesn't seem to be anything to do with horsepower!

Also, the longer it goes on the slower it seems to get. We are now down to c.1300K/Sec and 24000mins. At this rate of slow down it might never be synced! The backup file seesm to be about 40MB and does not seem to be growing or shrinking.

Is this behaviour normal?

**rubylaser** · December 24th, 2013

No, this is not normal and could be pointing to a failing disk. Have you tried tuning your array, or setting the min_sync_speed options?

**sparticle2000** · December 24th, 2013

Disks are clean in SMART.

Code:

SDA

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   173   172   021    Pre-fail  Always       -       4350
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       18
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       306
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       18
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       9
194 Temperature_Celsius     0x0022   124   111   000    Old_age   Always       -       23
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SDB

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   176   175   021    Pre-fail  Always       -       4175
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       18
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       306
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       18
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       9
194 Temperature_Celsius     0x0022   125   110   000    Old_age   Always       -       22
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged


SDC

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   174   174   021    Pre-fail  Always       -       4258
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       18
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       306
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       18
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       9
194 Temperature_Celsius     0x0022   125   110   000    Old_age   Always       -       22
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SDD

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   176   175   021    Pre-fail  Always       -       4158
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       61
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       7148
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       60
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       35
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       25
194 Temperature_Celsius     0x0022   125   106   000    Old_age   Always       -       22
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

And I check it frequently. The Min and Max sync speed are set high. If you set min to say 200000 and the system is not capable what happens?

I used this post http://www.cyberciti.biz/tips/linux-...ild-speed.html to tune whatever i could whilst the reshape wsa happening. I could not set the bitmap or the stripe_cache.
Is there a way to stop the reshape to Raid 6 and go back to Raid 5?

Code:

Every 10.0s: cat /proc/mdstat                                                   Tue Dec 24 10:00:55 2013

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid6 sdd1[3] sdc1[2] sdb1[1] sda1[0]
      3906763776 blocks super 1.2 level 6, 512k chunk, algorithm 18 [4/3] [UUU_]
      [=>...................]  reshape =  5.4% (105652224/1953381888) finish=22209.5min speed=1386K/sec

unused devices: <none>

$ cat /proc/sys/dev/raid/speed_limit_min
200000
$ cat /proc/sys/dev/raid/speed_limit_max
500000
$ 
Every 10.0s: iostat -k 1 2                                                      Tue Dec 24 10:22:31 2013

Linux 3.8.0-34-generic (TWHGNAS)        24/12/13        _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.43    0.01    0.78    4.97    0.00   92.80

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               3.12       366.22       650.96  330233310  586992337
sdb               3.12       366.62       650.77  330591629  586822697
sdc               3.14       366.59       650.95  330564533  586986861
md0               4.24       341.94      1061.52  308337345  957217468
sdd               0.45         0.19       153.79     168830  138675829
sde               2.07         0.00       238.51       4365  215076524

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    2.01   94.97    0.00    3.02

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              17.00      4096.00      4097.00       4096       4097
sdb              17.00      4096.00      4096.50       4096       4096
sdc              17.00      4096.00      4096.50       4096       4096
md0               0.00         0.00         0.00          0          0
sdd               9.00         0.00      4097.00          0       4097
sde               3.00         0.00        80.00          0         80

As you can see it is gradually grinding slower. started off around 1500K/Sec.

Any help appreciated.

Cheers
Spart

**rubylaser** · December 24th, 2013

I agree, the disks look good. How do you have them connected to your computer (USB, PCI card, to the Motherboard SATA head, etc. )? If you set it higher than it can support, it will go as fast as it can. You "should" be able to safely stop the array, reboot and restore with the backup-file, but I wouldn't consider this without a backup of all of your data first. The reshape is rewriting all of the parity info and the superblock info on the disks, so a backup is really needed first.

Something is still not right though. I have done numerous reshapes over the years and never seen this level of slowness without an underlying issue. For example. I did a reshape recently on a RAID5 array with (8) 1TB disks that had 6TB of data on it. I converted it to a (10) disk RAID6, and the reshape was complete in a less than 2 days to reshape on an old AMD 4400+ X2 with 4GB of RAM.

If it was me, I would buy a 4TB external disk, and rsync the data to it. Then, I would start from scratch with a new RAID array. That way, you can test the build speeds of both RAID10 and RAID6.

**sparticle2000** · December 24th, 2013

Zack,

Thanks for the response. All a bit moot now. I have the benefit of a full backup of the data elsewhere on another data store. It appears that the USB stick was duff and the backup file stopped working and the array stopped working with a system hang I could only reboot. I could not recover the backup file to copy it off.

Anyway started, from scratch with a clean install of 12.04.3 and the array is now building as a fresh array. I am also copying data to it whilst it is syncing but when not copying data it is running at about 100,000K/Sec when copying data it is about 18,000K/Sec

Lets see where I end up overnight when all the data is restored.

I will run the script you pointed to before any reshape/resync in the future.

OAN is there a way to get the mdadm drivers for the array loaded early in the boot as it did not ask me the question when installing it.

Cheers
Spart

Thread: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Thread Tools

Display

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Re: 12.04.3 Server, Grow/Reshape Raid 5 array to Raid 6 array

Bookmarks

Bookmarks

Posting Permissions