Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data.

PumpAction · 11:10 AM

I write this guide because i found a lot of info on the subject but still i had some problems because a lot of the info was very out-of-context it dealt with very specific problems and no clue if I had/should do anything before or after. With this guide I try to cover the whole process, thats why I've included a plan over what I have to to before I actually comes with the specific commands to type in terminal. This guide got some bloat that you might not need, to skip it; search this page for "blabla" and take it from there. For more information about what those commands are suppose to do read the bloat:

I make this guide while experimenting to get the process right for creating and expanding a raid 5 array.
My situation is: I got three 320gb sata disks which is empty, and another three which is full, I want to combine all six sata disks in one raid 5 array.
I got no way to back it all up and format all disk and set up a 6 disk raid-5 from scratch, so if my idea was to set up a raid5 with 3 disks move over some data and, format the now empty disk and expand the array to cover the new disk. Then repeat the process until its all six disks in one array. This is of course not critical data that, that why I don't got backups. Loosing it is only inconvenient, I am aware of this SO SHOULD YOU BE IF YOU FOLLOW THIS GUIDE. IF DATA ARE IMPORTANT HAVE BACKUPS AND DONT BLAME ME IF YOU LOOSE DATA. I'm no Ubuntu/Linux/RAID expert, this guide is more of a collection of other guides aimed at solving one specific set of problems. I write this guide while testing everything i do on six small partitions spread over the three empty disks, it is my advise that you to "try-it-before-you-buy-it".

In advance I have made some preparations which i won't cover in this guide. This is:
Prepare some files with a checksum (sfv). which i use as a quick check to see if my files are ok after expansion.
I also know my partitions very well, almost to the point that i could point them out on the platters

Also my system is up and running with a working internet connection and has access to apt-get

This howto will try to cover the whole process of setting up a raid5 array and expanding/growing it on a system already up and running. This guide will NOT show you how to use a softraid5 array as basis for your operating system or anything like that. This simply shows you how to take a bunch of disks turn them into a raid5 and how to expand it later without loosing data.

[repeatmyself]
I write this guide because i found a lot of info on the subject but still i had some problems because a lot of the info was very out-of-context it dealt with very specific problems and no clue if I had/should do anything before or after. With this guide I try to cover the whole process, thats why I've included a plan over what I have to to before I actually comes with the specific commands to type in terminal.
[/repeatmyself]

What I've got:
Three full 320gb harddisk
Three empty 320gb harddisk
Ubuntu Feisty Fawn (7.04) 32-bit installed on some other completely separate disk.
Access to gparted, cksfv, mdadm either already installed or via apt-get install and a working internet connection

The todo list human mode:
Create raid5 with 3 empty disks/partitions
Copy data to raid5 array
Expand raid5 array and its filesystem with one (or more) disk/partition

This simple plan is a bit more complicated for the computer. Here is it's version:

The todo list computer mode:
----Make softraid section----
Prepare partitions to be used in array and set raid flag, format and set raid flag
Create raid array with mdadm
Format (make filesystem) raid array
Mount the brand new and flashingly good looking supercool new ready to be used raid 5 array
Copy files

----Expand section----
Unmount the brand new and flashingly good looking supercool new ready to be used raid 5 array
Prepare and the new partition to be added
Put the new partition under softraiddriver control
Tell driver to grow on to the new disk
Check filesystem integrity
Expand filesystem so you get more space to play with
mount the expanded array
Check the integrety of your files

The terminal commands, no more blabla:
This is where it get serious, these are the commands I've used. MAKE SURE YOU CHANGE EVERYTHING THAT NEEDS TO BE CHANGED like device names and path, number of raid devices and such. tore is my username, md0 is the raid-array I use, level=5 is raid5 vs. for example raid1 which is level=1, raid-devices=3, the number of partitions dedicated for this specific array. /home/tore/raid is where i mount my array.
REMEMBER THIS GUIDE COME WITH NO WARRANTY, IF YOU LOOSE DATA DON'T BLAME ME. Try-it-before-you-buy-it if you can.

EDIT: I've edited the guide, I split the guide into three sections. (Prepare partition - make raid5 and format it - expand array & filesystem)

This guide have been tested with success on Ubuntu Feisty Fawn (7.04) 32-bit.

If you test it on another system, tell me and i add a line above.

---Prepare the disk/partition---

What preparations needs to be done?
First you must know which partition/disk to use. I got no specific way of helping you here, best tips is to use gparted to get a graphic view of disks and partitions.

gparted can be installed by:

Code:

sudo apt-get install gparted

To run gparted type:

Code:

sudo gparted

When you have sorted out which partitions/disks to use there are four things to do:
1.remove it from fstab
2.format it to ext3
3.set raid flag
4.unmount partitions

When you have figured out which partitions to use you can proceed to removing it from fstab. I assume you know how to do this, but a quick refresher if you don't remember the details:

Code:

Back up your fstab:

sudo cp /etc/fstab /etc/fstab.backup

edit your original fstab with gedit:

sudo gedit /etc/fstab

put a # in front of the line containing the partition you want to use. Save&Exit

Now its time for format the disks/partition to be used in the array. This is also a very straight forward kind of operation. Just run gparted again.

Code:

sudo gparted

Since gparted is graphical I cant give you a command line guide. But if you could find your way in fstab you should be able to navigate you way to your partitions in gparted with ease. Once there just right click it and format to ext3. Remember to be sure to pick the right partition, if you pick the wrong one there is no going back.
After the partition has been formated right click it and “manage flags”, then hook-on the raid flag. Click "OK" & Exit.

The last step, make sure to unmount the partitions to be used in the array. Now they should be out of the fstab, this could be done via a reboot or:

Code:

sudo umount name_of_device

eksample:

sudo umount /dev/sda1

When this is done on all disks/partitions that are suppose to be a part of the initial array you are finished with preparing the disks/partitions.

----Make softraid section----

Install mdadm (softraid driver/application):

Code:

sudo apt-get install mdadm

(awnser a few questions, i choose: none - yes)

Create raid device with partitions you prepared in gparted(modify this line for correct number of raid-devices and make sure to use the right devices):

Code:

sudo mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1

The prosess of creating the array takes time, monitor it and do not continue this guide until its done(type it over again to update):

Code:

cat /proc/mdstat

Make a ext3 filesystem that spans the array:

Code:

sudo mkfs.ext3 /dev/md0

Mount the array (remember to choose were):

Code:

sudo mount -t ext3 /dev/md0 /home/tore/raid

Take ownership (replace tore with your username and correct the path)

Code:

sudo chown -R tore:tore /home/tore/raid

[optional]
The copy some files with valid sfv and check to see if everything is ok, and to have a refrence point after expansion.

Code:

cd /home/tore/raid

Code:

cksfv -r -q

[/optional]

----Expand section----
Unmount the array, i don't know if this is necessary, but doing so means less stuff to go wrong.

Code:

sudo umount /dev/md0

Then you need to go and prepare the disk(s) to be added. (The prepare section of the guide)

Then tell mdadm that it can play with it:

Code:

sudo mdadm --add /dev/md0 /dev/sda2

Then tell mdadm to stop playing and use it to expand/grow your raid5 array(make sure that the number of raid devices is correct, it should be the number of disks/partitions the grown/expanded array will have):

Code:

sudo mdadm --grow /dev/md0 --raid-devices=4

View and wait for the new array to reshape:

Code:

cat /proc/mdstat

Check the filesystem (required to resize it):

Code:

sudo e2fsck -f /dev/md0

Resize the filesystem on the new array so that you can use it for something useful.

Code:

sudo resize2fs /dev/md0

Mount the expanded/grown array with it's resized filesystem(remember to change the mount point):

Code:

sudo mount -t ext3 /dev/md0 /home/tore/raid

[optional]
Go check your files:

Code:

cd /home/tore/raid

Code:

cksfv -r -q

[/optional]
_____________________
Need to expand more? Repeat the expand section. Remember to change number of raid devices and choose the correct device to add. Also, you can add more then one disk at the time, I jump strait from 3 to 6.
_____________________
Was this not the right guide for you? Try one of these (my guide is composed from some of these):

http://scotgate.org/?p=107
http://www.linuxhomenetworking.com/w..._Software_RAID
http://tldp.org/HOWTO/Software-RAID-HOWTO.html
http://ubuntuforums.org/showthread.php?t=408461
http://www.howtoforge.com/linux_resi...xt3_partitions
http://linux.about.com/library/cmd/b..._resize2fs.htm
http://www.mail-archive.com/linux-ra.../msg08539.html
http://ubuntuforums.org/showthread.p...ighlight=mdadm
http://gd.tuwien.ac.at/linuxcommand....es/mdadm8.html

EDIT1: Fix a buch of typos.
EDIT2: Split the guide into three sections, added some info on fstab, fix more typos.

tuckie

Just wanted to say thanks for the writeup, the bulk of it I knew, but its nice to have it all in one place. I'm going to be using this on Monday.

tuckie

Quick question, anyone know if it is necessary to umount the raid volume before growing? I'd rather not have to go through 7 hours of downtime.

tuckie

well im growing it without umounting the volume. Wish me luck. Only 13 hours remaining!

matthodge

How did it go tuckie? RAID up and running without unmounting?

tuckie

well if you looked at my other posts you'd get a good idea

Essentally, growing the raid went fine, but I had to unmount it to grow the ext3 partition. Pretty much 5 hours after everything was up and working, one of the drives dropped from the array and somehow the partition got corrupted in the process (2tb of data gone!).

--Live and learn and always have backups.

ubdime · 03:32 PM

I've done this twice. From 4 to 5 disk raid5, then 5 to 6. It was online and mounted the entire time. Here is my log from 5 to 6.

The mdadm --grow took about 10-12 hours. And the resize2fs took a bit under 30 minutes. While it's growing, you can use watch -n3 cat /proc/mdstat to watch its progress. And while it's doing the resize2fs, you can use watch -n3 df to watch. Also I should mention that while it was doing all this, I even downloaded a few gigs worth of stuff and was heavily reading from the raid the entire time (seeding 500+ torrents with about 5-10 of them active at any given point and watching shows/movies from it).

root@lanfear:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid5 sdd1[0] sde1[4] sdb1[3] sdc1[2] sda1[1]
1953535744 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

md0 : active raid1 sdg1[0] sdh1[1]
243167744 blocks [2/2] [UU]

unused devices: <none>

root@lanfear:~# mdadm --add /dev/md1 /dev/sdf1
mdadm: added /dev/sdf1

root@lanfear:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid5 sdf1[5](S) sdd1[0] sde1[4] sdb1[3] sdc1[2] sda1[1]
1953535744 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]

md0 : active raid1 sdg1[0] sdh1[1]
243167744 blocks [2/2] [UU]

unused devices: <none>

root@lanfear:~# mdadm --detail /dev/md1
/dev/md1:
Version : 00.90
Creation Time : Fri Oct 31 23:10:19 2008
Raid Level : raid5
Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 1
Persistence : Superblock is persistent

Update Time : Sat Nov 8 16:49:59 2008
State : clean
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 64K

UUID : f6e524b8:284a0034:0f20f41d:3c651377 (local to host lanfear)
Events : 0.324790

Number Major Minor RaidDevice State
0 8 49 0 active sync /dev/sdd1
1 8 1 1 active sync /dev/sda1
2 8 33 2 active sync /dev/sdc1
3 8 17 3 active sync /dev/sdb1
4 8 65 4 active sync /dev/sde1

5 8 81 - spare /dev/sdf1

root@lanfear:~# mdadm --grow /dev/md1 --raid-devices=6
mdadm: Need to backup 1280K of critical section..
mdadm: ... critical section passed.

root@lanfear:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid5 sdf1[5] sdd1[0] sde1[4] sdb1[3] sdc1[2] sda1[1]
1953535744 blocks super 0.91 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
[>....................] reshape = 0.0% (81280/488383936) finish=800.8min speed=10160K/sec

md0 : active raid1 sdg1[0] sdh1[1]
243167744 blocks [2/2] [UU]

unused devices: <none>

root@lanfear:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid5 sdf1[5] sdd1[0] sde1[4] sdb1[3] sdc1[2] sda1[1]
1953535744 blocks super 0.91 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
[===================>.] reshape = 99.9% (488369920/488383936) finish=0.0min speed=19503K/sec

md0 : active raid1 sdg1[0] sdh1[1]
243167744 blocks [2/2] [UU]

unused devices: <none>

root@lanfear:~# resize2fs /dev/md1
resize2fs 1.41.3 (12-Oct-2008)
Filesystem at /dev/md1 is mounted on /media/save; on-line resizing required
old desc_blocks = 117, new_desc_blocks = 146
Performing an on-line resize of /dev/md1 to 610479920 (4k) blocks.
The filesystem on /dev/md1 is now 610479920 blocks long.

root@lanfear:~# df | awk /md1/
/dev/md1 2403601996 1117790744 1285811252 47% /media/save

zorrek

I've done this twice. Both times, 4 disk RAID5 to 7 Disk RAID5, all 500 Gig drives.
Both times took just over 4 days to grow the array. Both times the arrays were on-line. Neither array was actually used while it was grown.
ext3 took 20 minutes to resize, again while on-line but not used.

roob85

just wanted to say thanks for this great how-to. I used it to set up my first raid5 setup a few months back and im about to use it again to grow my array!

Fredrik_b · 02:33 PM

greate article, but I cant manage to execute the command "sudo resize2fs /dev/md0" without gettign the error message:

Code:

[root@localhost ~]# sudo resize2fs /dev/md0
resize2fs 1.40.8 (13-Mar-2008)
resize2fs: Device or resource busy while trying to open /dev/md0
Couldn't find valid filesystem superblock.

August 4th, 2007	#1
PumpAction First Cup of Ubuntu Join Date: Aug 2006 Beans: 3	Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. I write this guide because i found a lot of info on the subject but still i had some problems because a lot of the info was very out-of-context it dealt with very specific problems and no clue if I had/should do anything before or after. With this guide I try to cover the whole process, thats why I've included a plan over what I have to to before I actually comes with the specific commands to type in terminal. This guide got some bloat that you might not need, to skip it; search this page for "blabla" and take it from there. For more information about what those commands are suppose to do read the bloat: I make this guide while experimenting to get the process right for creating and expanding a raid 5 array. My situation is: I got three 320gb sata disks which is empty, and another three which is full, I want to combine all six sata disks in one raid 5 array. I got no way to back it all up and format all disk and set up a 6 disk raid-5 from scratch, so if my idea was to set up a raid5 with 3 disks move over some data and, format the now empty disk and expand the array to cover the new disk. Then repeat the process until its all six disks in one array. This is of course not critical data that, that why I don't got backups. Loosing it is only inconvenient, I am aware of this SO SHOULD YOU BE IF YOU FOLLOW THIS GUIDE. IF DATA ARE IMPORTANT HAVE BACKUPS AND DONT BLAME ME IF YOU LOOSE DATA. I'm no Ubuntu/Linux/RAID expert, this guide is more of a collection of other guides aimed at solving one specific set of problems. I write this guide while testing everything i do on six small partitions spread over the three empty disks, it is my advise that you to "try-it-before-you-buy-it". In advance I have made some preparations which i won't cover in this guide. This is: Prepare some files with a checksum (sfv). which i use as a quick check to see if my files are ok after expansion. I also know my partitions very well, almost to the point that i could point them out on the platters Also my system is up and running with a working internet connection and has access to apt-get This howto will try to cover the whole process of setting up a raid5 array and expanding/growing it on a system already up and running. This guide will NOT show you how to use a softraid5 array as basis for your operating system or anything like that. This simply shows you how to take a bunch of disks turn them into a raid5 and how to expand it later without loosing data. [repeatmyself] I write this guide because i found a lot of info on the subject but still i had some problems because a lot of the info was very out-of-context it dealt with very specific problems and no clue if I had/should do anything before or after. With this guide I try to cover the whole process, thats why I've included a plan over what I have to to before I actually comes with the specific commands to type in terminal. [/repeatmyself] What I've got: Three full 320gb harddisk Three empty 320gb harddisk Ubuntu Feisty Fawn (7.04) 32-bit installed on some other completely separate disk. Access to gparted, cksfv, mdadm either already installed or via apt-get install and a working internet connection The todo list human mode: Create raid5 with 3 empty disks/partitions Copy data to raid5 array Expand raid5 array and its filesystem with one (or more) disk/partition This simple plan is a bit more complicated for the computer. Here is it's version: The todo list computer mode: ----Make softraid section---- Prepare partitions to be used in array and set raid flag, format and set raid flag Create raid array with mdadm Format (make filesystem) raid array Mount the brand new and flashingly good looking supercool new ready to be used raid 5 array Copy files ----Expand section---- Unmount the brand new and flashingly good looking supercool new ready to be used raid 5 array Prepare and the new partition to be added Put the new partition under softraiddriver control Tell driver to grow on to the new disk Check filesystem integrity Expand filesystem so you get more space to play with mount the expanded array Check the integrety of your files The terminal commands, no more blabla: This is where it get serious, these are the commands I've used. MAKE SURE YOU CHANGE EVERYTHING THAT NEEDS TO BE CHANGED like device names and path, number of raid devices and such. tore is my username, md0 is the raid-array I use, level=5 is raid5 vs. for example raid1 which is level=1, raid-devices=3, the number of partitions dedicated for this specific array. /home/tore/raid is where i mount my array. REMEMBER THIS GUIDE COME WITH NO WARRANTY, IF YOU LOOSE DATA DON'T BLAME ME. Try-it-before-you-buy-it if you can. EDIT: I've edited the guide, I split the guide into three sections. (Prepare partition - make raid5 and format it - expand array & filesystem) This guide have been tested with success on Ubuntu Feisty Fawn (7.04) 32-bit. If you test it on another system, tell me and i add a line above. ---Prepare the disk/partition--- What preparations needs to be done? First you must know which partition/disk to use. I got no specific way of helping you here, best tips is to use gparted to get a graphic view of disks and partitions. gparted can be installed by: Code: sudo apt-get install gparted To run gparted type: Code: sudo gparted When you have sorted out which partitions/disks to use there are four things to do: 1.remove it from fstab 2.format it to ext3 3.set raid flag 4.unmount partitions When you have figured out which partitions to use you can proceed to removing it from fstab. I assume you know how to do this, but a quick refresher if you don't remember the details: Code: Back up your fstab: sudo cp /etc/fstab /etc/fstab.backup edit your original fstab with gedit: sudo gedit /etc/fstab put a # in front of the line containing the partition you want to use. Save&Exit Now its time for format the disks/partition to be used in the array. This is also a very straight forward kind of operation. Just run gparted again. Code: sudo gparted Since gparted is graphical I cant give you a command line guide. But if you could find your way in fstab you should be able to navigate you way to your partitions in gparted with ease. Once there just right click it and format to ext3. Remember to be sure to pick the right partition, if you pick the wrong one there is no going back. After the partition has been formated right click it and “manage flags”, then hook-on the raid flag. Click "OK" & Exit. The last step, make sure to unmount the partitions to be used in the array. Now they should be out of the fstab, this could be done via a reboot or: Code: sudo umount name_of_device eksample: sudo umount /dev/sda1 When this is done on all disks/partitions that are suppose to be a part of the initial array you are finished with preparing the disks/partitions. ----Make softraid section---- Install mdadm (softraid driver/application): Code: sudo apt-get install mdadm (awnser a few questions, i choose: none - yes) Create raid device with partitions you prepared in gparted(modify this line for correct number of raid-devices and make sure to use the right devices): Code: sudo mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1 The prosess of creating the array takes time, monitor it and do not continue this guide until its done(type it over again to update): Code: cat /proc/mdstat Make a ext3 filesystem that spans the array: Code: sudo mkfs.ext3 /dev/md0 Mount the array (remember to choose were): Code: sudo mount -t ext3 /dev/md0 /home/tore/raid Take ownership (replace tore with your username and correct the path) Code: sudo chown -R tore:tore /home/tore/raid [optional] The copy some files with valid sfv and check to see if everything is ok, and to have a refrence point after expansion. Code: cd /home/tore/raid Code: cksfv -r -q [/optional] ----Expand section---- Unmount the array, i don't know if this is necessary, but doing so means less stuff to go wrong. Code: sudo umount /dev/md0 Then you need to go and prepare the disk(s) to be added. (The prepare section of the guide) Then tell mdadm that it can play with it: Code: sudo mdadm --add /dev/md0 /dev/sda2 Then tell mdadm to stop playing and use it to expand/grow your raid5 array(make sure that the number of raid devices is correct, it should be the number of disks/partitions the grown/expanded array will have): Code: sudo mdadm --grow /dev/md0 --raid-devices=4 View and wait for the new array to reshape: Code: cat /proc/mdstat Check the filesystem (required to resize it): Code: sudo e2fsck -f /dev/md0 Resize the filesystem on the new array so that you can use it for something useful. Code: sudo resize2fs /dev/md0 Mount the expanded/grown array with it's resized filesystem(remember to change the mount point): Code: sudo mount -t ext3 /dev/md0 /home/tore/raid [optional] Go check your files: Code: cd /home/tore/raid Code: cksfv -r -q [/optional] _____________________ Need to expand more? Repeat the expand section. Remember to change number of raid devices and choose the correct device to add. Also, you can add more then one disk at the time, I jump strait from 3 to 6. _____________________ Was this not the right guide for you? Try one of these (my guide is composed from some of these): http://scotgate.org/?p=107 http://www.linuxhomenetworking.com/w..._Software_RAID http://tldp.org/HOWTO/Software-RAID-HOWTO.html http://ubuntuforums.org/showthread.php?t=408461 http://www.howtoforge.com/linux_resi...xt3_partitions http://linux.about.com/library/cmd/b..._resize2fs.htm http://www.mail-archive.com/linux-ra.../msg08539.html http://ubuntuforums.org/showthread.p...ighlight=mdadm http://gd.tuwien.ac.at/linuxcommand....es/mdadm8.html EDIT1: Fix a buch of typos. EDIT2: Split the guide into three sections, added some info on fstab, fix more typos. Last edited by PumpAction; August 6th, 2007 at 11:10 AM..

September 21st, 2007	#2
tuckie Just Give Me the Beans! Join Date: Aug 2006 Beans: 53	Re: Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. Just wanted to say thanks for the writeup, the bulk of it I knew, but its nice to have it all in one place. I'm going to be using this on Monday.

September 23rd, 2007	#3
tuckie Just Give Me the Beans! Join Date: Aug 2006 Beans: 53	Re: Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. Quick question, anyone know if it is necessary to umount the raid volume before growing? I'd rather not have to go through 7 hours of downtime.

September 24th, 2007	#4
tuckie Just Give Me the Beans! Join Date: Aug 2006 Beans: 53	Re: Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. well im growing it without umounting the volume. Wish me luck. Only 13 hours remaining!

October 8th, 2007	#5
matthodge First Cup of Ubuntu Join Date: Oct 2007 Beans: 12	Re: Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. How did it go tuckie? RAID up and running without unmounting?

November 9th, 2008	#7
ubdime 5 Cups of Ubuntu Join Date: Nov 2008 Beans: 17	Re: Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. I've done this twice. From 4 to 5 disk raid5, then 5 to 6. It was online and mounted the entire time. Here is my log from 5 to 6. The mdadm --grow took about 10-12 hours. And the resize2fs took a bit under 30 minutes. While it's growing, you can use watch -n3 cat /proc/mdstat to watch its progress. And while it's doing the resize2fs, you can use watch -n3 df to watch. Also I should mention that while it was doing all this, I even downloaded a few gigs worth of stuff and was heavily reading from the raid the entire time (seeding 500+ torrents with about 5-10 of them active at any given point and watching shows/movies from it). root@lanfear:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid5 sdd1[0] sde1[4] sdb1[3] sdc1[2] sda1[1] 1953535744 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU] md0 : active raid1 sdg1[0] sdh1[1] 243167744 blocks [2/2] [UU] unused devices: <none> root@lanfear:~# mdadm --add /dev/md1 /dev/sdf1 mdadm: added /dev/sdf1 root@lanfear:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid5 sdf1[5](S) sdd1[0] sde1[4] sdb1[3] sdc1[2] sda1[1] 1953535744 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU] md0 : active raid1 sdg1[0] sdh1[1] 243167744 blocks [2/2] [UU] unused devices: <none> root@lanfear:~# mdadm --detail /dev/md1 /dev/md1: Version : 00.90 Creation Time : Fri Oct 31 23:10:19 2008 Raid Level : raid5 Array Size : 1953535744 (1863.04 GiB 2000.42 GB) Used Dev Size : 488383936 (465.76 GiB 500.11 GB) Raid Devices : 5 Total Devices : 6 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Sat Nov 8 16:49:59 2008 State : clean Active Devices : 5 Working Devices : 6 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 64K UUID : f6e524b8:284a0034:0f20f41d:3c651377 (local to host lanfear) Events : 0.324790 Number Major Minor RaidDevice State 0 8 49 0 active sync /dev/sdd1 1 8 1 1 active sync /dev/sda1 2 8 33 2 active sync /dev/sdc1 3 8 17 3 active sync /dev/sdb1 4 8 65 4 active sync /dev/sde1 5 8 81 - spare /dev/sdf1 root@lanfear:~# mdadm --grow /dev/md1 --raid-devices=6 mdadm: Need to backup 1280K of critical section.. mdadm: ... critical section passed. root@lanfear:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid5 sdf1[5] sdd1[0] sde1[4] sdb1[3] sdc1[2] sda1[1] 1953535744 blocks super 0.91 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU] [>....................] reshape = 0.0% (81280/488383936) finish=800.8min speed=10160K/sec md0 : active raid1 sdg1[0] sdh1[1] 243167744 blocks [2/2] [UU] unused devices: <none> root@lanfear:~# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md1 : active raid5 sdf1[5] sdd1[0] sde1[4] sdb1[3] sdc1[2] sda1[1] 1953535744 blocks super 0.91 level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU] [===================>.] reshape = 99.9% (488369920/488383936) finish=0.0min speed=19503K/sec md0 : active raid1 sdg1[0] sdh1[1] 243167744 blocks [2/2] [UU] unused devices: <none> root@lanfear:~# resize2fs /dev/md1 resize2fs 1.41.3 (12-Oct-2008) Filesystem at /dev/md1 is mounted on /media/save; on-line resizing required old desc_blocks = 117, new_desc_blocks = 146 Performing an on-line resize of /dev/md1 to 610479920 (4k) blocks. The filesystem on /dev/md1 is now 610479920 blocks long. root@lanfear:~# df \| awk /md1/ /dev/md1 2403601996 1117790744 1285811252 47% /media/save Last edited by ubdime; November 9th, 2008 at 03:32 PM..

December 12th, 2008	#8
zorrek First Cup of Ubuntu Join Date: Feb 2008 Beans: 6 Ubuntu 8.04 Hardy Heron	Re: Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. I've done this twice. Both times, 4 disk RAID5 to 7 Disk RAID5, all 500 Gig drives. Both times took just over 4 days to grow the array. Both times the arrays were on-line. Neither array was actually used while it was grown. ext3 took 20 minutes to resize, again while on-line but not used.

June 30th, 2009	#9
roob85 First Cup of Ubuntu Join Date: Aug 2006 Beans: 10	Re: Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. just wanted to say thanks for this great how-to. I used it to set up my first raid5 setup a few months back and im about to use it again to grow my array!

June 30th, 2010	#10
Fredrik_b Just Give Me the Beans! Join Date: Aug 2006 Beans: 52	Re: Howto: Create RAID5 and Expand/Grow it using mdadm without loosing data. greate article, but I cant manage to execute the command "sudo resize2fs /dev/md0" without gettign the error message: Code: [root@localhost ~]# sudo resize2fs /dev/md0 resize2fs 1.40.8 (13-Mar-2008) resize2fs: Device or resource busy while trying to open /dev/md0 Couldn't find valid filesystem superblock. Last edited by Fredrik_b; June 30th, 2010 at 02:33 PM..

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode