HOWTO: Linux Software Raid using mdadm

[kragen]

1) Introduction:

Recently I went out and bought myself a second hard drive with the purpose of setting myself up a performance raid (raid0). It took me days and a lot of messing about to get sorted, but once I figured out what I was doing I realised that it's actually relatively simple, so I've written this guide to share my experiences I went for raid0, because I'm not too worried about loosing data, but if you wanted to set up a raid 1, raid 5 or any other raid type then a lot of the information here would still apply.

2) 'Fake' raid vs Software raid:

When I bought my motherboard, (The ASRock ConRoeXFire-eSATA2), one of the big selling points was an on board raid, however some research revealed that rather than being a true hardware raid controller, this was in fact more than likely what is know as 'fake' raid. I think wikipedia explains it quite well: http://en.wikipedia.org/wiki/Redunda...ependent_disks

Hybrid RAID implementations have become very popular with the introduction of inexpensive RAID controllers, implemented using a standard disk controller and BIOS (software) extensions to provide the RAID functionality. The operating system requires specialized RAID device drivers that present the array as a single block based logical disk. Since these controllers actually do all calculations in software, not hardware, they are often called "fakeraids", and have almost all the disadvantages of both hardware and software RAID.

After realising this, I spent some time trying to get this fake raid to work - the problem is that although the motherboard came with drivers that let windows see my two 250 GB drives as one large 500 GB raid array, Ubuntu just saw the two separate drives and ignored the 'fake' raid completely. There are ways to get this fake raid working under linux, however if you are presented with this situation then my advice to you is to abandon the onboard raid controller and go for software raid instead. I've seen arguments as to why software raid is faster and more flexible, but I think the best reason is that software raid is far easier to set up!

3) The Basics of Linux Software Raid:

For the basics of raids try looking on Wikipedia again: http://en.wikipedia.org/wiki/Redunda...ependent_disks. I don't want to discuss it myself because its been explained many times before by people who are far more qualified to explain it than I am. I will however go over a few things about software raids:

Linux software raid is more flexible than hardware raid or true raid because rather than forming a raid arrays between identical disks, the raid arrays are created between identical partitions. As far as I understand, if you are using hardware raid between (for example) two disks, then you can either create a raid 1 array between those disks, or a raid 0 array. Using software raid however, you could create two sets of identical partitions on the disks, and for a raid 0 array between two of those partitions, and a raid 1 array between the other two. If you wanted to you could probably even create a raid array between two partitions on the same disk! (not that you would want to!)

The process of setting up the a raid array is simple:

1. Create two identical partitions
2. Tell the software what the name of the new raid array is going to be, what partitions we are going to use, and what type of array we are creating (raid 0, raid 1 etc...)

Once we have created this array, we then format and mount it in a similar way to the way we would format a partition on a physical disk.

4) Which Live CD to use:

You want to download and burn the alternate install Ubuntu cd of your choosing, for example, I used:

ubuntu-6.10-alternate-amd64.iso

If you boot up the ubuntu desktop live CD and need to access your raid, then you will need to install mdadm if you want to access any software raid arrays:

sudo apt-get update
sudo apt-get install mdadm

Don't worry too much about this for now - you will only need this if you ever use the Ubuntu desktop cd to fix your installation, the alternate install cd has the mdadm tools installed already.

5) Finally, lets get on with it!

Boot up the installer
Boot up the alternate install CD and run through the text based installation until you reach the partitioner, and select "Partition Manually".

Create the partitions you need for each raid array
You now need to create the partitions which you will (in the next step) turn into software raid arrays. I recommend using the space at the start, or if your disks are identical, the end of your disks. That way once you've set one disk up, you can just enter exactly the same details for the second disk. The partitioner should be straightforward enough to use - when you create a partition which you intend to use in a raid, you need to change the type to "Linux RAID Autodetect".

How you partition your installation is upto you, however there are a few things to bear in mind:

1. If (like me) you are going for a performance raid, then you will need to create a separate /boot partition, otherwise grub wont be able to boot - it doesn't have the drivers needed to access raid 0 arrays. It sounds simple, but it took me so long to figure out.
2. If, on the other hand, you are doing a server installation (for example) using raid 1 / 5 and the goal is reliability, then you probably want the computer to be able to boot up even if one of the disks is down. In this situation you need to do something different with the /boot partition again. I'm not sure how it works myself, as I've never used raid 1, but you can find some more information in the links at the end of this guide. Perhaps I'll have a play around and add this to the guide later on, for completeness sake.
3. If you are looking for performance, then there isn't a whole load of point creating a raid array for swap space. The kernel can manage multiple swap spaces by itself (we will come onto that later).
4. Again, if you are looking for reliability however, then you may want to build a raid partition for your swap space, to prevent crashes should one of your drives fail. Again, look for more information in the links at the end.

On my two identical 250 GB drives, I created two 1 GB swap partitions, two +150 GB partitions (to become a raid0 array fro my /home space), and two +40 GB partitions (to become a raid 0 array for my root space), all inside an extended partition at the end of my drives. I then also created a small 500 MB partition on the first drive, which would become my /boot space. I left the rest of the space on my drives for ntfs partitions.

Assemble the partitions as raid devices
Once you've created your partitions, select the "Configure software raid" option. The changes to the partition table will be written to the disk, and you will be allowed to create and delete raid devices - to create a raid device, simply select "create", select the type of raid array you want to create, and select the partitions you want to use. Remember to check which partition numbers you are going to use in which raid arrays - if you forget, hit <Esc> a few times to bring you back to the partition editor screen where you can see whats going on.

Tell the installer how to use the raid devices
Once you are done, hit finish - you will be taken back to the partitioner where you should see some new raid devices listed. Configure these in the same way you would other partitions - set them mounts points, and decide on their filesystem type.

Finish the instalation
Once you are done setting up these raid devices (and swap / boot partitions you decide to keep as non-raid), the installation should run smootly.

6) Configuring Swap Space

I mentioned before that the linux kernel automatically manages multiple swap partitions, meaning you can spread swap partitions across multiple drives for a performance boost without needing to create a raid array. A slight tweak may be needed however; each swap partition has a priority, and if you want the kernel to use both at the same time, you need to set the priority of each swap partition to be the same. First, type

swapon -s

to see your current swap usage. Mine outputs the following:

Filename                                Type            Size    Used    Priority
/dev/sda5                               partition       979956  39080   -1
/dev/sdb5                               partition       979956  0       -2

As you can see, the second swap partition isn't being used at the moment, and won't be until the first one is full. I want a performance gain, so I need to fix this by setting the priority of each partition to be the same. Do this in /etc/fstab, by adding pri=1 as an option to each of your swap partitions. My /etc/fstab file now looks like this:

# /dev/sda5
UUID=551aaf44-5a69-496c-8d1b-28a228489404 pri=1            swap    sw              0       0
# /dev/sdb5
UUID=807ff017-a9e7-4d25-9ad7-41fdba374820 pri=1            swap    sw              0       0

7) How to do things manually

As I mentioned earlier, if you ever boot into your instalation with a live cd, you will need to install mdadm to be able to access your raid devices, so its a good idea to at least roughly know how mdadm works. http://man-wiki.net/index.php/8:mdadm has some detailed information, but the important options are simply:

-A, --assemble
    Assemble a pre-existing array that was previously created with --create.
-C, --create
    Create a new array. You only ever need to do this once, if you try to create arrays with partitions that are part of other arrays, mdadm will warn you.
--stop
    Stop an assembled array. The array must be unmounted before this will work.

When using --create, the options are:

mdadm --create md-device --chunk=X --level=Y --raid-devices=Z devices
-c, --chunk=
    Specify chunk size of kibibytes.  The default is 64.
-l, --level=
    Set  raid  level, options are: linear, raid0, 0, stripe, raid1, 1, mirror, raid4, 4, raid5,  5,  raid6, 6,  multipath,  mp, fautly.
-n, --raid-devices=
    Specify the number of active devices in the array.

for example:

mdadm --create /dev/md0 --chunk=4 --level=0 --raid-devices=2 /dev/sda1 /dev/sdb1

will create a raid0 array /dev/md0 formed from /dev/sda1 and /dev/sdb1, with chunk size 4.

When using --assemble, the usage is simply:

mdadm --assemble md-device component-devices

for example

mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1

Which will assemble the raid array /dev/md0 from the partitions /dev/sda1 and /dev/sdb1

Alternatively you can use:

mdadm --assemble --scan

and it will assemble any raid arrays it can detect automatically.

Lastly,

mdadm --stop /dev/md0

will stop the assembled array md0, so long as its not mounted.

If you wish you can set the partitions up yourself manually using fdisk and mdadm from the command line. Either boot up a desktop live cd and apt-get mdadm as described before, or boot up the alternate installer and hit escape until you see a list of the different stages of instalation - the bottom one should read execute shell - which will drop you at a shell with fdisk, mdadm and mkfs etc... available.

Note that if you ever need to create another raid partition, you create filesystems on them in exactly the same way you would a normal physical partition. For example, to create an ext3 filesystem on /dev/md0 I would use:

mkfs.ext3 /dev/md0

And to create a swapspace on /dev/sda7 I woud use:

mkswap /dev/sda7

Lastly, mdadm has a configuration file located at

/etc/mdadm/mdadm.conf

this file is usually automatically generated, and mdadm will probably work fine without it anyway. If you're interested then http://man-wiki.net/index.php/5:mdadm.conf has some more information.

And that's pretty much it. As long as you have mdadm available, you can create / assemble raid arrays out of identical partitions. Once you've assembled the array, treat it the same way you would a partition on a physical disk, and you can't really go wrong!

I hope this has helped someone! At the moment I've omitted certain aspects of dealing with raids with redundancy (like raid 1 and raid 5), such a rebuilding failed arrays, simply because I've never done it before. Again, I may have a play around and add some more information later (for completeness), or if anyone else running a raid 1 wants to contribute, it would be most welcome.

Other links

The Linux Software Raid Howto:
http://tldp.org/HOWTO/Software-RAID-HOWTO.html
This guide refers to a package "raidtools2" which I couldn't find in the Ubuntu repositories - use mdadm instead, it does the same thing.

Quick HOWTO: Linux Software Raid
http://www.linuxhomenetworking.com/w..._Software_RAID

Using mdadm to manage Linux Software Raid arrays
http://www.linuxdevcenter.com/pub/a/...2/05/RAID.html

Ubuntu Fake Raid HOWTO In the community contributed documentation
https://help.ubuntu.com/community/Fa...ght=%28raid%29

I hope this guide helps people understand and set up their raids

[Fraoch]

Thanks very much for this - it worked flawlessly!

[mgrusin]

Great howto! Please consider adding this to the community documentaion wiki. There is currently nothing in there (at least that I could find) on the use of mdadm.

-MG

[bauhaus]

Excellent information and really useful as I am about to setup a RAID 5 myself and very much needed some pointers. I already have a working install of Feisty so I it looks like I will have to install mdadm and go from there. I'll let you know if it works out

[gnychis]

so i followed your guide, and i did the manual configuration because the minimal server installer CD was giving me problems.

so I created /dev/md0 and used mdadm --create to create a RAID0.

I made use of your note about RAID0 and boot partitions, so I created one on my first disk and made sure the two disks matched and all.

The install went perfectly fine, but when I reboot i get:

Starting up ...
Loading, please wait...
mdadm: No devices listed in conf file were found.

and nothing happens.

where is the config file stored? how do i get it to reassemble my raid disks on boot?

thanks!

[Qumefox]

My feisty install hangs at the exact same spot when one of the discs of my raid5 array is missing. (which I don't understand, since it should still be able to run on the 2 remaining drives, since that's the whole point of raid5) You might try using the instructions on accessing your array via the desktop live cd, to make sure the array is indeed there and not corrupted somehow. If it's corrupted, with a raid0 array, you'd be missing your whole root fs, and the kernel would have nothing else to load.

[Qumefox]

I just discovered that if you let it sit, it'll timeout and dump you to the initramfs prompt, and from there you should be able to see exactly why it's not booting correctly.

My guess is that mdadm is trying to build the array before udev has found all the devices though, so your raid0 array isn't initializing. And therefor, you have no / filesystem.

At least this is what I gleaned from the bug reports about similar symptoms.

[lns]

Thanks for this howto!! This is great. I wish I knew that Feisty alternate-install (x86-64) CD didn't have an installer option for software RAID before I nuked my newly installed LTSP server Oh well. A RAID-1/5 HOWTO would be awesome. Maybe I'll try and write one up, although I'm not very experienced at writing HOWTOs.

[gnychis]

Originally Posted by gnychis so i followed your guide, and i did the manual configuration because the minimal server installer CD was giving me problems. so I created /dev/md0 and used mdadm --create to create a RAID0. I made use of your note about RAID0 and boot partitions, so I created one on my first disk and made sure the two disks matched and all. The install went perfectly fine, but when I reboot i get: Starting up ... Loading, please wait... mdadm: No devices listed in conf file were found. and nothing happens. where is the config file stored? how do i get it to reassemble my raid disks on boot? thanks!

bahhh... i fixed my problem by removing 'savedefault' from the grub config

[uclalinux]

thanks