Introduction
This is a step-by-step tutorial/guide for setting up a complete Kubuntu 7.10 (Gutsy Gibbon) system on RAID5. By this I mean that the root partition (and other system partitions) are on RAID5. If you plan to install your system on RAID1 and only use RAID5 for data, you may find this guide helpful. But it's focus is installing the system on RAID5.
I've installed Kubuntu because I prefer the KDE desktop. But this should work just as well for Ubuntu or one of the other Ubuntu-derived variants. I have not installed anything other than Kubuntu as described here. So be forewarned. But I can think of no reason why it shouldn't work.
I did a lot of research on exactly how to install to RAID5. Here are a few sources that I bookmarked and found particularly useful. I want to thank the authors for their assistance!
These sources where very useful, particularly because they showed me the basics and provided ideas. But I never found a source to do specifically what I wanted: system installation to RAID5. So I decided to jump in head first and learn it.
I believe I installed from scratch a half dozen times. At least once, leftover partition information on my hard drives prevented me from getting the installer to do what I needed. So I zeroed all my hard drives to remove anything left over and started again. I kept a lot of notes and when I had it all working, I zeroed the drives and started again, following my notes to the letter to be sure I had it right. I've now written this guide directly from my notes.
I've tried to write this guide for someone with limited Linux experience. For experience users, it will probably seem too detailed. The RAID support in Linux is solid and stable. But as some of us have found, the installation isn't so easy. So I've tried to focus on detailed step-by-step instructions. The guide is a bit long. But if you have average experience with Linux, you should be able to complete the installation in a couple hours. It may take longer if you have very large partitions for your RAID arrays. It does take some time for the system to build the arrays. But you can take a break while it's doing that.
If you don't know much about RAID, you should do a little self-education. It will help if you know some of the basics, like the difference between RAID0, RAID1 and RAID5. If you know how RAID5 works then you'll understand why you need at least three drives and why in any RAID5 array you can have one drive fail but not two. And you'll understand why some partitions, like /boot, can't reside on RAID5.
Prerequisites
The prerequisite list is pretty short. Here's what you'll need:
- An Intel- or AMD-based PC.
- At least 3 hard drives of reasonable size.
- A CDROM drive.
- Connected monitor, keyboard and mouse.
- The Kubuntu 7.10 Alternate Install CD which you can download and burn.
Also recommended:
- A companion working system so you can consult the Internet if you have issues.
- The Kubuntu 7.10 Desktop CD which you can download and burn.
A working system connected to the Internet is highly recommended and almost a must have. You probably won't need the Desktop CD. It's nice to have if you run into issues because you can boot a working Kubuntu system in memory and use it to troubleshoot.
Important Learning Experiences
There are a few things I learned while going through this process. I'll point those out up front and then expand on them later.
- The installer will not install to a RAID5 partition. RAID5 is available in the partitioning menus, but it doesn't work. RAID1 seems to work very well. I'm not certain of the reason for this. But I suspect it is due to limited RAID support in the installer for actually running RAID, as opposed to configuring RAID. The process presented here works around this issue by installing the system to a non-RAID partition and then moving it to RAID5 partitions created later.
- The default init process doesn't support putting /var or /tmp in partitions separate from the root partition. The issue is in the sequence services are started. The RAID daemon is not started soon enough. I will show you how to change things so it will work.
- If you don't install the root partition as a RAID partition, the installer will not include RAID support when it builds the kernel. I will show you how to fix this as well. It adds an extra step or two.
My Hardware ... For the Curious
I'm building a medium to high-end server system. I have three desktops and a laptop at home, plus the laptop owned by my employer that I carry between home and work. These are all individual systems, three of the four running ******* and the other running Kubuntu. I don't have much of a network, just a couple of ******* shares and printer sharing. I want to set up a good network with a central server. I've recently had a couple of disk failures. Although I didn't lose any data, one was a close call. I've gotten lax about backups and I want to correct that. I have a lot of data and would like to centralize some of it and then also do daily backups from all the desktops and laptops to the server. I kicked the idea around a while and finally decided to do RAID5. The choice of Linux was a no-brainer for me. The thing I like about RAID5 is that once the initial installation is done, if I add another like-size disk I increase the amount of storage by the size of the disk (roughly). This isn't true of RAID1; I'd have to add two disks to double the size.
My priority when choosing hardware was to find a motherboard that supported a high number of SATA drives. Given that I also needed a case with ample space and cooling for the drives and a big power supply. But since it's a server, I don't need a high-end video card. I could have saved a few dollars on memory, but I got a great deal so I filled it up. Here's what I have for hardware:
- Antec P182 Mid-Tower case
- CoolerMaster Real Power Pro 1000W power supply
- Gigabyte GA-P35-DS3P Rev 2 motherboard
- Intel Core 2 Duo E6750 2.66Ghz CPU
- 8 GB Patriot 800Mhz PC6400 (4x2GB) memory
- 5 Seagate ST3500320AS 500GB, 7200rpm, 32MB cache, hard drives
- Samsung SH-S202G DVD burner
- EVGA e-GeForce 6200LE PCI-E graphics card
- Sony MPF920 1.44MB floppy drive
The BIOS supports AHCI (Advanced Host Controller Interface) for SATA. The Linux kernel has excellent support for AHCI. So I enabled it in the BIOS. It wasn't enabled by default. I've seen no issues with it so far.
Decide on Partition Scheme
There are a lot of different ideas about how to partition your system. It really comes down to a matter of personal preference. I am from the "old school" on this one and I like separate partitions for /var and /tmp. The reason is simple: These two directories are used by programs for temporary output files which may be very large. For example, the print spooler uses temp space under /var/spool. If someone prints a very large file, the spooled output could be enormous. If /var is in the root filesystem, its default location, and the spooled output fills the filesystem, the system will crash. If /var is in a different filesystem and the same thing happens, the system does not crash and you can fix the issue. As I said before, this is a personal preference. It does make things more complicated.
As I mentioned above, putting /var and/or /tmp in separate partitions complicates the installation process. If you keep them in the root partition, where they are by default, there are a number of steps in this guide that you can skip. I will point these out. But you may want to think about that ahead of time while you're deciding on how you will set up your partitions.
A RAID5 installation requires at least two partitions, one for /boot and one for the root partition. You cannot put /boot on a RAID5 partition. The reason is quite simple. The boot loader has no built-in RAID support. It expects to find the kernel in a partition on a single disk. RAID5 uses striping. So any particular file is spread across all the disks of the array. You have two choices here: 1) you can create identical non-RAID partitions on each disk for /boot and manually keep them in sync, or 2) you can use a RAID1 partition; the RAID software will keep them in sync for you. The latter is really the way to go. Note that I didn't list a third option which is to put /boot on only one disk. This really isn't an option in my opinion. Why do RAID and then leave this as a single point of failure?
Root will be on a RAID5 partition when we're all done. If you don't want to use RAID5 for your system installation, then this guide isn't for you. You may find it useful. But there are other sources that are more specific to your situation.
Once you decide on what your partitions will be, you need to decide what size to make them. Opinions vary on this as well. There is general agreement that /boot can be realtively small. A hundred megabytes is enough to hold three or four versions of the kernel. I've observed that the base system requires approximately 2.25GB. So if you put it all in the root partition, you need at least that. But, of course, you'll want some room for growth.
Here are my partitions and their sizes:
- /boot (128 MB)
- / (10 GB)
- /usr (20 GB)
- /var (20 GB)
- /tmp (10 GB)
- swap (16 GB)
- /home (what's left)
Here are some of my thoughts on the decisions I made:
- I have five 500 GB disk drives. So, on average, none of the system partitions are large. But they are definitely large enough for growth.
- Most add-on software will be installed in /usr, so it should be larger than the root.
- I wanted to provide ample room for system temporary storage, so I decided to make /var larger than perhaps it really needs to be.
- There are some heated arguments about how large swap should be. Many will argue that a GB is more than enough. I simply used the old rule of thumb that says it should be twice the size of your RAM. It could easily be with 8 GB of RAM, mine will never get used!
There is an argument for leaving a little unused space on each drive, maybe about 5% of the drive, to account for slightly smaller drives added to the array at a later time. For example, let's say you start with some 500GB drives which are actually 502GB (manufacturers round the numbers). You set up your partitions to consume the entire disk. Then in a couple of years you decide to expand your array. The exact drives you previously purchased are no longer available, so you buy a different model advertised as 500GB, but it's actual size is 497GB. You can't make the partitions exactly the same on the new drive; it's not big enough. If you'd left 5% (10GB) unused, then you could. I agree with this reasoning. But there are other factors. In my case, I have a 16GB swap partition that I would not need to create on the new drive. There is really no need to have another spare. Also, I wouldn't have to include space to expand any of my system partitions. Having them on five drives is enough. So it all depends. You should think through these issues as you decide how to set up your partitions.
A little more discussion on swap. Some will argue that you don't need to put swap on a RAID partition. The basis for the argument is that it's the most transient of all your data. But the other side of the argument is that if you don't put swap on RAID and you lose the disk where swap is, or one of multiple disks when swap is on multiple disks, your system will crash. So again, to me it doesn't make sense to have this single point of failure. My swap is on a RAID1 partition. Why RAID1? Why not RAID5? Overall, performance for swap is a little better with RAID1. (I actually, set up the partition as RAID1 on two drives with the other three drives as spares.) The point about performance is open to debate. But I believe that the read:write ratio is closer to 1:1 for swap than most other types of data. For example, if you have a media server, you'd typically write a media file once, but then read it many times. For swap, it's only used when RAM is full. So it seems logical if the system finds it needs to read something from swap, it probably also needs to write something out to swap to make room in RAM. Write performance is better with RAID1 than with RAID5. So the closer the read:write ratio gets to 1:1, the better RAID1 will perform compared to RAID5.
Before leaving this discussion about partitioning, I need to point out another very important aspect that should enter into your decisions. The effective size of a RAID1 partition is equal to the size of a single component partition that makes up the RAID array. If you have a 1 GB partition on two disks or on 10 disks and you create a RAID1 array will all the partitions, you have 1 GB of usable space. But this isn't the case with RAID5. Since RAID5 includes striping the effective size of the array is equal to the size of the smallest component partition times one less than the total number of component partitions. So in my case, assuming all 5 disks are in my RAID5 array, the effective size of a RAID5 partition is the size of a component partition on a single disk times 4 (5 - 1). This means in my case that when I create the partitions that will make up a RAID5 array I want to make each one-fourth the effective size of the full RAID5 partition. For example, my 10 GB root partition will be composed of five 2.5 GB partitions.
Installation From CD
With all the decisions made on partitions, it's time to start the installation of the system. The only thing you'll need is the bootable Kubuntu 7.10 alternate installation CD and a network connection.
Start the installation.
- Boot from the installation CD. You may need to change your EPROM settings to enable booting from the CD-ROM drive.
- Select Install in text mode when the menu comes up.
- Proceed with installation until you reach the partitioning menu.
NOTE: When I say "select" in these instructions, that means use the arrow keys to move to that entry and the press ENTER.
NOTE: The steps I outline here are based on my partitioning decisions. If you're setting up different partitions, you'll need to make adjustments. If your partitions are different sizes, then adjust the sizes. If you have fewer partitions or more partitions, then adjust the names of the devices accordingly.
Configure partitions.
- Select Manual partioning at the partitioning menu. The screen will show some instructions at the top and display your drives. The drive display should be similar to this:
Code:
SCSI1 (0,0,0) (sda) - 500.1 GB ATA ST3500320AS
SCSI2 (0,0,0) (sdb) - 500.1 GB ATA ST3500320AS
SCSI3 (0,0,0) (sdc) - 500.1 GB ATA ST3500320AS
SCSI4 (0,0,0) (sdd) - 500.1 GB ATA ST3500320AS
SCSI5 (0,0,0) (sde) - 500.1 GB ATA ST3500320AS
- Select SCSI1 in the list of drives.
- It will ask if you want to create a new empty partition table on the device. Select Yes. It will return you to the menu and the drive display should be similar to this:
Code:
SCSI1 (0,0,0) (sda) - 500.1 GB ATA ST3500320AS
pri/log 500.1 GB FREE SPACE
SCSI2 (0,0,0) (sdb) - 500.1 GB ATA ST3500320AS
SCSI3 (0,0,0) (sdc) - 500.1 GB ATA ST3500320AS
SCSI4 (0,0,0) (sdd) - 500.1 GB ATA ST3500320AS
SCSI5 (0,0,0) (sde) - 500.1 GB ATA ST3500320AS
- Select the line that begins pri/log ... to start creating the /boot partition.
- It will ask if you want to create a new partition. Select Yes.
- It will ask for the size of the partition. Enter 128mb.
- It will ask for the type for the new partition. Select Primary.
- It will ask for the location for the new partition. Select Beginning.
- It will present default settings for the partition. Select the Use as line.
- It will present a list of partition types. Select physical volume for RAID.
- It will return to the previous menu. Select Bootable flag. This will set the bootable flag to on.
- Select Done setting up the partition. It will return to the list of drives and it should now be similar to this:
Code:
SCSI1 (0,0,0) (sda) - 500.1 GB ATA ST3500320AS
#1 primary 131.6 MB B K raid
pri/log 500.1 GB FREE SPACE
SCSI2 (0,0,0) (sdb) - 500.1 GB ATA ST3500320AS
SCSI3 (0,0,0) (sdc) - 500.1 GB ATA ST3500320AS
SCSI4 (0,0,0) (sdd) - 500.1 GB ATA ST3500320AS
SCSI5 (0,0,0) (sde) - 500.1 GB ATA ST3500320AS
- Select the line that begins pri/log ... again to start creating what will eventually be the root partition.
- It will ask if you want to create a new partition. Select Yes.
- It will ask for the size of the partition. Enter 2.5gb.
- It will ask for the type for the new partition. Select Logical.
- It will ask for the location for the new partition. Select Beginning.
- It will present default settings for the partition. Select the Use as line.
- It will present a list of partition types. Select physical volume for RAID.
- It will return to the previous menu. Select Done setting up the partition. It will return to the list of drives and it should now be similar to this:
Code:
SCSI1 (0,0,0) (sda) - 500.1 GB ATA ST3500320AS
#1 primary 131.6 MB B K raid
#5 logical 2.5 GB K raid
pri/log 500.1 GB FREE SPACE
SCSI2 (0,0,0) (sdb) - 500.1 GB ATA ST3500320AS
SCSI3 (0,0,0) (sdc) - 500.1 GB ATA ST3500320AS
SCSI4 (0,0,0) (sdd) - 500.1 GB ATA ST3500320AS
SCSI5 (0,0,0) (sde) - 500.1 GB ATA ST3500320AS
- Now that you've created a couple partitions, you should know how to do it. Create four more logical partitions in the same manner. The sizes are 5GB, 5GB, 2.5GB and 16GB in that order. These will become the /usr, /var, /tmp and swap partitions, respectively.
- Now create the last partition. This will eventually be the /home partition. But right now it will be used as a temporary root partition. Select the line that begins pri/log ... to start.
- It will ask if you want to create a new partition. Select Yes.
- It will present all the remaining free space as the size for the partition. Press ENTER to use that size.
- It will ask for the type for the new partition. Select Logical.
- It will ask for the location for the new partition. Select Beginning.
- Set Use as to Ext3.
- Set Mount point to /.
- Select Done setting up the partition. It will return to the list of drives and it should now be similar to this:
Code:
SCSI1 (0,0,0) (sda) - 500.1 GB ATA ST3500320AS
#1 primary 131.6 MB B K raid
#5 logical 2.5 GB K raid
#6 logical 5.0 GB K raid
#7 logical 5.0 GB K raid
#8 logical 2.5 GB K raid
#9 logical 16.0 GB K raid
#10 logical 469.0 GB f ext3 /
SCSI2 (0,0,0) (sdb) - 500.1 GB ATA ST3500320AS
SCSI3 (0,0,0) (sdc) - 500.1 GB ATA ST3500320AS
SCSI4 (0,0,0) (sdd) - 500.1 GB ATA ST3500320AS
SCSI5 (0,0,0) (sde) - 500.1 GB ATA ST3500320AS
- Now repeat this entire process for each of the remaining drives with the following exception: Make the last (very large) partition on each drive a physical volume for RAID partition. This will set the type correctly for later on when the RAID array is created. All the other partitions should be set up identically to the first drive. When you are done the list of drives and partitions should look similar to this:
Code:
SCSI1 (0,0,0) (sda) - 500.1 GB ATA ST3500320AS
#1 primary 131.6 MB B K raid
#5 logical 2.5 GB K raid
#6 logical 5.0 GB K raid
#7 logical 5.0 GB K raid
#8 logical 2.5 GB K raid
#9 logical 16.0 GB K raid
#10 logical 469.0 GB f ext3 /
SCSI2 (0,0,0) (sdb) - 500.1 GB ATA ST3500320AS
#1 primary 131.6 MB B K raid
#5 logical 2.5 GB K raid
#6 logical 5.0 GB K raid
#7 logical 5.0 GB K raid
#8 logical 2.5 GB K raid
#9 logical 16.0 GB K raid
#10 logical 469.0 GB K raid
SCSI3 (0,0,0) (sdc) - 500.1 GB ATA ST3500320AS
#1 primary 131.6 MB B K raid
#5 logical 2.5 GB K raid
#6 logical 5.0 GB K raid
#7 logical 5.0 GB K raid
#8 logical 2.5 GB K raid
#9 logical 16.0 GB K raid
#10 logical 469.0 GB K raid
SCSI4 (0,0,0) (sdd) - 500.1 GB ATA ST3500320AS
#1 primary 131.6 MB B K raid
#5 logical 2.5 GB K raid
#6 logical 5.0 GB K raid
#7 logical 5.0 GB K raid
#8 logical 2.5 GB K raid
#9 logical 16.0 GB K raid
#10 logical 469.0 GB K raid
SCSI5 (0,0,0) (sde) - 500.1 GB ATA ST3500320AS
#1 primary 131.6 MB B K raid
#5 logical 2.5 GB K raid
#6 logical 5.0 GB K raid
#7 logical 5.0 GB K raid
#8 logical 2.5 GB K raid
#9 logical 16.0 GB K raid
#10 logical 469.0 GB K raid
- Review the partition configuration to this point and be sure it is as you want.
- Now it's time to configure the RAID1 partitions for /boot and swap. Select Configure software RAID. The menu selection is above the list of drives and partitions.
- It will ask if you want to write the changes to the devices and configure software RAID. Select Yes. It will create the Ext3 filesystem for the root partition. This may take a minute or so because the partition is large.
- When it's done it will present the menu for configuring RAID. Select Create MD device.
- Select RAID1 as the multidisk device type.
- Set the number of active devices for the RAID1 array to 5.
- Set the number of spare devices for the RAID1 array to 0.
- It will now present a list of all the mapped devices for the partitions you created and ask which ones you want to be the active devices in the RAID array. Select /dev/sda1, /dev/sdb1, /dev/sdc1, /dev/sdd1 and /dev/sde1. Use the arrow keys to move in the list. Use the space bar to select a list entry.
- When you've selected the correct devices, press ENTER.
- It will return to the menu for configuring RAID. Select Create MD device.
- Select RAID1 as the multidisk device type.
- Set the number of active devices for the RAID1 array to 2.
- Set the number of spare devices for the RAID1 array to 3.
- It will now present a list of the mapped devices for the partitions you created and ask which ones you want to be the active devices in the RAID array. (NOTE: I found that some of the devices I put in the first array were not in the list, but others were. Don't be concerned about that.) Select /dev/sda9 and /dev/sdb9 and press ENTER.
- It will present the list again and ask which ones you want to be the spare devices in the RAID array. Select /dev/sdc9, /dev/sdd9 and /dev/sde9.
- It will return to the menu for configuring RAID. Select Finish.
- It will return to the partitioning menu and the list of drives and partitions should look similar to this:
Code:
RAID1 device #0 - 131.5 MB Software RAID device
#1 131.5 MB
RAID1 device #1 - 16.0 GB Software RAID device
#1 16.0 GB
SCSI1 (0,0,0) (sda) - 500.1 GB ATA ST3500320AS
... << the rest of the list of drives and partitions will follow >>
- Select the line that reads #1 131.5 MB.
- Set Use as to Ext3 journaling file system and Mount point to /boot. Leave the rest of the settings on default values. Select Done setting up the partition.
- It will return to the list of drives and partitions. Select the line that reads #1 16.0 GB (the line under RAID device #1).
- Set Use as to swap area.
- Select Done setting up the partition.
- It will return to the list of drives and partitions. It should now look similar to this:
Code:
RAID1 device #0 - 131.5 MB Software RAID device
#1 131.5 MB F ext3 /boot
RAID1 device #1 - 16.0 GB Software RAID device
#1 16.0 GB f swap swap
SCSI1 (0,0,0) (sda) - 500.1 GB ATA ST3500320AS
... << the rest of the list of drives and partitions will follow >>
- Partitioning is now done! Select Finish partitioning and write changes to disk.
- It will ask if you want to write the changes to disk. Select Yes.
- It will format the RAID partitions and then the installation will continue.
Finish the installation.
Follow the prompts and complete the remainder of the installation. When it reaches the end, it will eject the CD. Remove the CD, close the tray, and reboot. It should boot into the OS. At this point, you've installed GRUB, the boot loader, and the kernel on the very small RAID1 array at the beginning of the disk and everything else temporarily on the large partition on the first disk. swap is on the other RAID1 array that was configured. You have a working system!
Reconfiguration for RAID5
Now that the initial installation is complete and you have a complete system installed, it can be reconfigured to put the system on RAID5 arrays. This process involves setting up the arrays, copying system files into place, and doing some reconfiguration of the startup process.
Log in and look around.
You can now log in as the user you set up during the installation process. Besides root, that is the only user. And by default, root isn't allowed to log in. So start by logging in and bringing up the desktop. First, get familiar with the initial configuration.
- Open a Konsole window. Click the icon in the lower left corner of the screen. Go to System --> Konsole. You can resize the Konsole window by dragging the edges with a mouse. All the commands in this guide are entered in a Konsole window.
- In the Konsole window, enter cat /proc/mdstat. This will show you the status of the current RAID arrays. Remember this command. You'll use it quite a bit. The output should be similar to this:
Code:
root@ubuntu:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sda9[0] sde9[4](S) sdd9[3](S) sdc9[2](S) sdb9[1]
15623104 blocks [2/2] [UU]
md0 : active raid1 sda1[0] sde1[4] sdd1[3] sdc1[2] sdb1[1]
128384 blocks [5/5] [UUUUU]
unused devices: <none>
root@ubuntu:~#
The U letters at the end of the second line of each array indicate how many component partitions are up and running. The S in parens indicates a spare. - Enter mount. This will show you the file systems which are mounted. You should see something like this:
Code:
root@ubuntu:~# mount
/dev/sda10 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
lrm on /lib/modules/2.6.22-14-generic/volatile type tmpfs (rw)
/dev/md0 on /boot type ext3 (rw)
securityfs on /sys/kernel/security type securityfs (rw)
root@ubuntu:~#
The swap partition isn't in the above list because it isn't a mounted file system. You can take a look at swap by entering cat /proc/swaps. You'll see something like this:
Code:
root@ubuntu:~# cat /proc/swaps
Filename Type Size Used Priority
/dev/md1 partition 15623096 0 -1
root@ubuntu:~#
Create the RAID5 partitions.
The first step to reconfigure the system for RAID5 is to create the RAID partitions that will hold the system files.
NOTE: All the commands I present here need to be run as root. You will need to preface each one with sudo ("switch user and do") to run it as root. Personally, I think it gets to be a real pain using sudo and needing to enter a password occasionally. I prefer to simply switch user to root. Yes, it may be a little more dangerous. But it has never been an issue for me. I don't normally do work as root, just system admin work and then I exit the root shell. If you want to be able to switch user to root, you need to give the root account a password. You can do this with the command sudo passwd. It will prompt you for the password to set. Once you've set the password you can become root any time with su - and entering the root password. This will open a new shell. You can exit the shell by entering the exit command.
- Create the RAID5 array for the root partition with the following command:
Code:
root@ubuntu:~# mdadm --create --verbose /dev/md2 --level=5 --raid-devices=5 /dev/sda5 /dev/sdb5 /dev/sdc5 /dev/sdd5 /dev/sde5
mdadm: layout details to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: size set to 2441728K
mdadm: array /dev/md2 started
root@ubuntu:~#
- Create three more RAID5 arrays for /usr, /var and /tmp.
Code:
root@ubuntu:~# mdadm --create --verbose /dev/md3 --level=5 --raid-devices=5 /dev/sda6 /dev/sdb6 /dev/sdc6 /dev/sdd6 /dev/sde6
...
root@ubuntu:~# mdadm --create --verbose /dev/md4 --level=5 --raid-devices=5 /dev/sda7 /dev/sdb7 /dev/sdc7 /dev/sdd7 /dev/sde7
...
root@ubuntu:~# mdadm --create --verbose /dev/md5 --level=5 --raid-devices=5 /dev/sda8 /dev/sdb8 /dev/sdc8 /dev/sdd8 /dev/sde8
...
- Wait until the arrays are completely built before creating the file systems. You can watch the progress with the command cat /proc/mdstat. There will be a progress meter and percentage to indicate how far along it is. I don't have a specific example to show. But it is easily discerned from the display.
- When all the "recovery" is complete, it is time to create the file systems on the RAID partitions.
Code:
root@ubuntu:~# mkfs -t ext3 /dev/md2
<< usual output from mkfs >>
root@ubuntu:~#
I don't have example output from mkfs. - Repeat the command for /dev/md3, /dev/md4 and /dev/md5.
- Now update mdadm.conf to retain the RAID configuration when the system restarts. Be sure you are in a non-system directory (root's home directory is fine) and run this command:
Code:
root@ubuntu:~# mdadm --detail --scan > mdadm.new
root@ubuntu:~#
- Open /etc/mdadm/mdadm.conf in your chosen text editor. Find the lines that begin with "ARRAY" and comment them out. NOTE: In my version, the definition for the md1 array was on two lines.
- Open the mdadm.new file you just created and copy all the lines from that file into /etc/mdadm/mdadm.conf immediately after the lines you commented out. After the changes, it should look similar to this:
Code:
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions
# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
#ARRAY /dev/md0 level=raid1 num-devices=5 UUID=4af98750:2c546770:e0a38126:d70ed490
#ARRAY /dev/md1 level=raid1 num-devices=2 UUID=17f37a2d:96cbe834:401bc8f3:e3b6fa3e
# spares=3
ARRAY /dev/md0 level=raid1 num-devices=5 UUID=4af98750:2c546770:e0a38126:d70ed490
ARRAY /dev/md1 level=raid1 num-devices=2 spares=3 UUID=17f37a2d:96cbe834:401bc8f3:e3b6fa3e
ARRAY /dev/md2 level=raid5 num-devices=5 UUID=e22c986f:e0013765:f9935161:a927a204
ARRAY /dev/md3 level=raid5 num-devices=5 UUID=26600050:a9781133:f9935161:a927a204
ARRAY /dev/md4 level=raid5 num-devices=5 UUID=05d53ed8:1c2918a8:f9935161:a927a204
ARRAY /dev/md5 level=raid5 num-devices=5 UUID=f359ace2:fd8582ba:f9935161:a927a204
# This file was auto-generated on Mon, 04 Feb 2008 01:55:40 +0000
# by mkconf $Id: mkconf 324 2007-05-05 18:49:44Z madduck $
- Save the changes to /etc/mdadm/mdadm.conf.
- At this point I suggest you reboot. This isn't necessary. But I like to reboot occaisionally to be sure that changes I've made work as expected. If you wait too long before a reboot and things don't work as you expect, you have a lot more changes to investigate.
- After you reboot, log in, and check /proc/mdstat. You should see all the RAID arrays as active.
Restructure the installed OS.
Now that the RAID5 arrays for the system are configured and active it is time to move the different parts of the system to the arrays. Actually, we're going to copy them. The currently booted system will remain until we know the system will boot and run from the RAID arrays.
- Copy the root file system to it's new home. Mount the md2 array. Enter:
Code:
root@ubuntu:~# mount /dev/md2 /mnt
root@ubuntu:~#
- Use rsync to copy all the files. Enter:
Code:
root@ubuntu:~# rsync -daHv --exclude=proc/* --exclude=sys/* --exclude=boot/* --exclude=usr/* --exclude=var/* --exclude=tmp/* --exclude=mnt/* / /mnt
<< list of files copied >>
root@ubuntu:~#
It will list the files as they are copied. The proc and sys directory are in-memory file systems created by the kernel; they don't need to be copied; in fact, they can't be copied. The boot, usr, var and tmp directories will each be their on own file systems (we'll do those next). We can't recursively copy the mnt directory (it's the destination); so it is excluded. NOTE: If you want to see what it will copy without actually doing the copy, include the "-n" option. - When rsync is done, unmount the file system. Enter:
Code:
root@ubuntu:~# umount /mnt
root@ubuntu:~#
- Next we'll copy the /usr directory to its own file system. Mount the md3 array. Enter:
Code:
root@ubuntu:~# mount /dev/md3 /mnt
root@ubuntu:~#
- Copy it. Enter:
Code:
root@ubuntu:~# rsync -daHv /usr/ /mnt
<< list of files copied >>
root@ubuntu:~#
NOTE: Do NOT forget the trailing slash after /usr! This tells rsync to copy all files and directories under /usr, not /usr itself. If you leave off the trailing slash it will create a usr directory in /mnt and you don't want that. - When rsync is done, unmount the file system. Enter:
Code:
root@ubuntu:~# umount /mnt
root@ubuntu:~#
- Next we'll copy the /var directory to its own file system. Mount the md4 array. Enter:
Code:
root@ubuntu:~# mount /dev/md4 /mnt
root@ubuntu:~#
- Copy it. Enter:
Code:
root@ubuntu:~# rsync -daHv /var/ /mnt
<< list of files copied >>
root@ubuntu:~#
- When rsync is done, unmount the file system. Enter:
Code:
root@ubuntu:~# umount /mnt
root@ubuntu:~#
- The /tmp directory does not need to be copied since it contains only transient files. But the permissions on the mount point must be set correctly. To do that, enter the following commands:
Code:
root@ubuntu:~# mount /dev/md2 /mnt
root@ubuntu:~# mount /dev/md5 /mnt/tmp
root@ubuntu:~# chmod 01777 /mnt/tmp
root@ubuntu:~# umount /mnt/tmp
root@ubuntu:~#
Note that we left /dev/md2 mounted. We're going to make some changes to it. - The last thing to do in the restructuring part is to update /etc/fstab so the new file systems are automatically mounted. We'll do this to the mounted /dev/md2 file system and leave the original installed version alone.
- Edit /mnt/etc/fstab.
- Find the line where /dev/sda10 is mounted.
- Copy that line and paste the copy right below it.
- Comment out the original.
- In the pasted line change the string UUID=[long alphanumeric string] with /dev/md2.
- Make a copy of the line you just changed.
- Paste the copy under the line where /boot is mounted.
- In the pasted line change /dev/md2 to /dev/md3 and the mount point from / to /usr.
- Make a copy of the line you just changed and paste it right below that line.
- In the pasted line change /dev/md3 to /dev/md4 and the mount point from /usr to /var.
- Make a copy of the line you just changed and paste it right below that line.
- In the pasted line change /dev/md4 to /dev/md5 and the mount point from /var to /tmp.
- Save the changes. The new version should look similar to this:
Code:
# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# /dev/sda10
#UUID=2420f84c-8f9e-4021-a0e8-f02ee91b2853 / ext3 defaults,errors=remount-ro 0 1
/dev/md2 / ext3 defaults,errors=remount-ro 0 1
# /dev/md0
UUID=929843f5-3001-4b78-bf87-28ae3636d4b5 /boot ext3 defaults 0 2
/dev/md3 /usr ext3 defaults,errors=remount-ro 0 1
/dev/md4 /var ext3 defaults,errors=remount-ro 0 1
/dev/md5 /tmp ext3 defaults,errors=remount-ro 0 1
# /dev/md1
UUID=275a7465-da22-4a0e-be3d-785bbeb7f51f none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec 0 0
Update GRUB to boot from RAID.
Because we installed the root file system on a non-RAID partition, the kernel that is booted doesn't have RAID support. The RAID support is a module that must be included in the initrd.img file in /boot to allow the root partition to reside on a RAID array. The initrd.img is built by the installer. And if the root partition isn't on a RAID array, the installer doesn't include the module. So we need to add the module now.
The update-initramfs command is used to do the work. It uses /etc/fstab to determine if RAID support is needed. So we need to put the new fstab in place to run the command. Here is the procedure:
- Change directory. Enter:
Code:
root@ubuntu:~# cd /etc
root@ubuntu:/etc#
- Save the current file. Enter:
Code:
root@ubuntu:/etc# mv fstab fstab.save
root@ubuntu:/etc#
- Put the modified file in place. Enter:
Code:
root@ubuntu:/etc# cp /mnt/etc/fstab .
root@ubuntu:/etc#
That dot at the end is part of the command! - Update initrd.img. Enter:
Code:
root@ubuntu:/etc# update-initramfs -u
root@ubuntu:/etc#
- Restore the original fstab. Enter:
Code:
root@ubuntu:/etc# mv fstab.save fstab
root@ubuntu:/etc#
Now we need to update the GRUB menu so it will know how to load the root file system from the RAID array.
- Open /boot/grub/menu.lst in your editor.
- Find the menu at the end of the file.
- Copy the first two menu entries; the first one is 5 lines, the second is 4 lines.
- Paste the copy right before its current location.
- In the kernel line of each of the first two entries (the ones you just created), replace the string UUID=[long alphanumeric string] with /dev/md2. This tells GRUB that the root partion is on md2.
- On the two original entries (now the third and fourth entries), append "(non-RAID)" to the title line. When the menu displays, you'll be able to tell which ones will boot from original installation.
- While you're editing this file, find the line near the top that begins # kopt=root. Change the string UUID=[long alphanumeric string] to /dev/md2. This line, which is commented out with just a single sharp, defines the options to put on the kernel line in the menu when the menu is updated. If you don't change it and then later apply an update that needs to update the GRUB menu, your system won't boot anymore! I was bitten by this one and it took a while to determine what happened.
- Save the changes. The menu should look similar to this:
Code:
title Ubuntu 7.10, kernel 2.6.22-14-generic
root (hd0,0)
kernel /vmlinuz-2.6.22-14-generic root=/dev/md2 ro quiet splash
initrd /initrd.img-2.6.22-14-generic
quiet
title Ubuntu 7.10, kernel 2.6.22-14-generic (recovery mode)
root (hd0,0)
kernel /vmlinuz-2.6.22-14-generic root=/dev/md2 ro single
initrd /initrd.img-2.6.22-14-generic
title Ubuntu 7.10, kernel 2.6.22-14-generic (non-RAID)
root (hd0,0)
kernel /vmlinuz-2.6.22-14-generic root=UUID=2420f84c-8f9e-4021-a0e8-f02ee91b2853 ro quiet splash
initrd /initrd.img-2.6.22-14-generic
quiet
title Ubuntu 7.10, kernel 2.6.22-14-generic (recovery mode) (non-RAID)
root (hd0,0)
kernel /vmlinuz-2.6.22-14-generic root=UUID=2420f84c-8f9e-4021-a0e8-f02ee91b2853 ro single
initrd /initrd.img-2.6.22-14-generic
title Ubuntu 7.10, memtest86+
root (hd0,0)
kernel /memtest86+.bin
quiet
Note that we have left the existing installation in place. The only changes we've made to it's original pristine condition is we configured the RAID arrays and changed mdadm.conf to start them. And if you rebooted after those changes, then you know it will still boot.
Why is this important? If you have a typo or other mistake in the restructuring of the OS, it may not boot. If this is the case, then you can still boot the original installation and use it to troubleshoot the problem. We'll remove it later when we are more confident that the RAID installation is working.
Reconfigure when the RAID daemon is started.
By default, when the system starts up, it doesn't start the mdadm daemon until it goes into its final run level. If you don't know about run levels and want to learn, you can find plenty of sources on the web. Here's one explanation.
I don't want to attempt to explain the entire start up process here. But here's a quick overview:
- After the system boots, and near the end of the startup process, it runs the scripts in the /etc/rcS.d directory.
- Then it runs the scripts in the /etc/rcX.d directory, where X is the run level. For example, if it's going to run level 5, it runs the scripts in /etc/rc5.d.
- The scripts in these directories begin with either a K or an S. K means "kill". S mean "start". After the K or S is a two-digit number. The number determines the order for the scripts to run.
- When a run level is entered, the kill scripts for that level are run and then the start scripts for that level are run.
If you look at your system, you'll see that there is a /etc/rc1.d/K25mdadm script. This means mdadm will be killed when entering run level 1 (single-user mode). There is an S25mdadm script for run levels 2 thru 5. This means mdadm will be started when each of those run levels is entered. The problem is that is too late. By then the system has mounted the file systems in fstab and it can't mount the file systems on RAID arrays if the arrays aren't active!
We need to change the scripts so the mdadm daemon is started earlier and is always running no matter what run level we're in. This isn't difficult to do. We just need to remove the mdadm related scripts from all the rcX.d directories, where X is 1 thru 5, and add a script to rcS.d. Here are the commands:
Code:
root@ubuntu:~# cd /mnt/etc/rcS.d
root@ubuntu:/mnt/etc/rcS.d# ln -s ../init.d/mdadm S25mdadm
root@ubuntu:/mnt/etc/rcS.d# cd ..
root@ubuntu:/mnt/etc# rm rc1.d/K25mdadm
root@ubuntu:/mnt/etc# rm rc2.d/S25mdadm
root@ubuntu:/mnt/etc# rm rc3.d/S25mdadm
root@ubuntu:/mnt/etc# rm rc4.d/S25mdadm
root@ubuntu:/mnt/etc# rm rc5.d/S25mdadm
One final thing on the startup of the mdadm daemon. The script that starts the daemon stores its process ID in a file under /var. But the way we have things configured, /var is on its own RAID array and is not mounted when the mdadm daemon starts! So we'll change it so the file with the process ID is in /etc. This isn't ideal, but it will work.
Edit /mnt/etc/init.d/mdadm. Find the definition for RUNDIR and change it to /etc/mdadm. That is the directory where mdadm.conf is located. Save the changes.
NOTE: If you have /var, /tmp and /usr on your root partition, you can skip this whole section. It should work with the default configuration.
Reboot to the RAID5 system!
Unmount the file system from /mnt and reboot. Enter:
Code:
root@ubuntu:/mnt/etc# cd
root@ubuntu:~# umount /mnt
root@ubuntu:~# reboot
When it comes back up you'll be running Kubuntu from the RAID5 arrays!
Move /home to RAID5 and Clean Up
Almost done! The last major step is to create the final RAID5 partition for the /home file system. This will destroy the original system installation.
Change the partition type.
The first step is to change the type of the partition to "Linux raid autodetect". This is only necessary on the partition where the original installation was placed. The corresponding partitions on the other disks were set to the correct type during installation.
- Use fdisk to change the partition type. Enter fdisk /dev/sda
- It will probably warn that your disk is rather large and then prompt Command (m for help): Enter t
- It will prompt Partition number (1-10): Enter 10. Depending on how you partitioned your disks during installation, you may have a different number of partitions. You need to enter the number of the partition where the original root partition was installed.
- It will prompt Hex code (type L to list codes): Enter fd
- It will prompt Command (m for help): Enter w. It will save the changes and exit. Here's what it should all look like:
Code:
root@ubuntu:~# fdisk /dev/sda
The number of cylinders for this disk is set to 60801.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): t
Partition number (1-10): 10
Hex code (type L to list codes): fd
Command (m for help): w
root@ubuntu:~#
- Reboot the system so the kernel will load the new partition table from the disk.
Create the RAID array.
Now it's time to create the final RAID5 array and its file system.
- Create the RAID5 array for /home with the following command:
Code:
root@ubuntu:~# mdadm --create --verbose /dev/md6 --level=5 --raid-devices=5 /dev/sda10 /dev/sdb10 /dev/sdc10 /dev/sdd10 /dev/sde10
mdadm: /dev/sda10 appears to contain an ext2fs file system ...
Continue creating array? yes<ENTER>
mdadm: layout details to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: size set to 45798086K
mdadm: array /dev/md6 started
root@ubuntu:~#
You need to enter "yes" at the prompt to continue. - Check /proc/mdstat occasionally to check on the progress building the array. This will take a while. For my 469GB partitions, it took almost 3 hours!
- When all the "recovery" is complete, create the file system. Enter:
Code:
root@ubuntu:~# mkfs -t ext3 /dev/md6
<< usual output from mkfs >>
root@ubuntu:~#
Copy the files.
We need to copy the files from the existing /home directory to the new file system.
Code:
root@ubuntu:~# mount /dev/md6 /mnt
root@ubuntu:~# rsync -daHv /home/ /mnt
<< list of files copied >>
root@ubuntu:~# umount /mnt
root@ubuntu:~#
Update configuration.
Now we'll update the configuration so the array is started and the file system mounted when rebooting.
- Be sure you're in a safe directory, e.g. root's home. Enter:
Code:
root@ubuntu:~# mdadm --detail --scan > mdadm.new
root@ubuntu:~#
- Edit /etc/mdadm/mdadm.conf.
- Get the line for /dev/md6 from mdadm.new and paste it at the end of /etc/mdadm/mdadm.conf.
- Save the changes.
- Edit /etc/fstab.
- Find the line for /dev/md5. Copy it.
- Paste the copy right below the original.
- In the copy change /dev/md5 to /dev/md6 and /tmp to /home.
- Save the changes.
Clean up.
There are a few things to clean up now that the installation is complete. Remove a few files that have been copied and clean up configuration files. We'll do some of it from single-user mode.
- Reboot the system. When the message GRUB loading, please wait comes up, hit ESC to get the boot menu.
- Select the second entry for "recovery mode", and press ENTER to boot.
- When it prompts for the root password for mainenance, enter it.
- Enter the following commands:
Code:
root@ubuntu:~# umount /tmp
root@ubuntu:~# rm -rf /tmp/*
root@ubuntu:~# mount /tmp
root@ubuntu:~# umount /home
root@ubuntu:~# rm -rf /home/*
root@ubuntu:~# mount /home
root@ubuntu:~# init 5
These command remove all the files in the /tmp and /home directories in the root file system. They are no longer needed; they are on the mounted file systems. The last command will continue booting to the GUI. - Edit /etc/fstab and remove the original commented out entry for /dev/sda10. It is no longer valid. I also suggest changing the entries for /dev/md0 and /dev/md1 to use the device names instead of the UUIDs. But this is up to you. Here's what my final /etc/fstab looks like:
Code:
# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
/dev/md2 / ext3 defaults,errors=remount-ro 0 1
/dev/md0 /boot ext3 defaults 0 2
/dev/md3 /usr ext3 defaults,errors=remount-ro 0 1
/dev/md4 /var ext3 defaults,errors=remount-ro 0 1
/dev/md5 /tmp ext3 defaults,errors=remount-ro 0 1
/dev/md6 /home ext3 defaults,errors=remount-ro 0 1
/dev/md1 none swap sw 0 0
/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec 0 0
- Edit /etc/mdadm/mdadm.conf and remove the commented out array defintions.
- Edit /boot/grub/menu.lst and remove the menu entries for the non-raid boot options. These are no longer valid.
What's Left?
That's it, The installation is complete. However, at this point you should do some testing of your RAID arrays. You can fail different partitions and insure that the arrays continue to function in degraded mode. Then you can bring the partitions back and insure they are recovered. Also, you should set up mdadm to report status. You need to be aware if you have a failure. One failure on a RAID5 array is recoverable. But two simultaneously is not!
I hope you've found this guide to be helpful. If your configuration isn't exactly like mine, at least this may get you started on your way to a successful RAID5 installation.
Bookmarks