HOWTO: Linux Software Raid using mdadm

[Fafler]

So i received the drives yesterday, and i really hope you guys can help me on this one, 'cause by now i'm out of ideas. mdadm only adds 2 out of 3 drives, no matter what i do.

~# mdadm --create /dev/md0 --level=5 --raid-devices=3 --size=max /dev/sda1 /dev/sdb1 /dev/sdc1
raid5: raid level 5 set md0 active with 2 out of 3 devices, algorithm 2
mdadm: array /dev/md0 started
~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdc1[3] sdb1[1] sda1[0]
1285199744 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]

So far i've tried; changing SATA cables, changing the order of the drives, changed from Ubuntu 9.10 to Debian 5.04, moved the drives to another motherboard, tried with and without partitioning the drives and i even tried making a partition on each drive with about 50% of the total space.

The drives can be read and written without any problems and i can even make RAID 1 array with all three drives for a total of 4.5 tb without and troubles at all. But no matter what, i can't get a working RAID 5 array.

I really hope someone here is able to help me out.

BTW; the motherboards i've tried are based on Intel 965 and G35, each with a C2D processor and a 32 bit kernel. I think both of them carry the ICH9 south bridge. The drives are all Western Digital WD15EARS.

[bsmith1051]

Maybe it's defaulting to the 3rd drive being a spare? md0 shows all 3 partitions. What does the mdadm detail show?

> sudo mdadm --query --detail /dev/md0

[olesk]

I've been setting up a RAID 10 with fours disks and I am about to add two more to it. The two new disks are smaller than the ones already in the set, but as I want them to be another RAID1 set that can then be added to the stripe, it should be irrelevant. (It consists of two RAID 1 mirrors, which are then striped - all in one go, i.e. true RAID 10 with only one raid device, not raid 0+1 - the terminology gets a little confusing sometimes).

I use mdadm v2.6.7.1 on Ubuntu 9.10 with a 2.6.31-21-generic kernel.








I was hoping to clear up a couple of points before beginning to add two more disks:

1. I essentially used mdadm -v --create /dev/md0 --level=raid10 --raid-devices=4 /dev/sdb1 /dev/sdc1 etc... to build the array. Thus I do not know which disk pairs are mirrored (RAID1 pairs) - it could in theory be any combination of my 4 disks. Is there any way of finding out? Neither /proc/mdstat nor mdadm says anything about this (see below)
2. If I simply add another disk (which is a smaller size than the existing ones) using mdadm, will it be listed as hot spare?
3. If I finally add the second new disk, will mdadm realize that I now have two spares, build a mirror (RAID1) pair of them and add them to the RAID0 stripe, in effect expanding my RAID10 array, or do I have to do some sort of manual process here? mdadm does after all know that I am adding them to a RAID10 array...

Below is the output from cat /proc/mdstat (before adding any of the two new disks):

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid10 sde1[3] sdb1[0] sdd1[2] sdc1[1]
2930271872 blocks 64K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>

Update: mdadm --grow --help doesn''t mention raid10 at all, which perhaps means either that a) RAID10 cannot be grown, or b) I need a newer version of mdadm. It seems this has been discussed before, but perhaps this has changed lately?

[BMWolf]

EDIT: I changed fsck checking to 0 in fstab and everything boots. Is this required for raid? Problem solved, I guess.

Sorry for such a long winded post, but I'm at my wits end,PLEASE HELP!

Here go's...
I've recently built a new machine and needed to move /home and everything in it from one raid1 array to a new larger raid 1 array. To make sure I didn't lose any data by accident, I removed one of the working drives, set it aside, stopped the array and mounted it's partition manually. I'm using Server 9.10 64-Bit and doing everything via ssh.

I then placed one of the new soon to be RAIDed drives in, partitioned the drive, and setup the new array. 'sdc1' is the new disk and 'missing' is the mounted old disk(sdb1):

mdadm --create /dev/md0 --level=1 --raid-disks=2 missing /dev/sdc1

I then formatted /dev/md0

sudo mkfs.ext4 /dev/md0

...and copied files over to it following this tutorial(link). I used sudo su because the files would not copy due to permissions.

cd /home
sudo su
find . -depth -print0 | cpio --null --sparse -pvd /mnt/NEW_HOME

I shutdown removed the old smaller '/dev/sdb', added the new larger drive, powered on, and added the new drive to the array(after partitioning and formatting it).

sudo mdadm --add /dev/md0 /dev/sdb1

It synced the drives. I then checked the files and they were all there and in working order and both drives were in active sync. I then changed fstab to use the new array as '/home'.

bryan@NAS:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>

proc            /proc           proc    defaults        0       0

# / was on /dev/sda1 during installation
UUID=6d1a79fe-7431-4a14-b10c-a1bad83e566d       /       ext4    errors=remount-ro       0       1

# swap was on /dev/sda5 during installation
UUID=fef142b2-2bd9-4d96-9384-770c409a8618       none    swap    sw      0       0

#CD Rom
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto,exec,utf8       0       0

#data RAID1 array
UUID=ef49f27c:490d5346:cced5de7:ca715931        /home   ext4    nodev, nosuid        0       2

Then I reboot.

After the reboot I tried to log back on via ssh and the server would not connect. So I connected it to a monitor and it said it was waiting for /home UUID=ef49f27c:490d5346:cced5de7:ca715931 to come online. I entered the rescue shell, commented out the /home entry in fstab and rebooted. The server came up again but then complained that it couldn't chdir to /home as would be expected.

I check the array to make sure it is in working order; it's not:

bryan@NAS:/$ cat /proc/mdstat
Personalities :
md0 : inactive sdc1[1] sdb1[0]
      1953519872 blocks

unused devices: <none>

bryan@NAS:/$ dmesg | tail
...
[  293.858825] EXT4-fs (md0): unable to read superblock

So I stop the array and try to reassemble it:

bryan@NAS:/$ sudo mdadm --stop /dev/md0
mdadm: stopped /dev/md0

bryan@NAS:/$ sudo mdadm --assemble --scan
mdadm: /dev/md0 has been started with 2 drives.

bryan@NAS:/$ cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[0] sdc1[1]
      976759936 blocks [2/2] [UU]

unused devices: <none>

OK... Then:

bryan@NAS:/$ sudo mount -t ext4 /dev/md0 /home

bryan@NAS:/$ ls /home/
bryan  cupsys  lost+found  samba

EDIT: Forgot to mention that both drives are of type Linux_Raid_Autodetect(fd).

It works, so what the F is going on and what did I screw up? The drives are WD10EARS. These drives are not certified for RAID use, but they work without error once I do the above steps, so what's the deal? Once I reboot I have to do it all again.

Are the drives junk? Did I screw up by issuing the original commands as sudo su?

Which leads to my second question:
When looking at the ownership of most of the folders on these drives, root is listed as the owner and the group. Is this right? Wouldn't USER normally be the owner of /home/USER and all it's files?

Thanks in advance!!!

[LiLoather]

I created a 3 disk Linux software RAID 10 array to install Ubuntu 10.04 server on. Everything seemed to go well and it all seems to be working, but doing a --query of the third disk brings up the following report:

tony@Debbie:~$ sudo mdadm --query /dev/sdc1
/dev/sdc1: is not an md array
/dev/sdc1: device 2 in 3 device mismatch raid10 /dev/md0. Use mdadm --examine for more detail.
tony@Debbie:~$ sudo mdadm --examine /dev/sdc1
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.00
UUID : 6e735543:3dc59b4b:a8204fbe:f33447b2
Creation Time : Sat Sep 4 02:18:19 2010
Raid Level : raid10
Used Dev Size : 9765376 (9.31 GiB 10.00 GB)
Array Size : 9765376 (9.31 GiB 10.00 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0

Update Time : Sat Sep 4 13:25:59 2010
State : active
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Checksum : 0 - expected 8a517f27
Events : 40

Layout : near=2, far=1
Chunk Size : 64K

Number Major Minor RaidDevice State
this 2 8 33 2 active sync /dev/sdc1

0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
tony@Debbie:~$


What does the "device 2 in 3 device mismatch" and the "Checksum : 0 - expected 8a517f27" mean? Does anything need reconfiguring or is this array fatally flawed?

[AllGamer]

I keep getting this error:

mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: /dev/sdd has no superblock - assembly aborted

even after a successful creation, everytime after a reboot, and i try to re-assemble it manually

sudo mdadm --assemble /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde

The part that it complains about the missing superblock or the device in use changes everytime i reboot the machine

If i destroy the array and create a new one, it will work for as long as i keep the server running.

I want to make sure the RAID5 is mountable everytime after a reboot to automate the mounting process

I'll welcome any hints or info you can provide

thank you

[crtlbreak]

Hi AllGamer
In your assembly process

sudo mdadm --assemble /dev/md0 /dev/sdb /dev/sdc /dev/sdd /dev/sde

Shows you as trying to assemble a RAID5 with spare?
You are using the device /dev/sdx as opposed to the partitions /dev/sdxn - where n is the number.

[AllGamer]

Hi AllGamer
In your assembly process



Shows you as trying to assemble a RAID5 with spare?
You are using the device /dev/sdx as opposed to the partitions /dev/sdxn - where n is the number.

you mean the partitions?
like /dev/sdbp1?

the entire drive is used for the raid.

anyways i think i might have found what was causing the problem

when i did a mount, to list all the devices it shows this

/dev/sde1 on / type ext4 (rw,errors=remount-ro,user_xattr,commit=0)

that should definitely not be there, seems like i still got some remnants of the previous LVM RAID trials.

[nathan58]

I am totally agree with all the views under that forum. make sure you are root and that you and the backup file are in the root of the filesystem.

[TheR]

you mean the partitions?
like /dev/sdbp1?

the entire drive is used for the raid.

anyways i think i might have found what was causing the problem

when i did a mount, to list all the devices it shows this

/dev/sde1 on / type ext4 (rw,errors=remount-ro,user_xattr,commit=0)

that should definitely not be there, seems like i still got some remnants of the previous LVM RAID trials.

Try to wipe out beginning of disk with dd.

by
TheR