HDD not seen or recognized by server (Ubuntu Server 12.04.3, 64, mdadm RAID5)
Hi all--it's been a fun month of moving our servers; after lots of troubleshooting, upgrading to 12.04.3, spending an entire week rebuilding our backup RAID (ironic) and converting from a 5 to a 6, and now spending the last night solving fstab and mdadm.conf problems, I was proud to say I got everything exactly where I wanted it as of this morning without having to ask for help (googling about a hundred times, sure, but help? nah.). But...
So of course, everything was perfect for about 12 hours. Then I had one of our drives fail out of one of our raids (we currently have 1 4 disc RAID5, 1 5 disc RAID6, and 1 3 disc RAID5, the latter of which is the one that failed). Through various attempts to unmount (kept getting the "in use" error, even though I stopped all applicable programs), I finally just decided "meh" and rebooted so that I could diagnose the problem.
Here's where the fun begins.
On reboot, the server didn't pop back up on the network. I went to the basement to find it was in Bash. Weird. Rebooted to find the "blah blah blah RAID is degraded, attempt to mount degraded RAID?" message, so I hit Y. The screen went blank and stayed that way.
Reboot. This time hit N, ended up in Bash. Could see the degraded RAID, but the failed disc wasn't showing up. Huh. Tried removing the disc entirely from the rack, rebooted. Same problem as above. So I pulled all three drives and rebooted. Success! Quickly went into mdadm.conf and fstab and #ed out the lines connected to that RAID. Then slotted the drives back in and rebooted with the non-failed drives returned to make sure everything worked properly. It all does.
So now here's the weirdness: the two healthy drives are showing up as spares in the array, and the third drive won't show up. At all. Hot swapping it in results in the system identifying that the drive is there, but ls /dev/sd* shows no new drive. Nothing, not even showing up. I thought maybe it was a mechanical issue, so I brought it upstairs and plugged it into my USB diagnostic drive caddy, and OSX recognized it immediately, and putting my ear to the drive nothing sounded amiss. So...
Was the failure partition related or mechanical? If mechanical, why can my mac identify the existence of the drive but Ubuntu not? I should also mention that this is, of course, the replacement drive we just bought to replace another dead drive (wasn't part of the array, we built this 5 from the new drive and a previous RAID 0 that we destroyed), so does it sound like the drive may be just hosed? I spent the last hour searching around but couldn't find any topics that matched or explained/resolved this...
Last edited by greyday; September 27th, 2013 at 03:14 AM.