10.04 server x64 with RAID 5.

**jus71n742** · March 3rd, 2014

I built a media server about 4 years ago. It has 4, 1 TB HDD's in a RAID 5. Recently (After my utility board failed to keep power on for 30 days straight again...) I received an error that the home partition isn't available. I haven't had much luck getting this thing rebuilt. I think one of the drives are bad, so I have ordered a replacement drive for that.
My main concern is this: How can I get into the RAID in order to maybe pull the data off (Around 1TB) and move it to an external HDD until I get everything rebuilt JUST IN CASE everything goes south. I have been out of practice with Ubuntu/*nix for a little while since changing jobs.

**rubylaser** · March 4th, 2014

Hello, the easiest way to backup your data would be to boot the LiveCD, install mdadm, assemble your, array and mount it and the external hard drive.

Code:

sudo -i
apt-get update && apt-get install mdadm -y
mdadm --assemble /dev/md0 /dev/sd[abcd]1
mkdir /raid-array
mount /dev/md0 /storage
mkdir /external-drive
mount /dev/sde1 /external-drive
rsync -av --progress /raid-array/ /external-drive/

Obviously, these drive letters are just guesses on my part, so you will want to verify that the disks you assemble your array from are in fact the disks with mdadm superblock metadata on them.

**jus71n742** · March 5th, 2014

Thank you for the reply, I located the mdadm idea and tried it in a clearly less sophisticated way and received the issue of

Code:

assembled from 2 drives - not enough to start the array

So I am guessing that possibly have 2 drives bad instead of the one that I knew was weak (New drive sitting beside me now to replace it) when I built the Array. What is an easier way to check each disc to test the health of each drive in the array? I ran

Code:

 sudo badblocks -sv /dev/sda
Checking blocks 0 to 976762583
Checking for bad blocks (read-only test): done                                
Pass completed, 0 bad blocks found.

But as you can see there wasn't much issue with the one drive it checked.

**rubylaser** · March 5th, 2014

I would check the smart data on each disk and very the mdadm metadata counters on each disk to see if the match or are close.

Code:

sudo -i
apt-get install smartmontools -y
smartctl -a /dev/sda
smartctl -a /dev/sdb
smartctl -a /dev/sdc
smartctl -a /dev/sdd

Code:

mdadm -E /dev/sd[abcd]1

The drive letters above are only as an example. It would be nice if you would post the results of these commands back here, so we could help you troubleshoot this.

**jus71n742** · March 25th, 2014

here is sda

Code:

root@ubuntu:~# smartctl -a /dev/sda
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     Hitachi HDS721010CLA332
Serial Number:    JP2911HQ17ZBKA
Firmware Version: JP4OA39C
User Capacity:    1,000,204,886,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Mar 25 20:17:33 2014 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		 (9929) seconds.
Offline data collection
capabilities: 			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 166) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   099   099   016    Pre-fail  Always       -       3
  2 Throughput_Performance  0x0005   135   135   054    Pre-fail  Offline      -       96
  3 Spin_Up_Time            0x0007   120   120   024    Pre-fail  Always       -       315 (Average 314)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       150
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   130   130   020    Pre-fail  Offline      -       35
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       20670
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       150
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       200
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       200
194 Temperature_Celsius     0x0002   193   193   000    Old_age   Always       -       31 (Lifetime Min/Max 17/45)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 0
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

sdb

Code:

smartctl -a /dev/sdb
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     ST31000528AS
Serial Number:    5VP3XSAA
Firmware Version: CC38
User Capacity:    1,000,204,886,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Mar 25 20:18:42 2014 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		 ( 609) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 182) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x103f)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   115   099   006    Pre-fail  Always       -       96972989
  3 Spin_Up_Time            0x0003   095   095   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       286
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   072   060   030    Pre-fail  Always       -       19997738
  9 Power_On_Hours          0x0032   077   077   000    Old_age   Always       -       20684
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       143
183 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
184 Unknown_Attribute       0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Unknown_Attribute       0x0032   100   099   000    Old_age   Always       -       3
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   071   055   045    Old_age   Always       -       29 (Lifetime Min/Max 21/29)
194 Temperature_Celsius     0x0022   029   045   000    Old_age   Always       -       29 (0 17 0 0)
195 Hardware_ECC_Recovered  0x001a   022   013   000    Old_age   Always       -       96972989
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       109143708946997
241 Unknown_Attribute       0x0000   100   253   000    Old_age   Offline      -       3116039485
242 Unknown_Attribute       0x0000   100   253   000    Old_age   Offline      -       4194999855

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

sdc

Code:

root@ubuntu:~# smartctl -a /dev/sdc
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     Hitachi HDS721010CLA332
Serial Number:    JP2911HQ2MEGBH
Firmware Version: JP4OA39C
User Capacity:    1,000,204,886,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Mar 25 20:19:32 2014 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		 (10341) seconds.
Offline data collection
capabilities: 			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 172) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   094   094   016    Pre-fail  Always       -       983048
  2 Throughput_Performance  0x0005   134   134   054    Pre-fail  Offline      -       103
  3 Spin_Up_Time            0x0007   119   119   024    Pre-fail  Always       -       317 (Average 320)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       121
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   130   130   020    Pre-fail  Offline      -       35
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       20555
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       121
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       195
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       195
194 Temperature_Celsius     0x0002   206   206   000    Old_age   Always       -       29 (Lifetime Min/Max 18/45)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 0
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

sdd

Code:

oot@ubuntu:~# smartctl -a /dev/sdd
smartctl version 5.38 [x86_64-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     Hitachi HDS721010CLA332
Serial Number:    JP2911HQ2HKA2H
Firmware Version: JP4OA39C
User Capacity:    1,000,204,886,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Mar 25 20:20:29 2014 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		 (10576) seconds.
Offline data collection
capabilities: 			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 176) minutes.
SCT capabilities: 	       (0x003d)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   095   095   016    Pre-fail  Always       -       851972
  2 Throughput_Performance  0x0005   135   135   054    Pre-fail  Offline      -       99
  3 Spin_Up_Time            0x0007   120   120   024    Pre-fail  Always       -       316 (Average 315)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       121
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   132   132   020    Pre-fail  Offline      -       34
  9 Power_On_Hours          0x0012   098   098   000    Old_age   Always       -       20574
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       121
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       194
193 Load_Cycle_Count        0x0012   100   100   000    Old_age   Always       -       194
194 Temperature_Celsius     0x0002   200   200   000    Old_age   Always       -       30 (Lifetime Min/Max 17/46)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 0
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

**jus71n742** · March 25th, 2014

Here is mdadm output
Sorry for the delayed reply...been busy at work and around the house.

So I am guessing I do have 2 failed drives and until at least one is saved I will not be able to recover anything? I have one HDD here ready to install.....but if I install it will the array work again or am I just doomed and will have to redo everything anyways?

Code:

root@ubuntu:~# mdadm -E /dev/sd[abcd]1
mdadm: No md superblock detected on /dev/sda1.
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : ec563622:818fec90:cb9462bf:680dd3fd
  Creation Time : Mon Jan 10 04:30:19 2011
     Raid Level : raid5
  Used Dev Size : 959183808 (914.75 GiB 982.20 GB)
     Array Size : 2877551424 (2744.25 GiB 2946.61 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Mon Jun 10 03:51:09 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 22e9f3ab - correct
         Events : 1358544

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
/dev/sdc1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : ec563622:818fec90:cb9462bf:680dd3fd
  Creation Time : Mon Jan 10 04:30:19 2011
     Raid Level : raid5
  Used Dev Size : 959183808 (914.75 GiB 982.20 GB)
     Array Size : 2877551424 (2744.25 GiB 2946.61 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Mon Jun 10 03:47:09 2013
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 22e9f2ad - correct
         Events : 1358540

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       33        2      active sync   /dev/sdc1

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       49        3      active sync   /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : ec563622:818fec90:cb9462bf:680dd3fd
  Creation Time : Mon Jan 10 04:30:19 2011
     Raid Level : raid5
  Used Dev Size : 959183808 (914.75 GiB 982.20 GB)
     Array Size : 2877551424 (2744.25 GiB 2946.61 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0

    Update Time : Mon Jun 10 03:47:09 2013
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 22e9f2bf - correct
         Events : 1358540

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       49        3      active sync   /dev/sdd1

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       33        2      active sync   /dev/sdc1
   3     3       8       49        3      active sync   /dev/sdd1

**jus71n742** · April 4th, 2014

So are my assumptions correct? Not enough to rebuild anything and I will have to redo everything once the new drives are placed in?

**lukeiamyourfather** · April 4th, 2014

Originally Posted by jus71n742

So are my assumptions correct? Not enough to rebuild anything and I will have to redo everything once the new drives are placed in?

Correct, two drive failures in a RAID 5 array means total loss of data. If you need the data from the array look into data recovery services which will be expensive. If you don't need the data just create a new array with the old functional disks and new disks. Consider an array that offers more redundancy like RAID 6, or RAID Z2 with ZFS.

**rubylaser** · April 5th, 2014

I wouldn't say everything is lost yet. Can you let me see the superblock metadata on /dev/sda3 like this?

Code:

mdadm -E /dev/sda3

If that disk's event counter matches /dev/sd[cd]1, then you should be able to assemble with the three sync'd disks like this.

Code:

mdadm --assemble --force /dev/md0 /dev/sda3 /dev/sd[cd]1

Another option for bulk home storage is SnapRAID if you do end up needing to start over. If you'd like to read more, I have a tutorial to set it up on my website.