Results 1 to 7 of 7

Thread: RAID 6 writing slow and stopping / timing out - cannot write large files to server

  1. #1
    Join Date
    Jun 2013
    Beans
    20

    RAID 6 writing slow and stopping / timing out - cannot write large files to server

    UPDATE:
    Upgraded drives on Server and installed 12.04. All seems OK now

    *************************************************

    Hi all,
    I have set up a server using ubuntu 11.04 some 2 years ago. I have 10x 2TB WD Greens running RAID6 and an IDE 500GB HDD where the OS resides, and 2 NICs bonded. The MOBO is an ASUS P5W DX Deluse, 4GB RAM and Core 2 Quad Proc, 850W PS.

    During this time I have had only one drive failure (some 3 months ago). I replaced and rebuilt array and at this time I turned on bitmap, and left it on since.

    This is the output of mdadm -D /dev/md0

    Code:
    /dev/md0:
            Version : 1.2
      Creation Time : Sun Sep 11 11:21:49 2011
         Raid Level : raid6
         Array Size : 15628095488 (14904.11 GiB 16003.17 GB)
      Used Dev Size : 1953511936 (1863.01 GiB 2000.40 GB)
       Raid Devices : 10
      Total Devices : 10
        Persistence : Superblock is persistent
    
    
      Intent Bitmap : Internal
    
    
        Update Time : Sat Jun  8 21:59:39 2013
              State : active
     Active Devices : 10
    Working Devices : 10
     Failed Devices : 0
      Spare Devices : 0
    
    
             Layout : left-symmetric
         Chunk Size : 512K
    
    
               Name : LSERVER:0  (local to host LSERVER)
               UUID : f65fe11e:2ae4eb81:1c2f1f32:7af495f1
             Events : 31031
    
    
        Number   Major   Minor   RaidDevice State
           0       8        1        0      active sync   /dev/sda1
           1       8       33        1      active sync   /dev/sdc1
           2       8       17        2      active sync   /dev/sdb1
          11       8       48        3      active sync   /dev/sdd
           4       8       65        4      active sync   /dev/sde1
           5       8       81        5      active sync   /dev/sdf1
           6       8      113        6      active sync   /dev/sdh1
           7       8      129        7      active sync   /dev/sdi1
          10       8      144        8      active sync   /dev/sdj
           9       8      161        9      active sync   /dev/sdk1
    The output of cat /proc/mdstat is

    Code:
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md0 : active raid6 sdk1[9] sdh1[6] sdj[10] sdi1[7] sdc1[1] sdd[11] sda1[0] sdb1[2] sde1[4] sdf1[5]
          15628095488 blocks super 1.2 level 6, 512k chunk, algorithm 2 [10/10] [UUUUUUUUUU]
          bitmap: 1/15 pages [4KB], 65536KB chunk
    
    
    unused devices: <none>
    Just in case, the output of ifconfig is:

    Code:
    bond0     Link encap:Ethernet  HWaddr 00:1b:fc:6a:c2:79
              inet addr:192.168.0.31  Bcast:192.168.0.255  Mask:255.255.255.0
              inet6 addr: fe80::21b:fcff:fe6a:c279/64 Scope:Link
              UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
              RX packets:62999 errors:0 dropped:0 overruns:0 frame:0
              TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:5470760 (5.4 MB)  TX bytes:468 (468.0 B)
    
    
    eth0      Link encap:Ethernet  HWaddr 00:1b:fc:6a:d1:00
              inet addr:192.168.0.198  Bcast:192.168.0.255  Mask:255.255.255.0
              inet6 addr: fe80::21b:fcff:fe6a:d100/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:21518530 errors:0 dropped:0 overruns:0 frame:0
              TX packets:5010704 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:26784067777 (26.7 GB)  TX bytes:9103414619 (9.1 GB)
              Interrupt:17
    
    
    eth1      Link encap:Ethernet  HWaddr 00:1b:fc:6a:c2:79
              UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
              RX packets:62999 errors:0 dropped:0 overruns:0 frame:0
              TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:5470760 (5.4 MB)  TX bytes:468 (468.0 B)
              Interrupt:16 Base address:0xac00
    
    
    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              inet6 addr: ::1/128 Scope:Host
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:1744 errors:0 dropped:0 overruns:0 frame:0
              TX packets:1744 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:114656 (114.6 KB)  TX bytes:114656 (114.6 KB)
    The volume is approx. 65% full. I know it is a lot of data but why would it slow down and time out over the last few months? I am prompted to write this today as it has become impossible to copy large data to it now. It runs at approx. 1~2MB write (over the network, from a Win 8 or Win 7 box).

    Thanks in advance for any assistance

    M
    Last edited by MarcusL; June 20th, 2013 at 05:37 AM.

  2. #2
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    367
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: RAID 6 writing slow and stopping / timing out - cannot write large files to serve

    It is know that bitmap causes a bit of performance decrease, but it shouldn't be this much. Anyways, I would test the speeds without bitmap as a start.

  3. #3
    Join Date
    Jun 2013
    Beans
    20

    Re: RAID 6 writing slow and stopping / timing out - cannot write large files to serve

    Hi SaturnusDJ,
    I have disabled bitmap, but effect is the same. Copy starts runing up to 40MB/s then drops off to 0MB/s and times out.

    Thanks.

  4. #4
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    367
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: RAID 6 writing slow and stopping / timing out - cannot write large files to serve

    Okay, time for some more tests.

    How is internal (not using network) transfer performing?

    What does these commands put out for every array member and array itself?
    Code:
    hdparm -tT /dev/...
    Why is it that you added array member number 10 and 11 as whole disks?

    During transfers, do some monitoring at CPU/Mem/Swap/etc...

    Check SMART for every HDD:
    Code:
    smartctl -a /dev/...

  5. #5
    Join Date
    Jun 2013
    Beans
    20

    Re: RAID 6 writing slow and stopping / timing out - cannot write large files to serve

    Hi SaturnusDJ,
    Thanks for the help.

    Not sure how I screwed up and added 10&11 as whole disks... I noticed this difference in cat /proc/mdstat. Is there a way to re-align these two disks?

    On internal copy, it does the same. The RAID6 is one big partition (15TB) and it contains 4 folders. I copied large data from one folder to another, the copy process kicks off at 90+MB/s then drops off to 0 then 30KB/s.

    By the way, I have another thread going for an issue where the server sees machines on local network OK, but does not see the Internet. I am sure these two issues are not related, but just in case! http://ubuntuforums.org/showthread.php?t=2152581

    Here's the output from your suggestions (sdg1 seems to be performing lower than other disks although it looks OK in smartctl):
    hdparm -tT /dev/...
    Code:
    /dev/sda1:
     Timing cached reads:   10552 MB in  2.00 seconds = 5280.51 MB/sec
     Timing buffered disk reads: 352 MB in  3.02 seconds = 116.70 MB/sec
    
    /dev/sdb1:
     Timing cached reads:   9662 MB in  2.00 seconds = 4835.29 MB/sec
     Timing buffered disk reads: 312 MB in  3.01 seconds = 103.51 MB/sec
    
    /dev/sdc1:
     Timing cached reads:   8970 MB in  2.00 seconds = 4488.21 MB/sec
     Timing buffered disk reads: 308 MB in  3.01 seconds = 102.22 MB/sec
    
    /dev/sde1:
     Timing cached reads:   9610 MB in  2.00 seconds = 4809.27 MB/sec
     Timing buffered disk reads: 354 MB in  3.01 seconds = 117.54 MB/sec
    
    /dev/sdf1:
     Timing cached reads:   9504 MB in  2.00 seconds = 4755.65 MB/sec
     Timing buffered disk reads: 308 MB in  3.02 seconds = 102.13 MB/sec
    
    /dev/sdg1:
     Timing cached reads:   9246 MB in  2.00 seconds = 4626.31 MB/sec
     Timing buffered disk reads: 244 MB in  3.02 seconds =  80.83 MB/sec
    
    /dev/sdh1:
     Timing cached reads:   9806 MB in  2.00 seconds = 4906.67 MB/sec
     Timing buffered disk reads: 310 MB in  3.01 seconds = 103.06 MB/sec
    
    /dev/sdi1:
     Timing cached reads:   9422 MB in  2.00 seconds = 4714.29 MB/sec
     Timing buffered disk reads: 282 MB in  3.00 seconds =  93.87 MB/sec
    
    /dev/sdj1:
     Timing cached reads:   9786 MB in  2.00 seconds = 4896.76 MB/sec
     Timing buffered disk reads: 294 MB in  3.00 seconds =  97.87 MB/sec

    smartctl -a /dev/sda1
    Code:
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green (Adv. Format) family
    Device Model:     WDC WD20EARS-00MVWB0
    Serial Number:    WD-WMAZA1234380
    Firmware Version: 51.0AB51
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:44:50 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x82) Offline data collection activity
                                            was completed without error.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (37260) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x3035) SCT Status supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   169   167   021    Pre-fail  Always       -       6516
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       776
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7454
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       774
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       247
    193 Load_Cycle_Count        0x0032   169   169   000    Old_age   Always       -       95588
    194 Temperature_Celsius     0x0022   121   097   000    Old_age   Always       -       29
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.


    smartctl -a /dev/sdb1
    Code:
    
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green family
    Device Model:     WDC WD20EADS-00R6B0
    Serial Number:    WD-WCAVY0235763
    Firmware Version: 01.00A01
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:46:00 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (42360) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x303f) SCT Status supported.
                                            SCT Error Recovery Control supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1
      3 Spin_Up_Time            0x0027   147   144   021    Pre-fail  Always       -       9608
      4 Start_Stop_Count        0x0032   096   096   000    Old_age   Always       -       4333
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       9608
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       793
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       83
    193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       4250
    194 Temperature_Celsius     0x0022   118   097   000    Old_age   Always       -       34
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.

    smartctl -a /dev/sdc1
    Code:
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green family
    Device Model:     WDC WD20EADS-00R6B0
    Serial Number:    WD-WCAVY0316492
    Firmware Version: 01.00A01
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:47:17 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (39960) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x303f) SCT Status supported.
                                            SCT Error Recovery Control supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   146   145   021    Pre-fail  Always       -       9666
      4 Start_Stop_Count        0x0032   097   097   000    Old_age   Always       -       3796
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   077   077   000    Old_age   Always       -       17430
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       831
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       108
    193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       3688
    194 Temperature_Celsius     0x0022   118   098   000    Old_age   Always       -       34
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       2
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       1
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    smartctl -a /dev/sde1
    Code:
    
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green (Adv. Format) family
    Device Model:     WDC WD20EARS-00MVWB0
    Serial Number:    WD-WMAZA4106478
    Firmware Version: 51.0AB51
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:48:22 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (37500) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x3035) SCT Status supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   253   253   021    Pre-fail  Always       -       1150
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       706
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       6311
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       704
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       193
    193 Load_Cycle_Count        0x0032   168   168   000    Old_age   Always       -       98027
    194 Temperature_Celsius     0x0022   120   100   000    Old_age   Always       -       30
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       20
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
    # 1  Short offline       Completed without error       00%      1668         -
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    smartctl -a /dev/sdf1
    Code:
    
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green family
    Device Model:     WDC WD20EADS-00R6B0
    Serial Number:    WD-WCAVY0235144
    Firmware Version: 01.00A01
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:48:53 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (41580) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x303f) SCT Status supported.
                                            SCT Error Recovery Control supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   149   146   021    Pre-fail  Always       -       9525
      4 Start_Stop_Count        0x0032   097   097   000    Old_age   Always       -       3018
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7971
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       758
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       96
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2922
    194 Temperature_Celsius     0x0022   117   097   000    Old_age   Always       -       35
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       9
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    smartctl -a /dev/sdg1
    Code:
    
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Blue EIDE family
    Device Model:     WDC WD5000AAKB-00H8A0
    Serial Number:    WD-WCASYF841013
    Firmware Version: 05.04E05
    User Capacity:    500,107,862,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:49:14 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (11160) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 131) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x3037) SCT Status supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   160   159   021    Pre-fail  Always       -       4983
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       618
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -       4560
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       616
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       135
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       618
    194 Temperature_Celsius     0x0022   113   094   000    Old_age   Always       -       34
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       5
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    smartctl -a /dev/sdh1
    Code:
    
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green family
    Device Model:     WDC WD20EADS-00R6B0
    Serial Number:    WD-WCAVY0238964
    Firmware Version: 01.00A01
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:49:32 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (41580) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x303f) SCT Status supported.
                                            SCT Error Recovery Control supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   199   199   051    Pre-fail  Always       -       70394
      3 Spin_Up_Time            0x0027   141   139   021    Pre-fail  Always       -       9916
      4 Start_Stop_Count        0x0032   096   096   000    Old_age   Always       -       4379
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       9660
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       806
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       122
    193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       4257
    194 Temperature_Celsius     0x0022   120   101   000    Old_age   Always       -       32
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   196   196   000    Old_age   Always       -       1446
    198 Offline_Uncorrectable   0x0030   200   197   000    Old_age   Offline      -       274
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   195   094   000    Old_age   Offline      -       1151
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    smartctl -a /dev/sdi1
    Code:
    
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green family
    Device Model:     WDC WD20EADS-00R6B0
    Serial Number:    WD-WCAVY0805261
    Firmware Version: 01.00A01
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:49:53 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: FAILED!
    Drive failure expected in less than 24 hours. SAVE ALL DATA.
    See vendor-specific Attribute list for failed Attributes.
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (43200) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x303f) SCT Status supported.
                                            SCT Error Recovery Control supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   197   197   051    Pre-fail  Always       -       302296
      3 Spin_Up_Time            0x0027   144   140   021    Pre-fail  Always       -       9791
      4 Start_Stop_Count        0x0032   097   097   000    Old_age   Always       -       3704
      5 Reallocated_Sector_Ct   0x0033   138   138   140    Pre-fail  Always   FAILING_NOW 496
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   081   081   000    Old_age   Always       -       14426
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       814
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       676
    193 Load_Cycle_Count        0x0032   125   125   000    Old_age   Always       -       226893
    194 Temperature_Celsius     0x0022   119   099   000    Old_age   Always       -       33
    196 Reallocated_Event_Count 0x0032   001   001   000    Old_age   Always       -       353
    197 Current_Pending_Sector  0x0032   198   196   000    Old_age   Always       -       769
    198 Offline_Uncorrectable   0x0030   200   198   000    Old_age   Offline      -       84
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1698
    200 Multi_Zone_Error_Rate   0x0008   195   144   000    Old_age   Offline      -       1138
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    smartctl -a /dev/sdj1
    Code:
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green family
    Device Model:     WDC WD20EADS-00R6B0
    Serial Number:    WD-WCAVY0199369
    Firmware Version: 01.00A01
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:50:09 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (42360) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x303f) SCT Status supported.
                                            SCT Error Recovery Control supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   139   136   021    Pre-fail  Always       -       10016
      4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2367
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       9710
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1542
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       104
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1502
    194 Temperature_Celsius     0x0022   120   098   000    Old_age   Always       -       32
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    smartctl -a /dev/sdk1
    Code:
    
    smartctl 5.40 2010-07-12 r3124 [x86_64-unknown-linux-gnu] (local build)
    Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
    
    
    === START OF INFORMATION SECTION ===
    Model Family:     Western Digital Caviar Green family
    Device Model:     WDC WD20EADS-00R6B0
    Serial Number:    WD-WCAVY0324756
    Firmware Version: 01.00A01
    User Capacity:    2,000,398,934,016 bytes
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:   8
    ATA Standard is:  Exact ATA specification draft version not indicated
    Local Time is:    Mon Jun 10 08:50:27 2013 EST
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled
    
    
    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    
    General SMART Values:
    Offline data collection status:  (0x84) Offline data collection activity
                                            was suspended by an interrupting command from host.
                                            Auto Offline Data Collection: Enabled.
    Self-test execution status:      (   0) The previous self-test routine completed
                                            without error or no self-test has ever
                                            been run.
    Total time to complete Offline
    data collection:                 (42360) seconds.
    Offline data collection
    capabilities:                    (0x7b) SMART execute Offline immediate.
                                            Auto Offline data collection on/off support.
                                            Suspend Offline collection upon new
                                            command.
                                            Offline surface scan supported.
                                            Self-test supported.
                                            Conveyance Self-test supported.
                                            Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
                                            power-saving mode.
                                            Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
                                            General Purpose Logging supported.
    Short self-test routine
    recommended polling time:        (   2) minutes.
    Extended self-test routine
    recommended polling time:        ( 255) minutes.
    Conveyance self-test routine
    recommended polling time:        (   5) minutes.
    SCT capabilities:              (0x303f) SCT Status supported.
                                            SCT Error Recovery Control supported.
                                            SCT Feature Control supported.
                                            SCT Data Table supported.
    
    
    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   148   145   021    Pre-fail  Always       -       9575
      4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2415
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   087   087   000    Old_age   Always       -       9848
     10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1572
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       108
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1996
    194 Temperature_Celsius     0x0022   120   092   000    Old_age   Always       -       32
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1
    
    
    SMART Error Log Version: 1
    No Errors Logged
    
    
    SMART Self-test log structure revision number 1
    No self-tests have been logged.  [To run self-tests, use: smartctl -t]
    
    
    
    
    SMART Selective self-test log data structure revision number 1
     SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.
    Thank you for taking the time to help!

    M
    Last edited by MarcusL; June 10th, 2013 at 12:05 AM.

  6. #6
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    367
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: RAID 6 writing slow and stopping / timing out - cannot write large files to serve

    Re-adding the disks is possible, but the risk for just this is too high. (Read along.)

    So it looks like it's not a network problem.
    Hdparm looks fine also.

    Then comes the SMART data:
    Multiple disks have some small errors (Disks with UDMA_CRC_Error_Count should be getting a new SATA cable)...but /dev/sdi is dying, and /dev/sdh might be following soon.

    In my opinion /dev/sdi should be replaced immediately, but this brings the risk of failing /dev/sdh.

    Edit:
    Just realized it's RAID6 here, so two disks may fail, making it a little bit less dangerous.

    Do you have backups?
    Last edited by SaturnusDJ; June 10th, 2013 at 09:09 AM.

  7. #7
    Join Date
    Jun 2013
    Beans
    20

    Re: RAID 6 writing slow and stopping / timing out - cannot write large files to serve

    Hi SaturnusDJ,
    So you think this is the reason for the time outs? Makes sense if it is... I have seen a couple of times only I/O errors on sdi... but never sdh.

    Hmmm how to find enoug space to back up 12TB of data!

    Thanks again for the help!

    M

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •