Hey all again,
I finally got all the pieces to build this out, and settled back home. When I initially got the drives, I ran a long test and everything came back fine. Today after getting the rest of the hardware and bringing it overseas before setting it up into a ZFS array, I did another long test and I am seeing a drive fail out and I can't tell if it is the HBA / SATA card, or the drive itself. Attached is a log file of ata12 and /dev/sdf: https://pastebin.ubuntu.com/p/yFJpTMHSqh/
I was trying to make this a low-power consumption server, so it could be that powertop or a BIOS configuration caused this, but, I just wanted to be absolutely sure that it wasn't the drive itself failing before I dig in any further.
Apart from a restart, what is the best way to "power on" a drive that has been powered off in the error message above, I assume due to udisksctl? I also saw another drive fail with a "SmartSelfTestStatus: Interrupted" message, so my hope here is that the actual NVME to ACHI/SATA adapter is failing, and needs replacement, and not the drives itself.
The smartctl test results of the Interrupted drive is shown below:
Code:
=== START OF READ SMART DATA SECTION ===SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x80) Offline data collection activity
was never started.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 37) The self-test routine was interrupted
by the host with a hard or soft reset.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: (2621) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate PO-R-- 100 100 001 - 0
2 Throughput_Performance --S--- 148 148 000 - 48
3 Spin_Up_Time POS--- 086 086 001 - 328 (Average 295)
4 Start_Stop_Count -O--C- 100 100 000 - 12
5 Reallocated_Sector_Ct PO--CK 100 100 001 - 0
7 Seek_Error_Rate -O-R-- 091 091 000 - 6
8 Seek_Time_Performance --S--- 140 140 000 - 15
9 Power_On_Hours -O--C- 100 100 000 - 54
10 Spin_Retry_Count -O--C- 100 100 000 - 0
12 Power_Cycle_Count -O--CK 100 100 000 - 10
22 Unknown_Attribute PO---K 100 100 025 - 6553700
71 Unknown_Attribute P----- 100 100 001 - 0
90 Unknown_Attribute P---CK 100 100 001 - 493921239040
192 Power-Off_Retract_Count -O--CK 100 100 000 - 19
193 Load_Cycle_Count -O--C- 100 100 000 - 19
194 Temperature_Celsius -O---- 065 065 000 - 31 (Min/Max 18/39)
196 Reallocated_Event_Count -O--CK 100 100 000 - 0
197 Current_Pending_Sector -O---K 100 100 000 - 0
198 Offline_Uncorrectable ---R-- 100 100 000 - 0
199 UDMA_CRC_Error_Count -O-R-- 100 100 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
General Purpose Log Directory Version 1
SMART Log Directory Version 1 [multi-sector log support]
Address Access R/W Size Description
0x00 GPL,SL R/O 1 Log Directory
0x01 SL R/O 1 Summary SMART error log
0x02 SL R/O 1 Comprehensive SMART error log
0x03 GPL R/O 1 Ext. Comprehensive SMART error log
0x04 GPL R/O 256 Device Statistics log
0x04 SL R/O 255 Device Statistics log
0x06 SL R/O 1 SMART self-test log
0x07 GPL R/O 1 Extended self-test log
0x08 GPL R/O 2 Power Conditions log
0x09 SL R/W 1 Selective self-test log
0x0c GPL R/O 19043 Pending Defects log
0x10 GPL R/O 1 NCQ Command Error log
0x11 GPL R/O 1 SATA Phy Event Counters log
0x12 GPL R/O 1 SATA NCQ Non-Data log
0x13 GPL R/O 1 SATA NCQ Send and Receive log
0x15 GPL R/W 1 Rebuild Assist log
0x21 GPL R/O 1 Write stream error log
0x22 GPL R/O 1 Read stream error log
0x24 GPL R/O 256 Current Device Internal Status Data log
0x25 GPL R/O 256 Saved Device Internal Status Data log
0x2f GPL - 1 Set Sector Configuration
0x30 GPL,SL R/O 9 IDENTIFY DEVICE data log
0x80-0x9f GPL,SL R/W 16 Host vendor specific log
0xb7 SL VS 1 Device vendor specific log
0xd8-0xd9 GPL,SL VS 1 Device vendor specific log
0xe0 GPL,SL R/W 1 SCT Command/Status
0xe1 GPL,SL R/W 1 SCT Data Transfer
SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
No Errors Logged
SMART Error Log Version: 1
No Errors Logged
SMART Extended Self-test Log Version: 1 (1 sectors)
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 50% 50 -
# 2 Short offline Completed without error 00% 35 -
# 3 Extended offline Completed without error 00% 30 -
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 50% 50 -
# 2 Short offline Completed without error 00% 35 -
# 3 Extended offline Completed without error 00% 30 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
SCT Status Version: 3
SCT Version (vendor specific): 256 (0x0100)
Device State: Active (0)
Current Temperature: 31 Celsius
Power Cycle Min/Max Temperature: 22/34 Celsius
Lifetime Min/Max Temperature: 18/39 Celsius
Under/Over Temperature Limit Count: 0/0
SMART Status: 0xc24f (PASSED)
Minimum supported ERC Time Limit: 70 (7,0 seconds)
SCT Temperature History Version: 2
Temperature Sampling Period: 1 minute
Temperature Logging Interval: 1 minute
Min/Max recommended Temperature: 0/60 Celsius
Min/Max Temperature Limit: -40/70 Celsius
Temperature History Size (Index): 128 (53)
Index Estimated Time Temperature Celsius
54 2024-01-24 12:24 31 ************
55 2024-01-24 12:25 31 ************
56 2024-01-24 12:26 31 ************
57 2024-01-24 12:27 32 *************
58 2024-01-24 12:28 31 ************
... ..( 96 skipped). .. ************
27 2024-01-24 14:05 31 ************
28 2024-01-24 14:06 32 *************
29 2024-01-24 14:07 31 ************
30 2024-01-24 14:08 32 *************
... ..( 20 skipped). .. *************
51 2024-01-24 14:29 32 *************
52 2024-01-24 14:30 31 ************
53 2024-01-24 14:31 31 ************
SCT Error Recovery Control:
Read: 70 (7,0 seconds)
Write: 70 (7,0 seconds)
Device Statistics (GP Log 0x04)
Page Offset Size Value Flags Description
0x01 ===== = = === == General Statistics (rev 1) ==
0x01 0x008 4 10 --- Lifetime Power-On Resets
0x01 0x010 4 54 --- Power-on Hours
0x01 0x018 6 0 --- Logical Sectors Written
0x01 0x020 6 0 --- Number of Write Commands
0x01 0x028 6 47049 --- Logical Sectors Read
0x01 0x030 6 1752 --- Number of Read Commands
0x01 0x038 6 197655650 --- Date and Time TimeStamp
0x03 ===== = = === == Rotating Media Statistics (rev 1) ==
0x03 0x008 4 54 --- Spindle Motor Power-on Hours
0x03 0x010 4 54 --- Head Flying Hours
0x03 0x018 4 19 --- Head Load Events
0x03 0x020 4 0 --- Number of Reallocated Logical Sectors
0x03 0x028 4 0 --- Read Recovery Attempts
0x03 0x030 4 28 --- Number of Mechanical Start Failures
0x04 ===== = = === == General Errors Statistics (rev 1) ==
0x04 0x008 4 0 --- Number of Reported Uncorrectable Errors
0x04 0x010 4 0 --- Resets Between Cmd Acceptance and Completion
0x05 ===== = = === == Temperature Statistics (rev 1) ==
0x05 0x008 1 31 --- Current Temperature
0x05 0x010 1 32 N-- Average Short Term Temperature
0x05 0x018 1 - N-- Average Long Term Temperature
0x05 0x020 1 39 --- Highest Temperature
0x05 0x028 1 18 --- Lowest Temperature
0x05 0x030 1 37 N-- Highest Average Short Term Temperature
0x05 0x038 1 25 N-- Lowest Average Short Term Temperature
0x05 0x040 1 - N-- Highest Average Long Term Temperature
0x05 0x048 1 - N-- Lowest Average Long Term Temperature
0x05 0x050 4 0 --- Time in Over-Temperature
0x05 0x058 1 60 --- Specified Maximum Operating Temperature
0x05 0x060 4 0 --- Time in Under-Temperature
0x05 0x068 1 0 --- Specified Minimum Operating Temperature
0x06 ===== = = === == Transport Statistics (rev 1) ==
0x06 0x008 4 22 --- Number of Hardware Resets
0x06 0x010 4 6 --- Number of ASR Events
0x06 0x018 4 0 --- Number of Interface CRC Errors
0xff ===== = = === == Vendor Specific Statistics (rev 1) ==
0xff 0x040 7 0 --- Vendor Specific
0xff 0x048 7 0 --- Vendor Specific
0xff 0x050 7 0 --- Vendor Specific
0xff 0x058 7 0 --- Vendor Specific
0xff 0x060 7 0 --- Vendor Specific
0xff 0x068 7 0 --- Vendor Specific
0xff 0x070 7 0 --- Vendor Specific
0xff 0x078 7 0 --- Vendor Specific
0xff 0x080 7 0 --- Vendor Specific
|||_ C monitored condition met
||__ D supports DSN
|___ N normalized value
Pending Defects log (GP Log 0x0c)
No Defects Logged
SATA Phy Event Counters (GP Log 0x11)
ID Size Value Description
0x0001 2 0 Command failed due to ICRC error
0x0002 2 0 R_ERR response for data FIS
0x0003 2 0 R_ERR response for device-to-host data FIS
0x0004 2 0 R_ERR response for host-to-device data FIS
0x0005 2 0 R_ERR response for non-data FIS
0x0006 2 0 R_ERR response for device-to-host non-data FIS
0x0007 2 0 R_ERR response for host-to-device non-data FIS
0x0008 2 0 Device-to-host non-data FIS retries
0x0009 2 65535+ Transition from drive PhyRdy to drive PhyNRdy
0x000a 2 3 Device-to-host register FISes sent due to a COMRESET
0x000b 2 0 CRC errors within host-to-device FIS
0x000d 2 0 Non-CRC errors within host-to-device FIS
Bookmarks