ATA disk timing issue - results in 30 second freeze

[Doug S]

The issue herein is carried forward from late in the 12.04 release cycle. The issue persists with the server 13.04 development system. I have made progress on the issue to the point of isolating it down to udev revision 2760

What am I looking for? Someone else with the same issue.

The issue: On a very pathetic computer with an old style ATA type disk and under intensive disk I/O, occasionally the system will lock up for 30 seconds. Example kern.log entry:

Nov 20 15:31:10 test-smy kernel: [ 7493.285721] ata1: lost interrupt (Status 0x58)
Nov 20 15:31:10 test-smy kernel: [ 7493.289578] ata1: drained 65536 bytes to clear DRQ
Nov 20 15:31:10 test-smy kernel: [ 7493.292936] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Nov 20 15:31:10 test-smy kernel: [ 7493.295863] ata1.00: failed command: READ DMA
Nov 20 15:31:10 test-smy kernel: [ 7493.297694] ata1.00: cmd c8/00:20:48:86:7c/00:00:00:00:00/e0 tag 0 dma 16384 in
Nov 20 15:31:10 test-smy kernel: [ 7493.297694]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Nov 20 15:31:10 test-smy kernel: [ 7493.304284] ata1.00: status: { DRDY }
Nov 20 15:31:10 test-smy kernel: [ 7493.308525] ata1: soft resetting link
Nov 20 15:31:10 test-smy kernel: [ 7493.478162] ata1.00: configured for MWDMA2
Nov 20 15:31:10 test-smy kernel: [ 7493.478243] ata1.00: device reported invalid CHS sector 0
Nov 20 15:31:10 test-smy kernel: [ 7493.478365] sd 0:0:0:0: [sda]
Nov 20 15:31:10 test-smy kernel: [ 7493.478394] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 20 15:31:10 test-smy kernel: [ 7493.478428] sd 0:0:0:0: [sda]
Nov 20 15:31:10 test-smy kernel: [ 7493.478451] Sense Key : Aborted Command [current] [descriptor]
Nov 20 15:31:10 test-smy kernel: [ 7493.478493] Descriptor sense data with sense descriptors (in hex):
Nov 20 15:31:10 test-smy kernel: [ 7493.478514]         72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
Nov 20 15:31:10 test-smy kernel: [ 7493.478633]         00 00 00 00
Nov 20 15:31:10 test-smy kernel: [ 7493.478693] sd 0:0:0:0: [sda]
Nov 20 15:31:10 test-smy kernel: [ 7493.478724] Add. Sense: No additional sense information
Nov 20 15:31:10 test-smy kernel: [ 7493.478762] sd 0:0:0:0: [sda] CDB:
Nov 20 15:31:10 test-smy kernel: [ 7493.478782] Read(10): 28 00 00 7c 86 48 00 00 20 00
Nov 20 15:31:10 test-smy kernel: [ 7493.478890] end_request: I/O error, dev sda, sector 8160840
Nov 20 15:31:10 test-smy kernel: [ 7493.483030] ata1: EH complete

I realize that typically these types of errors indicate a failing disk. Such is not the case here, the issue has been repeated with multiple disks, cd-rom drives, and cabling. I can also create or eliminate the issue with udev compiled with / without the revision 2760 (or 2761) change (well, I did that for 12.04, but not yet for 13.04).

Disclaimer: The problem computer is underpowered, and that might be the root issue. I have not been able to repeat the issue on any of my other computers.

References:
https://bugs.launchpad.net/ubuntu/+s...ev/+bug/986654
http://www.smythies.com/~doug/networ...ace/index.html
http://ubuntuforums.org/showthread.php?t=1958838

[ventrical]

Disclaimer: The problem computer is underpowered, and that might be the root issue. I have not been able to repeat the issue on any of my other computers.

Sounds like a good call.

I am just curious as to your hardware config info. I have had older machines cough when I have too many hdds attatched. Also 4500rpm drives could stall.. old Maxtors.. even 7200rpms.WDs Some hdds really draw a lot of power after time.

[Doug S]

Sorry, I should have said "only 200Mhz CPU" when I said "underpowered". As far as I know the power supply is fine.

The hardware config. (note: the memory information is actually incorrect in that lshw -html listing. see also here.)

I probably should just give up and toss the computer. However I liked contributing via minimum requirements testing with it. My thinking was always that if testing passes with this computer, then we know for sure one that actually meets the minimum requirements (300Mhz) will be O.K. However, conversely, we know nothing if mine fails testing, as it started to really late in the 12.04 cycle.

[ventrical]

Yepper .. the traffic cops are going WAIT, WAIT, WAIT.

[zika]

I would try to look in http://ubuntuforums.org/showthread.php?t=2086325 and try to see if there is a solution of Your problem also. Since I do not have HW to test it I'll leave i to You... File mentioned there is a good place to start in my opinion and I'm very confidnt since I was looking for a solution for quite a long time...

[ventrical]

Thats an Ultra ATA so there may be a conflict with your 33MHz IDE bus and 133Mhz hdd clock?. I am not sure if it would help to put an 80strand/40pin cable on that but you could try ... or if there is a clock setting for UltraATA in your BIOS .. I know some of the older machines would have that option in the BIOS.

EDIT: or you can check to see if that PC supports UDMA , choose 4, or if PIO, then 3 or 4.

[Doug S]

@zika: Thanks for that reference . I am so dense. Previously when I tried to revert the single line rule change of udev rev 2760, I did NOT do:

sudo update-initramfs -u

and so I thought that quicker than complete re-compile method did not work. I am re-testing it now.

@ventical: another HD I have tried is ATA-100. I have done tests with multple different 80 pin cables and ... I'll look at bios settings after above tests are done (usually about 4 hours is enough to know for sure).

Summary of the issue introduced by udev rev 2760 (and the same line was modified again in rev 2761): The issue can be eliminated be reverting the single line rules change of /lib/udev/rules.d/60-persistent-storage.rules to the rev 2759 and prior state.
Specifically change this:

ACTION=="add", ATTR{removable}=="1", ATTR{events_poll_msecs}=="-1", ATTR{events_poll_msecs}="2000"

Back to this:

ACTION=="add", KERNEL=="sr*", ATTR{events_poll_msecs}=="0", ATTR{events_poll_msecs}="2000"

And yes, those rules are with respect to removeable storage, yet the issue is with non-removeable storage. The other workaround I have found is to unplug the CD-ROM drive, if I don't need it.

It took me quite a lot of time (months) to get this issue isolated down to this single line rule change.

So, I am questioning if the rule change should have been made in the first place, as it broke my system. Yes, there is a workaround, but I try do this work from the perspective of a inexperienced user that just wants to install Ubuntu server edition and it works.

[ventrical]

@zika: Thanks for that reference . I am so dense. Previously when I tried to revert the single line rule change of udev rev 2760, I did NOT do:

sudo update-initramfs -u

and so I thought that quicker than complete re-compile method did not work. I am re-testing it now.


So, I am questioning if the rule change should have been made in the first place, as it broke my system. Yes, there is a workaround, but I try do this work from the perspective of a inexperienced user that just wants to install Ubuntu server edition and it works.

If you're dense , then I'm denser hehehaha .. hey .. Dense and Denser .. new theme for a cyber movie .. no jk... but yes.. some times the linux command line interface is all new to me, even after 2 years of it. 25 years of Microsoft really took it's toll on me.

I am pretty well a novice Ubuntu user but I have lots of hardware experience and my instincts always lead me there.

[dino99]

its a good idea to start with this test:

http://support.asus.com/powersupply.aspx

(even if there is no Asus hardware)

[Doug S]

O.K. so now I have tested on 13.04 with: That one rule reverted back to rev 2759 state, works fine; Rule restored to rev 2761 state, has problems; Rule reverted back to rev 2759 state, works fine; Rule restored to rev 2761 state, has problems; Rules changed as per the link that zika gave, has problems.

I also checked my bois for the settings ventrical suggested, and they don't exist in my bios.

I remain convinced that there is a subtle timing issue or race condition introduced by udev rev 2760 (and 2761), perhaps worsed on my computer because it is so pathetic, but still a failure window.

I am looking for anyone else that has experienced this issue. The problem was introduced with that one line rule change in udev 175-0ubuntu7 somewhere around late March or early April 2012, and continues today with udev 175-0ubuntu13.