Disk write speed reduced by half over night

[SaturnusDJ]

After a night of sleep the write speed of my mdadm array is around 110MB/s, while before going to bed it was arround 220MB/s.

The used disks don't enter stand-by or a power saving state, nor do the s-ata links as far as I know.

What could have done this?

Found the cause here: http://ubuntuforums.org/showthread.p...7#post12580567

[SaturnusDJ]

Details:

1. Rebooting doesn't fix this problem. Powering off and on again does!
2. Sending the drives to stand-by/sleep (hdparm -y /-Y) doesn't fix or cause this problem.

Who has an idea please??

[rnerwein]

hi
it's just a feeling. for me it looks like a fully filled cache. you can try: creat a file (smaller than your cache) --> it's very, very fast. but then type: time sync ups !
ciao

[SaturnusDJ]

Thanks for the reply.
I don't understand how this will debug or solve the problem, could you explain in more detail?

This is the script I use to test:

dd if=/dev/zero of=/md/testfile2 bs=1M count=10000
dd if=/md/testfile2 of=/dev/null bs=1M
hdparm -t /dev/md0

So, three small tests: 1. Write 10GB to array. 2. Read 10GB from array. 3. Test read speed of array.

Output when used with time sync:

# time ./bench 
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 89.592 s, 117 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 37.5878 s, 279 MB/s

/dev/md0:
 Timing buffered disk reads: 758 MB in  3.00 seconds = 252.39 MB/sec

real	2m15.037s
user	0m0.144s
sys	0m43.979s

[SaturnusDJ]

Found out something new:

I stopped the array and tested the disks individually.
/dev/sdd had a write of around 70MB/s. Other two have around 140MB/s. This combined in the array gives ~120MB/s. (See above.) The array normally has ~220MB/s.

I unplugged /dev/sdd data, and plugged back in, still slow speed.
Then I unplugged /dev/sdd data and power, and plugged back in, speeds where normal again!


How is this possible, I don't understand.

[tgalati4]

Try moving cables around. If the problem moves with the cable, then you might get a new cable to see if that fixes the problem. If /dev/sdd remains slow, regardless of cable used, then it's possible that the on-board SATA controller is failing and it resorts to a slower speed to maintain connectivity.

smartctl -a /dev/sdd

might show what data mode it is using.

[SaturnusDJ]

Thanks, good idea. I'll use a different cable at a different port.

After booting, kernel always reports "SATA link up 3.0 Gbps." During the night, nothing is added to the logs, so I assume link speed is still the same.

S.M.A.R.T. is clean. Nothing weird or different between the drives when looking at hdparm/smartctl information. Cron disabled, no fix.


After lots of google searched, I tried one more time and found this: http://forums.overclockers.co.uk/sho...php?p=22512444
Sound like my problem. I'll continue debugging first.

[sdowney717]

Cosmic radiation killed your chips.
Reading the other forum thread indicates the drive is faulty.
Electromagnetic radiation permeates everything.
http://www.newscientist.com/blog/tec...lerts-for.html
Drives have firmware memory.

[SaturnusDJ]

Thanks for the reply. The problem with that is a power off (not reboot) fixes the problem temporary.

To my surprise I just found out today all drives are slow when tested individually. Search 1-2 hours for clues...nothing.
So I created a partition in the middle of the disk and the speeds scaled down with this (as normally with a hdd). At the moment I can't think how to conclude something out of that information, but maybe someone else can.

Power off -> power on -> fixed. But of course, after the next night the writes will be slow again. So I guess changing cables/ports isn't going to do anything.

It's so hard to debug this...

[sdowney717]

I am not convinced it is not a firmware corruption issue. the drive is mechanically ok and also somewhat electronically ok as smart data shows no errors.
The input output chips and internal memory buffers, the data is perhaps being corrupted and then has to be internally corrected on the fly as it reads and writes.
Or some IO pipeline process is being throttled.
Sometimes you can upload new firmware to drives.
Firmware software tells the drive how to function. Bad firmware means it wont run or the drive will run poorly.
If the drives are put into another PC and have the same behavior, it is the drives problem.
Any chance you can do that or boot another OS to further test them?