Page 3 of 4 FirstFirst 1234 LastLast
Results 21 to 30 of 37

Thread: Disk write speed reduced by half over night

  1. #21
    Join Date
    Feb 2007
    Location
    West Hills CA
    Beans
    8,371
    Distro
    Ubuntu 12.10 Quantal Quetzal

    Re: Disk write speed reduced by half over night

    If you bought your 3 drives at the same time and the serial numbers are close, then it could be a production problem. The fact that you have run many throughput tests seems to indicate that these drives are not performing as you expect. Where these new, retail disk drives, or were they refurbished, special-deal from Fry's Electronics?
    -------------------------------------
    Oooh Shiny: PopularPages

    Unumquodque potest reparantur. Patientia sit virtus.

  2. #22
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    369
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Disk write speed reduced by half over night

    They are bought at the same time, however the serialnumbers are not from the same range.
    As far as I know the disks were new. That's how they were sold. S.M.A.R.T. attributes were all zero.

    I just benched the other pc after a night of sleep, and the speeds are fine!
    I don't want to make a wrong conclusion, so debugging doesn't stop here. A big difference with the normal pc is that this one has 8GB RAM, the normal had 2GB.
    Last edited by SaturnusDJ; March 16th, 2013 at 09:13 AM.

  3. #23
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    369
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Disk write speed reduced by half over night

    Yesterday I changed back to the server pc.

    I changed a lot of things and this morning the speeds were normal. But I'm not convinced yet, it could be coincidence.

    Changes:
    - Pulled the energy meter from between the socket pdu and computer.
    - Pulled 1GB RAM, leaving 1GB there, resulting in 50% less RAM, and less speed because of switching from dual channel to single channel. [No fix.]
    - Change some IRQ things in BIOS. From auto to manual, and then per IRQ from auto to reserved for pci/other.
    - Changed some other things in the BIOS from auto to manual, however I know the outcome is the same.
    - Didn't start the big raid5 array (that's build with --assume-clean for testing), only the small one of 3*10GB. [No fix.]
    - Hard disk's outside the tower. Resulting in a ~6-10 degrees Celsius higher temperature, because there's no fan. [No fix.]
    - Different cables. Because the internal ones are only 20cm.
    - Left 'free -m' commands running over night. [No fix.]
    - Computer time/date was set a little bit ahead so it doesn't run behind over night because cron ntp stuff is disabled. [No fix.]

    - Much less storage tweaking:
    Now:
    Code:
    md2 stripe_cache_size: 4096
    md3 stripe_cache_size: -
    
    read ahead sdb: 256
    read ahead sdc: 256
    read ahead sdd: 256
    read ahead md2: 4096
    read ahead md3: -
    
    sda scheduler: noop deadline [cfq]
    sdb scheduler: noop deadline [cfq]
    sdc scheduler: noop deadline [cfq]
    sdd scheduler: noop deadline [cfq]
    (All stock except scs.)

    Normally (tuned):
    Code:
    md2 stripe_cache_size: 4096
    md3 stripe_cache_size: 4096
    
    read ahead sdb: 64
    read ahead sdc: 64
    read ahead sdd: 64
    read ahead md2: 4096
    read ahead md3: 4096
    
    sda scheduler: [noop] deadline cfq
    sdb scheduler: noop [deadline] cfq
    sdc scheduler: noop [deadline] cfq
    sdd scheduler: noop [deadline] cfq
    The raid write speeds are not very high, I think because of a lack of (dual channel) memory.
    Normally: 220MB/s write, 280MB/s read.
    Now: 200MB/s write, 280MB/s read.

    Non-raid speeds are fine.

    I think I'll leave it this way one more night to be sure, before turning back the disadvantages of this config.
    Last edited by SaturnusDJ; March 20th, 2013 at 07:34 AM. Reason: Adding what's not the solution

  4. #24
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    369
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Disk write speed reduced by half over night

    After changing back to 3x 20cm data sata cables, the third (/dev/sdd) dropped speeds. It's a little strange because I remember the same happening with longer cables. Only not in the last few days.

    Changing to long tonight.

  5. #25
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    369
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Disk write speed reduced by half over night

    Okay speed drop (all three) with long cables. This makes me sick......

  6. #26
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    369
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Disk write speed reduced by half over night

    In some way it has to do something with the cables.
    Because:
    - Earlier the long cables survived quite some nights.
    - Directly after I replaced the short cable of a drive (that wouldn't speed up after a power off and on) with a long cable the speed was fine.

    But tonight all drives had the longer cables and still dropped speed. So there is something else.

  7. #27
    Join Date
    Sep 2008
    Location
    Netherlands
    Beans
    369
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Disk write speed reduced by half over night

    I consider the problem solved.

    After thinking everything over, without any assumptions, a combination of symptoms led to the cause.

    What I forgot to mention was the room temperature. At night around 14 degrees Celsius. During the day not more than 20 degrees Celsius.

    That's perfect right?? Apparently not. The cold temperature at night causes the write speed to drop.

    After turning off all four 120mm fans, the problem is solved.
    Last edited by SaturnusDJ; October 14th, 2013 at 03:52 PM.

  8. #28
    Join Date
    Feb 2007
    Location
    West Hills CA
    Beans
    8,371
    Distro
    Ubuntu 12.10 Quantal Quetzal

    Re: Disk write speed reduced by half over night

    That is the strangest thing I have ever heard. It's possible that the drive controller firmware has a routine to check temperature and reduce speed if it is cold. Perhaps it's a bearing wear issue. You can use hddtemp or smartctl or the panel monitor to log temperatures. Use another routine to log write speeds and see at what temperature the drives slow down. That would be helpful for setting your temperature controller.
    Last edited by tgalati4; March 30th, 2013 at 08:19 PM.
    -------------------------------------
    Oooh Shiny: PopularPages

    Unumquodque potest reparantur. Patientia sit virtus.

  9. #29
    Join Date
    Jan 2011
    Beans
    48

    Re: Disk write speed reduced by half over night

    Greetings.
    I'm from Florida. I have an irrational fear of turning off the fans.
    I ran "sar -m TEMP 60 360" - my system is running at 50 and 67C.
    (And there are 3 fans running on the case. 72F inside the house.)


    I tend to think of problems that I have encountered in the past - which may not have much to your specific set up.

    The problem is very interesting. especially the cables.
    Try a magnifying glass and look at the connectors.
    I have had past bad experience of fan vibration and connectors.
    Probably not in these times - but worth a look.

    Some straw targets:
    PowerSupply - maybe your house power is better at night than in the day and your PS may be overloaded (marginally) so, maybe the DC power fluctuates in the daytime.
    Maybe taking out the fans reduces DC load?
    Got a DC voltmeter and some exposed test point? Be careful..

    Vibration - Are the fans jiggling the drives?

    Here a cache, there a cache, every where....
    Maybe the benchmark program runs faster with clean caches in CPU?
    Would like to know more about the benchmark program.
    Maybe it reaches a delay working with directories?
    I'm thinking of writing a simple python program to fill up your disk with 100k files -
    see if files in the beginning write faster than later.
    If you want to try this, I can post some code...
    Hmm... another favorite ghost in the closet is garbage collection. I know nothing about it - so it is easy for me to blame...

    About the disks.
    There is an old, but thorough diagnostic made by Gibson Research Company that boots off a cd and does extensive diagnostics on each sector. It saves your data, writes patterns, reads patterns and restores your data. It was called spin-rite.
    ?? Does any else think this might be worthwhile??

    Best Wishes,
    Jay

  10. #30
    Join Date
    Feb 2005
    Location
    Melbourne, Australia
    Beans
    13,505
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Disk write speed reduced by half over night

    Quote Originally Posted by SaturnusDJ View Post
    I consider the problem solved.

    After thinking everything over, without any assumptions, a combination of symptoms led to the cause.

    What I forgot to mention was the room temperature. At night around 14 degrees Celsius. During the day not more than 20 degrees Celsius.

    That's perfect right?? Apparently not. The cold temperature at night causes the write speed to drop.

    After turning of all four 120mm fans, the problem is solved. With summer coming, a fan controller with temperature sensors is not a bad idea.
    It is possible that you have a hardware issue like a bad solder joint or component on the MB around the disk controller, when the MB cools down overnight something no longer makes good contact to run at normal data rates and the hardware throttles down to do the best it can.

    You can usually identify this sort of thing with a can of Spray Freeze that electronics technicians used to use back in the day, you essentially freeze various bits of a running system to see if you can trigger the problem and that lets you narrow down the physical location of the issue.

    As well, any contamination on the MB can cause all sorts of issues - I once had a server go unreliable because the customer had it sitting on carpet in an office that was steam-cleaned twice a year and all the steam was sucked into it (along with the dirt in the air) creating a slimy coating all over the MB and a server that started rebooting itself enough to be replaced. It was only when the MB was properly cleaned that it behaved correctly again (and the customer learnt an expensive lesson about putting servers in appropriate environments). Dust in IT equipment is one thing, dust + moisture/humidity is just asking for trouble.

    If you do have these sort of hardware problems then eventually it will cause bigger issues, like loss of data.
    Last edited by dcstar; March 31st, 2013 at 01:04 PM.
    Regards, David.
    Please use the Forum search and Wiki search for immediate help
    Please mark your thread as Solved when appropriate
    New to technical forums?: How To Ask Questions The Smart Way

Page 3 of 4 FirstFirst 1234 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •