Very slow SDD/HDD performance

[jazzgossen]

My fresh Kubuntu 12.10 installation often (but not always) "blocks" when doing things like "sudo apt-get update" and often spends several minutes on "Reading package lists". Other things that take unusally long is the linking stage after compiling programs.

My setup and symptoms are similar as the ones in this thread.
I have a Dell Precision M4700, with one SSD and one HDD. Root is mounted on the SSD, and /home and /var are mounted on the HDD. I have /tmp mounted as tmpfs.

Any suggestions?

[dabl]

The lags that happen during "apt-get update" and "apt-get upgrade" are not hardware-related, in all likelihood, they reflect congestion or lags at the repository server. I'm not sure about the linking issue. What makes you attribute the lag to hdd or ssd performance? Have you used hdparm to test the speed of your ssd and hdd?

sudo hdparm -tT /dev/sda

If the speed figures are respectable (google will help you with that), then you can safely turn your attention to some other source of delay in linking.

[jazzgossen]

Thanks for your reply.

Running hdparm -tT gives me reasonable results, I think: cached reads are around 4000 MB/s and buffered reads 100 MB/s for the HDD, and for the SSD I get between 4000 and 11000 MB/s cached reads, and 258 MB/s buffered reads.

However, I don't think that the "Reading package lists" delay is related to network lag. I just tried "time sudo apt-get update", and it fetched all the data in 5 seconds, and while I'm writing this, it is slowly making its way through reading the downloaded package lists (currently at 7%, and I guess it will be done in a few minutes.)

KDE's system load applet shows around 40% CPU time spent in I/O wait. A weird thing is that I don't see that when running htop. The line for "apt-get update" shows zero in the IO column.

I hear a bit of intermittent HDD activity while doing this. I'm not sure which process is doing it though. Since only /home and /var are mounted on the HDD, I would assume that apt-get doesn't need to access the HDD at all.

[jazzgossen]

Update: apt-get update just finished, and the result of "time sudo apt-get update" is

real 34m36.434s
user 0m28.938s
sys 0m7.140s

[jazzgossen]

I just learned that "time" can output a lot of other information as well. I timed "apt-get update" again, but this time with some more output. I've pasted the output below. I'm not sure what to make of this, though --- whether the volontary and involontary context switches are a problem, for example.

Anyway, the fact that the process only spends seconds in user and kernel mode, while the wall-clock time is 28 minutes, is puzzling to me.

> 'time' -f "%E real, %U user, %S sys, %F faults, %I inputs, %O outputs, %R minor faults, %W swap, %c invol. cntx-switch, %w vol. cntx-switch" sudo apt-get update
[sudo] password for martin: 
Ign http://se.archive.ubuntu.com quantal InRelease
Ign http://deb.opera.com stable InRelease                                                                                                                                                    
Ign http://ppa.launchpad.net quantal InRelease                                                                                                                                               
Hit http://repository.spotify.com stable InRelease                                                                                                                                           
Hit http://deb.opera.com stable Release.gpg                                                                                                                                                  
Hit http://deb.opera.com stable Release                                                                                                                                                      
Ign http://ppa.launchpad.net precise InRelease                                                                                                                                               
Ign http://extras.ubuntu.com quantal InRelease                                                                                                                                               
Hit http://deb.opera.com stable/non-free i386 Packages                                                                                                                                       
Hit http://repository.spotify.com stable/non-free Sources                                                                                                                                    
Ign http://ppa.launchpad.net quantal InRelease                                                                                                                                               
Hit http://extras.ubuntu.com quantal Release.gpg                                                                                                                                             
Ign http://se.archive.ubuntu.com quantal-updates InRelease                                                                                                                                
Hit http://repository.spotify.com stable/non-free i386 Packages                                                                                                     
Ign http://ppa.launchpad.net quantal InRelease                                                                                                                      
Ign http://ppa.launchpad.net quantal InRelease                                                                                                                      
Ign http://se.archive.ubuntu.com quantal-backports InRelease                                                                                                        
Ign http://security.ubuntu.com quantal-security InRelease                                                                              
Hit http://ppa.launchpad.net quantal Release.gpg                                                                                       
Hit http://extras.ubuntu.com quantal Release                                                                                           
Hit http://se.archive.ubuntu.com quantal Release.gpg                                                                                                          
Hit http://ppa.launchpad.net precise Release.gpg                                                                                                              
Hit http://se.archive.ubuntu.com quantal-updates Release.gpg                                                                            
Hit http://ppa.launchpad.net quantal Release.gpg                                                                                       
Hit http://extras.ubuntu.com quantal/main Sources                                                                                      
Get:1 http://security.ubuntu.com quantal-security Release.gpg [933 B]                                                                  
Hit http://ppa.launchpad.net quantal Release.gpg                                                                                                                        
Hit http://se.archive.ubuntu.com quantal-backports Release.gpg                                                                         
Ign http://deb.opera.com stable/non-free Translation-en_US                                                                             
Ign http://linux.dropbox.com precise InRelease                                                                                         
Hit http://extras.ubuntu.com quantal/main i386 Packages                                                                                
Ign http://deb.opera.com stable/non-free Translation-en                                                                                
Hit http://se.archive.ubuntu.com quantal Release                                                                                       
Hit http://ppa.launchpad.net quantal Release.gpg                                                                                                              
Hit http://se.archive.ubuntu.com quantal-updates Release                                                                                                      
Hit http://ppa.launchpad.net quantal Release                                                                                                                  
Get:2 http://security.ubuntu.com quantal-security Release [49.6 kB]                                                                                           
Hit http://se.archive.ubuntu.com quantal-backports Release                                                                                                              
Hit http://ppa.launchpad.net precise Release                                                                                                     
Hit http://ppa.launchpad.net quantal Release                                                                                                                            
Hit http://se.archive.ubuntu.com quantal/main Sources                                                                                                                   
Hit http://ppa.launchpad.net quantal Release                                                                                                                            
Hit http://se.archive.ubuntu.com quantal/restricted Sources                                                                                                             
Get:3 http://linux.dropbox.com precise Release.gpg [489 B]                                                                                                              
Hit http://ppa.launchpad.net quantal Release                                                                                                                
Ign http://repository.spotify.com stable/non-free Translation-en_US                                                                                                     
Hit http://ppa.launchpad.net quantal/main Sources                                                                                                
Ign http://repository.spotify.com stable/non-free Translation-en                                                           
Hit http://se.archive.ubuntu.com quantal/universe Sources                                                                  
Ign http://extras.ubuntu.com quantal/main Translation-en_US                                                          
Hit http://ppa.launchpad.net quantal/main i386 Packages                                     
Hit http://se.archive.ubuntu.com quantal/multiverse Sources                                 
Get:4 http://linux.dropbox.com precise Release [2,603 B]                                    
Ign http://extras.ubuntu.com quantal/main Translation-en                                                                     
Get:5 http://security.ubuntu.com quantal-security/main Sources [33.5 kB]                    
Hit http://se.archive.ubuntu.com quantal/main i386 Packages                                          
Hit http://ppa.launchpad.net precise/main Sources                                                    
Hit http://se.archive.ubuntu.com quantal/restricted i386 Packages              
Hit http://ppa.launchpad.net precise/main i386 Packages                        
Hit http://se.archive.ubuntu.com quantal/universe i386 Packages                
Get:6 http://linux.dropbox.com precise/main i386 Packages [1,148 B]                                               
Hit http://ppa.launchpad.net quantal/main Sources                                                                             
Get:7 http://security.ubuntu.com quantal-security/restricted Sources [14 B]                 
Hit http://se.archive.ubuntu.com quantal/multiverse i386 Packages                                                      
Hit http://ppa.launchpad.net quantal/main i386 Packages                                     
Get:8 http://security.ubuntu.com quantal-security/universe Sources [15.5 kB]                                       
Hit http://ppa.launchpad.net quantal/main Sources                                                    
Hit http://ppa.launchpad.net quantal/main i386 Packages                                                           
Hit http://se.archive.ubuntu.com quantal/main Translation-en                                                      
Get:9 http://security.ubuntu.com quantal-security/multiverse Sources [700 B]                
Hit http://se.archive.ubuntu.com quantal/multiverse Translation-en                                                
Hit http://ppa.launchpad.net quantal/main Sources                                           
Hit http://ppa.launchpad.net quantal/main i386 Packages               
Hit http://se.archive.ubuntu.com quantal/restricted Translation-en    
Get:10 http://security.ubuntu.com quantal-security/main i386 Packages [90.5 kB]
Hit http://se.archive.ubuntu.com quantal/universe Translation-en                                      
Hit http://se.archive.ubuntu.com quantal-updates/main Sources                                         
Hit http://se.archive.ubuntu.com quantal-updates/restricted Sources              
Hit http://se.archive.ubuntu.com quantal-updates/universe Sources                
Hit http://se.archive.ubuntu.com quantal-updates/multiverse Sources              
Hit http://se.archive.ubuntu.com quantal-updates/main i386 Packages              
Hit http://se.archive.ubuntu.com quantal-updates/restricted i386 Packages        
Hit http://se.archive.ubuntu.com quantal-updates/universe i386 Packages          
Hit http://se.archive.ubuntu.com quantal-updates/multiverse i386 Packages        
Get:11 http://security.ubuntu.com quantal-security/restricted i386 Packages [14 B]                                  
Hit http://se.archive.ubuntu.com quantal-updates/main Translation-en                                                       
Get:12 http://security.ubuntu.com quantal-security/universe i386 Packages [42.3 kB]           
Hit http://se.archive.ubuntu.com quantal-updates/multiverse Translation-en                                                    
Hit http://se.archive.ubuntu.com quantal-updates/restricted Translation-en                                                    
Hit http://se.archive.ubuntu.com quantal-updates/universe Translation-en                                            
Hit http://se.archive.ubuntu.com quantal-backports/main Sources                               
Get:13 http://security.ubuntu.com quantal-security/multiverse i386 Packages [1,397 B]         
Hit http://se.archive.ubuntu.com quantal-backports/restricted Sources                                                            
Hit http://se.archive.ubuntu.com quantal-backports/universe Sources                           
Hit http://se.archive.ubuntu.com quantal-backports/multiverse Sources                         
Hit http://se.archive.ubuntu.com quantal-backports/main i386 Packages                         
Hit http://se.archive.ubuntu.com quantal-backports/restricted i386 Packages                                         
Hit http://se.archive.ubuntu.com quantal-backports/universe i386 Packages                                           
Hit http://se.archive.ubuntu.com quantal-backports/multiverse i386 Packages                   
Hit http://security.ubuntu.com quantal-security/main Translation-en                           
Hit http://se.archive.ubuntu.com quantal-backports/main Translation-en                        
Hit http://se.archive.ubuntu.com quantal-backports/multiverse Translation-en                                        
Hit http://se.archive.ubuntu.com quantal-backports/restricted Translation-en                
Hit http://se.archive.ubuntu.com quantal-backports/universe Translation-en
Hit http://security.ubuntu.com quantal-security/multiverse Translation-en
Ign http://ppa.launchpad.net quantal/main Translation-en_US                                 
Ign http://ppa.launchpad.net quantal/main Translation-en                                    
Ign http://linux.dropbox.com precise/main Translation-en_US           
Ign http://ppa.launchpad.net precise/main Translation-en_US           
Ign http://ppa.launchpad.net precise/main Translation-en              
Ign http://ppa.launchpad.net quantal/main Translation-en_US           
Ign http://linux.dropbox.com precise/main Translation-en              
Ign http://ppa.launchpad.net quantal/main Translation-en              
Hit http://security.ubuntu.com quantal-security/restricted Translation-en
Ign http://ppa.launchpad.net quantal/main Translation-en_US
Ign http://ppa.launchpad.net quantal/main Translation-en
Ign http://ppa.launchpad.net quantal/main Translation-en_US           
Hit http://security.ubuntu.com quantal-security/universe Translation-en
Ign http://ppa.launchpad.net quantal/main Translation-en
Ign http://se.archive.ubuntu.com quantal/main Translation-en_US
Ign http://se.archive.ubuntu.com quantal/multiverse Translation-en_US
Ign http://se.archive.ubuntu.com quantal/restricted Translation-en_US
Ign http://se.archive.ubuntu.com quantal/universe Translation-en_US
Ign http://se.archive.ubuntu.com quantal-updates/main Translation-en_US
Ign http://se.archive.ubuntu.com quantal-updates/multiverse Translation-en_US
Ign http://se.archive.ubuntu.com quantal-updates/restricted Translation-en_US
Ign http://se.archive.ubuntu.com quantal-updates/universe Translation-en_US
Ign http://se.archive.ubuntu.com quantal-backports/main Translation-en_US
Ign http://se.archive.ubuntu.com quantal-backports/multiverse Translation-en_US
Ign http://se.archive.ubuntu.com quantal-backports/restricted Translation-en_US
Ign http://se.archive.ubuntu.com quantal-backports/universe Translation-en_US
Ign http://security.ubuntu.com quantal-security/main Translation-en_US
Ign http://security.ubuntu.com quantal-security/multiverse Translation-en_US
Ign http://security.ubuntu.com quantal-security/restricted Translation-en_US
Ign http://security.ubuntu.com quantal-security/universe Translation-en_US
Fetched 239 kB in 4s (49.3 kB/s)
Reading package lists... Done
28:53.26 real, 24.37 user, 5.55 sys, 20 faults, 2728 inputs, 3851424 outputs, 495023 minor faults, 0 swap, 1695 invol. cntx-switch, 394480 vol. cntx-switch

[dabl]

Hmmmmm -- yes, there is something off on that system. I just ran the same command (thanks for that!) on a fully-updated Kubuntu 12.10 system, and got this result:

0:09.01 real, 1.22 user, 0.51 sys, 43 faults, 206672 inputs, 161104 outputs, 22915 minor faults, 0 swap, 501 invol. cntx-switch, 864 vol. cntx-switch

The dpkg system does its caching and logging in /var. It's not obvious to me, with what you've described about your system, why that would be relevant, but it is an unusual configuration to have /var mounted separately from the rest of the root filesystem. I still don't believe that hard drive performance is at the root of your issue, but possibly a caching function is not working correctly. As an experiment, could you mount /var on the SSD with the rest of the root filesystem? It might or might not reveal something about the source of the sluggishness.

[dabl]

I would change the mount line in /etc/fstab to reflect the new mount point, and then shutdown and reboot, to avoid disturbing the system logging processes.

How about the dpkg cache --- has it been cleared lately?

sudo apt-get clean

[dabl]

Question: Did you explicitly mount /tmp in /etc/fstab, or is that a default configuration? The reason I ask, my 12.10 system (a VM) does not show a separate mount or filesystem for /tmp.

[jazzgossen]

Now I have moved /var to the SSD, and at the moment things are blazing. However, as I mentioned in my original post, the problem with slow IO performance only happened sometimes, not all the time. So I don't dare to consider the problem solved just yet.

Regarding the /tmp mount point: yes, that was done manually. I wanted to move both /var and /tmp away from the SSD, to reduce the amount of writes to it. I hope putting /var back on the SSD won't kill it prematurely.

[Impavidus]

It seems it's actually allowed to put /var on a different partition: http://refspecs.linuxfoundation.org/...ROOTFILESYSTEM

If I'm not mistaken SSD wear depends on the number of erase cycles, not the number of writes. Only once in every so many lines written to a log (depending on the physical block size of the drive) one block is erased, whilst normal updates erase many blocks simultaniously.