[SOS] Everything stops (system lags-freezes) when the user is idle

[RodGer GR]

I have a really serious problem which I don't know where it comes from. When I don't move the mouse, everything slows down and stops.
1)The clock stops refreshing
2)The cursor stops blinking
3)Some times video freezes
4)Downloading slows down or even stops
5)Copying files slows down or even stops
6)Some times the wireless network disconnects (connects again when I move the mouse but some times asks again for a password)
7)Screen saver rarely shows up and when it starts stays still without moving as it should.
8 )Animated images (e.g. some forum icons etc) stop moving

I would be grateful if you could give me a hint on what could cause such a problem. A work around would be nice but I would surely prefer to find what could really cause such a problem.

Please help it is really disturbing and there are many things (e.g. download a large file) I can't do due to this problem. I've also spent so much time configuring my system the way I wanted it and it's such a pity to reinstall.

Extra info that may help:
Release: Ubuntu 9.10 (karmic)
GNOME: 2.28.1 (Ubuntu 2009-11-03)
kernel: 2.6.31-16-generic (#53-Ubuntu SMP Tue Dec 8 04:01:29 UTC 2009)

I also use hardy heron on the same computer and there is no such problem

Even if you don't have any solution, please ask me for any information, I should provide, that would help to make more clear what causes the problem.

Thanks!

[RodGer GR]

It is so strange ... Even if I choose to log out and stay idle after the little countdown timer shows up, it will practically won't log out as it stops counting after about ten seconds.

[RodGer GR]

Finally I found a work-around. I just added the kernel option nohz=off at the end of the ubuntu 9.10 kernel line in my menu.lst file and now the problems I described earlier disappeared.

But this is not a real solution. It just disabled dynamic ticks. Disabling dynamic ticks may reduce the battery life. I still wonder if there is any way to fix the problem without disabling dynamic ticks.

Any ideas?

[granoff]

I believe I have the same problem on my 9.10 server. The system, seemingly at random intervals, "lags/freezes" as you put it. Existing network connections are dropped. New network connections cannot be made. The only thing that seems to "wake up" the system is to press return on the console (which is connected via a PS2 KVM switch; no USB here). I note that "pressing return" kicks an interrupt... This system does not run gdm or X, fwiw.

I see this problem in kernels 2.6.31-15 and 2.6.28-16.

I have stopped ntpd, because I thought that was causing the problem. (I had noticed ntpd had exited due to a SEGV one time, which made it the smoking gun.)

I have stopped cupsd, because every time this happens, the last messages on the console concern cups restarting.

I have added clocksource=acpi_pm to my kernel boot options, because the default clocksource, rtc, is reported as unstable.

The "worst" thing I've seen in the logs is a kernel oops in inotify_fsnoticy.c:129, which has been reported a lot, and there is an apparent fix in 2.6.32-rc. (However, some people running that have reported this oops still exists. Moving to 2.6.32-rc is not an option for me.) I've seen these oops' in the log at times at the same time as the "lag/freeze."

Anyway, I am out of smoking guns! This freeze happened just this morning and there is nothing in my logs now at the time it occurred to clue me into what is happening.

I am going to try your "nohz=off" trick and see if that helps me as well. (What does that actually do, anyway?)

Interestingly, I run 9.10 on 2 other systems (a Dell desktop and a Dell laptop) and don't see this issue.

I'll report back if nohz=off has a positive impact for me.

Thanks!

-Mark

[RodGer GR]

[.....]
I am going to try your "nohz=off" trick and see if that helps me as well. (What does that actually do, anyway?)

Interestingly, I run 9.10 on 2 other systems (a Dell desktop and a Dell laptop) and don't see this issue.

I'll report back if nohz=off has a positive impact for me.

Thanks!

-Mark

It looks like the problem comes from dynamic ticks.
Dynamic ticks is an patch merged in the Linux kernel (since kernel 2.6.21-rc1) which allows timer ticks only when they are needed. When the user is idle there are no timer interrupts (or.. they are significantly reduced to be more accurate). This results to a slight reduction of power consumption and cpu's temperature. The difference becomes more significant on laptops (battery life), servers(the difference accumulates for systems running 24/7), virtual machines (when you have several Linux instances running the same time).
Here (link) you can find more info about the patch.

In my case and as it seems your case too, the kernel (2.6.31-16) has a problem with dynamic ticks.
That's why I disabled dynamic ticks adding the nohz=off option in the menu.lst file.

I can't reproduce this problem on an other computer so it can be a hardware specific problem. Even on my laptop though (where the problem came up) there was no problem initially and I think that there will be no problem on a clean installation. It would be useful if we could gather more information concerning under what circumstances this problem shows up. If we find a way to reproduce it we can file a bug and try to fix it.

I found somebody describing a similar problem back when the patch was merged in the 2.6.21 kernel here (link):

Hi, I've been struggling for some time to get a kernel with dynamic ticks to be stable on my current hardware configuration. Right now, I'm using a Toshiba M40 with a Intel centrino cpu. Basically, what happens is that the system will start slowing down and then completely freeze up after about 5 seconds after the initial slowdown.

I know this to be an issue with dynamic ticks as disabling it in my kernel build fixed the problem. Disabling HPET timers in the kernel and adding clocksource=acpi_pm as a kernel boot parameter doesn't seem to fix the issue with dynamic ticks.

So, I'm wondering if others are having this problem (an issue in both 2.6.21 and 2.6.22); and if anyone knows any possible solutions outside of disabling dynamic ticks.

P.S.1: An other 'side effect' of the same problem I didn't mention is that it makes my system clock run slower and finally lose more than an hour per day. Do you have any similar "slow clock" problems?

P.S.2: If anyone reading this has any thoughts that could help us connect the 'dots' and see what may went wrong with dynamic ticks, please, speak freely.

[granoff]

It looks like the problem comes from dynamic ticks.
Dynamic ticks is an patch merged in the Linux kernel (since kernel 2.6.21-rc1) which allows timer ticks only when they are needed. When the user is idle there are no timer interrupts (or.. they are significantly reduced to be more accurate). ...

Thanks for the explanation.

I think "interrupts" is a clue here (and this is a just a wild theory at this point)... BUT, considering that, at least in my case, when this happens I cannot connect to the machine via the network, which would require interrupts on the network interface to be working, then perhaps interrupts is an area to be looked into with respect to dynamic ticks... dunno.

This results to a slight reduction of power consumption and cpu's temperature. The difference becomes more significant on laptops (battery life), servers(the difference accumulates for systems running 24/7), virtual machines (when you have several Linux instances running the same time).
Here (link) you can find more info about the patch.

I am not so much concerned with power consumption or CPU temp, BUT my system is a 24/7 server, fwiw.

...
I can't reproduce this problem on an other computer so it can be a hardware specific problem. Even on my laptop though (where the problem came up) there was no problem initially and I think that there will be no problem on a clean installation. It would be useful if we could gather more information concerning under what circumstances this problem shows up. If we find a way to reproduce it we can file a bug and try to fix it.

Yeah, I can't reproduce it either. And I see nothing in my logs that would suggest any consistent smoking gun. Plus, I can't necessarily say I know enough about the inner workings of the kernel that I know everywhere (other than log files) to look for hints.

P.S.1: An other 'side effect' of the same problem I didn't mention is that it makes my system clock run slower and finally lose more than an hour per day. Do you have any similar "slow clock" problems?

Yup! When I resusitate my system, by pressing enter on the console (which fires an interrupt!), the time on the system is at, or very very close to the time the system froze. I have to reset it using ntpdate. Then everything is A-OK.

P.S.2: If anyone reading this has any thoughts that could help us connect the 'dots' and see what may went wrong with dynamic ticks, please, speak freely.

Second that! This is a major issue, IMO. Something like handling interrupts (if that is not a red herring) should be a component of the kernel that is rock solid at this point. Anything I can do to help, I am happy to do.

-Mark

[RodGer GR]

Finally, did you try the nohz=off option? What was the result?

[granoff]

Finally, did you try the nohz=off option? What was the result?

So far so good.

The system has been stable since about 18:00 EDT last night, having booted the 2.6.31-16 kernel with clocksource=acpi_pm nohz=off.

BUT, stability over one night doesn't mean anything, at least in my case. I've seen the problem occur after several days of continuous, stable operation.

Interestingly, in a bug report here:

http://bugzilla.kernel.org/show_bug.cgi?id=14646

where I noted that when I saw the issue reported there along with the issues reported here, the assigned maintainer reported an issue he found with dmesg and sending too much data to the console possibly causing a clock skew.

I referenced this thread, in that thread, in the hopes of gathering more breadcrumbs and getting some more eyes on all this. We'll see what happens.

As and if my system continues to be stable, I will report back.

-Mark

[granoff]

My system remains stable after 1 day and 15+ hours, booted as noted above.

I am not 100% convinced yes, because I've seen the system run for several days before having a problem.

I do, however, have my fingers crossed, because I am going away for a few days and won't be able to fix anything, should anything happen, until the middle of next week when I return. (The good or bad news, depending on how you look at it, is that I'll know well before my return if something happens, because my system will be unreachable. Oh what a feeling of helplessness that will be!

Will continue to report status as necessary.

-Mark

[granoff]

The daily update...

Still up and running, 2 days and 17 hours...

But also still within the window, in my experience with this issue, of "run for a few days and then lag/freeze/etc."

Where I am, at this moment, remote, I am quite pleased that the system is apparently running smoothly!

-Mark