12.04 crash - syslog

[alon L]

Hi,

I am running Ubuntu 12.04 LTS on IBM 3550. Recently I faced 2 system crashes, We had to restart the machines.
I will be very happy if will give me a direction where and how to look for the cause.
Here is part of the syslog:

I marked the reboot point


Jun 19 17:35:28 xxxxxx kernel: [495394.004281] kvm: disabling virtualization on CPU20
Jun 19 17:35:28 xxxxxx kernel: [495394.004308] CPU 20 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.005397] CPU 8 MCA banks CMCI:2 CMCI:3 CMCI:5
Jun 19 17:35:28 xxxxxx kernel: [495394.008782] kvm: disabling virtualization on CPU21
Jun 19 17:35:28 xxxxxx kernel: [495394.008811] CPU 21 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.010233] CPU 9 MCA banks CMCI:2 CMCI:3 CMCI:5
Jun 19 17:35:28 xxxxxx kernel: [495394.013474] kvm: disabling virtualization on CPU22
Jun 19 17:35:28 xxxxxx kernel: [495394.013504] CPU 22 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.017719] kvm: disabling virtualization on CPU23
Jun 19 17:35:28 xxxxxx kernel: [495394.017769] CPU 23 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.021972] kvm: disabling virtualization on CPU3
Jun 19 17:35:28 xxxxxx kernel: [495394.022057] CPU 3 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.026097] kvm: disabling virtualization on CPU4
Jun 19 17:35:28 xxxxxx kernel: [495394.026134] CPU 4 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.029913] kvm: disabling virtualization on CPU5
Jun 19 17:35:28 xxxxxx kernel: [495394.029954] CPU 5 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.033099] kvm: disabling virtualization on CPU6
Jun 19 17:35:28 xxxxxx kernel: [495394.033219] CPU 6 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.034274] CPU 7 MCA banks CMCI:6 CMCI:8
Jun 19 17:35:28 xxxxxx kernel: [495394.036277] kvm: disabling virtualization on CPU7
Jun 19 17:35:28 xxxxxx kernel: [495394.036496] CPU 7 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.037923] CPU 8 MCA banks CMCI:6 CMCI:8
Jun 19 17:35:28 xxxxxx kernel: [495394.039810] kvm: disabling virtualization on CPU8
Jun 19 17:35:28 xxxxxx kernel: [495394.039976] CPU 8 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.045568] CPU 9 MCA banks CMCI:6 CMCI:8
Jun 19 17:35:28 xxxxxx kernel: [495394.047361] kvm: disabling virtualization on CPU9
Jun 19 17:35:28 xxxxxx kernel: [495394.048136] CPU 9 is now offline
Jun 19 17:35:28 xxxxxx kernel: [495394.048138] SMP alternatives: switching to UP code
Jun 19 17:35:28 xxxxxx kernel: [495394.084037] init: anacron main process (22205) killed by TERM signal
Jun 19 17:35:30 xxxxxx kernel: [495395.634805] SMP alternatives: switching to SMP code
Jun 19 17:35:30 xxxxxx kernel: [495395.648774] Booting Node 0 Processor 1 APIC 0x2
Jun 19 17:35:30 xxxxxx kernel: [495395.648777] smpboot cpu 1: start_ip = 98000
Jun 19 17:35:30 xxxxxx kernel: [495395.659784] Calibrating delay loop (skipped) alreadyJun 19 20:00:43 xxxxxx kernel: imklog 5.8.6, log source = /proc/kmsg started.
------------------- REBOOT POINT -----------------------------------------------------------
Jun 19 20:00:43 xxxxxx rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="1477" x-info="http://www.rsyslog.com"] start
Jun 19 20:00:43 xxxxxx rsyslogd: rsyslogd's groupid changed to 103
Jun 19 20:00:43 xxxxxx rsyslogd: rsyslogd's userid changed to 101
Jun 19 20:00:43 xxxxxx rsyslogd-2039: Could not open output pipe '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
Jun 19 20:00:43 xxxxxx kernel: [ 0.000000] Initializing cgroup subsys cpuset
Jun 19 20:00:43 xxxxxx kernel: [ 0.000000] Initializing cgroup subsys cpu
Jun 19 20:00:43 xxxxxx kernel: [ 0.000000] Linux version 3.2.0-24-generic (buildd@crested) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 (Ubuntu 3.2.0-24.39-generic 3.2.16)
Jun 19 20:00:43 xxxxxx kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.2.0-24-generic root=UUID=a6dcb63a-3873-4370-afb6-3361df0b79b3 ro quiet
Jun 19 20:00:43 xxxxxx kernel: [ 0.000000] KERNEL supported cpus:
Jun 19 20:00:43 xxxxxx kernel: [ 0.000000] Intel GenuineIntel
Jun 19 20:00:43 xxxxxx kernel: [ 0.000000] AMD AuthenticAMD
Jun 19 20:00:43 xxxxxx kernel: [ 0.000000] Centaur CentaurHauls


Thanks

[Soulou]

Hi,

I encounter the same issue.

http://pastebin.com/EDJvEt2N

The Server's CPUs are disabling and enabling in a loop. Once in a while the the system crashes.

Kernel issue ?

Thank you !

[ondrejpenicka]

Hi,

we discovered same behaviour on our KVM server. It was something related to power management. Uninstalling pm-tools resolved this issue.

Hope it helps.

Andrew

[TroN-0074]

I am facing this same problem. I am thinking on just switch to a different distro because the crashes occurs too often, sometimes when I am in the middle of work.
If there is a solution please let me know.

[ondrejpenicka]

In our case, this behaviour start after disk error (and previous thunderstorm maybe electricity surge). So i don't know if there is some bad signal from motherboard about frequency/power/sleep etc.

And this messages probably catch power management tools, there was script in /etc/pm/power.d/01cpu_online which switch on/off cpu (core) using 'echo 1 > /sys/devices/system/cpu/cpu*/online'

Uinstalling package pm-utils, resolved this issue.

Other thing i tried before this was disabling posibility of enabling/disabling cpu cores using /sys/devices/system/cpu/cpu*/online

There is kernel option nr_cpus= which can be set to number of used processors (cores). Setting this should disable hot plug feature of cpu (enabling/disabling). But in my case, after setting this to grub boot parameters, it has no effect on disabling loop (instead of missing /sys/devices/system/cpu/cpu*/online file).

Before, we tried upgrade distro from ubuntu natty (kernel 2.6) to ubuntu precise (kernel 3.2), without resolving.

Andrew

[gert cuykens]

same issue pc starts crashing but recovers sometimes.

Linux i7 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

[24119.931624] CPU 1 is now offline
[24119.932272] CPU 4 MCA banks CMCI:2 CMCI:3 CMCI:5
[24119.934452] CPU 2 is now offline
[24119.935371] CPU 5 MCA banks CMCI:2 CMCI:3 CMCI:5
[24119.937125] CPU 3 is now offline
[24119.937945] CPU 6 MCA banks CMCI:2 CMCI:3 CMCI:5
[24119.938721] Broke affinity for irq 16
[24119.939781] CPU 4 is now offline
[24119.940996] Broke affinity for irq 44
[24119.942075] CPU 5 is now offline
[24119.944257] CPU 6 is now offline
[24119.947244] CPU 7 is now offline
[24119.947246] SMP alternatives: switching to UP code
[24133.425495] SMP alternatives: switching to SMP code
[24133.429981] Booting Node 0 Processor 1 APIC 0x4
[24133.429983] smpboot cpu 1: start_ip = 99000
[24133.441005] Calibrating delay loop (skipped) already calibrated this CPU
[24133.484248] NMI watchdog enabled, takes one hw-pmu counter.
[24133.494502] Booting Node 0 Processor 2 APIC 0x6
[24133.494505] smpboot cpu 2: start_ip = 99000
[24133.505527] Calibrating delay loop (skipped) already calibrated this CPU
[24133.526529] NMI watchdog enabled, takes one hw-pmu counter.
[24133.669271] Booting Node 0 Processor 3 APIC 0x2
[24133.669273] smpboot cpu 3: start_ip = 99000
[24133.680294] Calibrating delay loop (skipped) already calibrated this CPU
[24133.701320] NMI watchdog enabled, takes one hw-pmu counter.
[24133.703827] Booting Node 0 Processor 4 APIC 0x5
[24133.703829] smpboot cpu 4: start_ip = 99000
[24133.714850] Calibrating delay loop (skipped) already calibrated this CPU
[24133.736007] NMI watchdog enabled, takes one hw-pmu counter.
[24133.869446] Booting Node 0 Processor 5 APIC 0x7
[24133.869448] smpboot cpu 5: start_ip = 99000
[24133.880369] Calibrating delay loop (skipped) already calibrated this CPU
[24133.901511] NMI watchdog enabled, takes one hw-pmu counter.
[24134.069618] Booting Node 0 Processor 6 APIC 0x3
[24134.069621] smpboot cpu 6: start_ip = 99000
[24134.080644] Calibrating delay loop (skipped) already calibrated this CPU
[24134.101906] NMI watchdog enabled, takes one hw-pmu counter.
[24134.218718] Booting Node 0 Processor 7 APIC 0x1
[24134.218721] smpboot cpu 7: start_ip = 99000
[24134.229683] Calibrating delay loop (skipped) already calibrated this CPU
[24134.250931] NMI watchdog enabled, takes one hw-pmu counter.