high load average, low cpu usage

[tokar93]

Hi

My Ubuntu 22.04.03 seems to be on a high load average whit barely any CPU activity. So i am trying to troubleshoot the problem and top dont seems to be giving me any clue on what can be running that make the load act like this.

top - 16:27:43 up 5:27, 1 user, load average: 7.00, 7.00, 7.00
Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 3858.9 total, 2246.5 free, 443.8 used, 1168.5 buff/cache
MiB Swap: 3859.0 total, 3859.0 free, 0.0 used. 3175.3 avail Mem

Any suggestions?

[TheFu]

Check the system logs.

CPU = Load, so the statement doesn't make sense.

Your box is 99% idle.
Now, if you are looking at the three "load numbers", those are averages over the last 1, 5, and 15 minutes, so if a job just completed, it will take some time for them to average back down. The manpage for top will have more details. Generally, load numbers will be tied to the number of cores engaged.

Here's the output from one of my systems:

top - 16:17:36 up 30 days,  8:29,  1 user,  load average: 0.04, 0.07, 0.02
Tasks: 235 total,   2 running, 233 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.3 us,  0.5 sy,  0.0 ni, 96.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.2 st
MiB Mem :   5874.0 total,    385.1 free,   5097.0 used,    391.9 buff/cache
MiB Swap:   1873.0 total,   1867.2 free,      5.8 used.    494.5 avail Mem

BTW, whenever posting terminal output, using code tags really helps so columns+spacing are maintained.

[tokar93]

Yes i know if it would take some time for the average to go down but the problem is that it don`t seems to be happen. It still seems to be around the same as the first post that i made.

But here is a new one whit the code tag being use.

top - 10:23:24 up 23:22,  1 user,  load average: 7.04, 7.05, 7.01
Tasks: 139 total,   1 running, 138 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.1 sy,  0.0 ni, 99.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3858.9 total,   1588.4 free,    812.5 used,   1458.0 buff/cache
MiB Swap:   3859.0 total,   3859.0 free,      0.0 used.   2822.4 avail Mem

Something is running here that is not showing or maybe related to some disk problem?

[TheFu]

Run xload. What does it show?
Also the uptime command will show the load.

Looks like something in a new kernel that top uses, but perhaps the others don't use ... my 22.04 box is still on 5.15.x kernels. If you are running a 6.x.x kernel, it could be a change in the reporting broke some old things.

[tokar93]

Run xload. What does it show?
Also the uptime command will show the load.

Looks like something in a new kernel that top uses, but perhaps the others don't use ... my 22.04 box is still on 5.15.x kernels. If you are running a 6.x.x kernel, it could be a change in the reporting broke some old things.

I dont have xload install and it seems like you need a display for that when you install and run it?

tokar86a@plex:~$ xload
Error: Can't open display:

Uptime is showing the same information.

tokar86a@plex:~$ uptime
 16:12:50 up 57 min,  1 user,  load average: 7.06, 7.06, 6.92

22.04.3 LTS (GNU/Linux 6.5.0-14-generic x86_64) yes i am running the new kernel and the other machine that i have use the same kernel and they have not the same problem. It is something that have happen in my plex server machine.

[TheFu]

I dont have xload install and it seems like you need a display for that when you install and run it?

tokar86a@plex:~$ xload
Error: Can't open display:

Uptime is showing the same information.

tokar86a@plex:~$ uptime
 16:12:50 up 57 min,  1 user,  load average: 7.06, 7.06, 6.92

22.04.3 LTS (GNU/Linux 6.5.0-14-generic x86_64) yes i am running the new kernel and the other machine that i have use the same kernel and they have not the same problem. It is something that have happen in my plex server machine.

Can you run a 5.xx kernel?

Do you know how to run remote X11 programs over ssh? It is a basic skill. Of course, the machine you sit behind will need an X/Server.

[tokar93]

No i don`t think i can remote in X11 programs over ssh. Nothing that i have done before and all the monitoring software that i have try showing the same numbers.

[MAFoElffen]

I would probably use this to look at the process state flags, filtering out all that are idle, sending it to a log to review

ps -e v | awk '$9 !~ /0\.0/' > temp-processes.log

Then look at the states from that output, and use this key from man page 'ps' to interpret the states

state    The state is given by a sequence of characters, for example, "RWNA". The      first character indicates the run state of the process:
D    Marks a process in disk (or other short term, uninterruptible) wait.
I    Marks a process that is idle (sleeping for longer than about 20 seconds).  
L    Marks a process that is waiting to acquire a lock.
R    Marks a runnable process.
S    Marks a process that is sleeping for less than about 20 seconds.
T    Marks a stopped process.
W    Marks an idle interrupt thread.
Z    Marks a dead process (a "zombie").

"D" and "R" in the 3rd Column (STAT) are the ones that usually be the culprits.

That is where I would start...

[TheFu]

No i don`t think i can remote in X11 programs over ssh. Nothing that i have done before.

https://blog.jdpfu.com/2010/09/07/ru...e-applications explains how.

But if your workstation has an X/Server, then you can run this command at the server to run almost any program remotely.

ssh -X userid@IP-address xload

Give it 2 seconds for the little xload window to display locally on your system.

My web browser is running this way, for example.
Same for my email program and a number of other tools.

It is how I work and how all old-timer UNIX people work and have since the 1980s. The system where something is displayed has very little to do with there it runs. There's no way to tell if a program is running locally or remote under X11 used this way. The integration into a desktop is complete.

[tokar93]

I would probably use this to look at the process state flags, filtering out all that are idle, sending it to a log to review

ps -e v | awk '$9 !~ /0\.0/' > temp-processes.log

Then look at the states from that output, and use this key from man page 'ps' to interpret the states

state    The state is given by a sequence of characters, for example, "RWNA". The      first character indicates the run state of the process:
D    Marks a process in disk (or other short term, uninterruptible) wait.
I    Marks a process that is idle (sleeping for longer than about 20 seconds).  
L    Marks a process that is waiting to acquire a lock.
R    Marks a runnable process.
S    Marks a process that is sleeping for less than about 20 seconds.
T    Marks a stopped process.
W    Marks an idle interrupt thread.
Z    Marks a dead process (a "zombie").

"D" and "R" in the 3rd Column (STAT) are the ones that usually be the culprits.

That is where I would start...

This is what i get in the temp file.

    PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
      1 ?        Ss     0:03    168     0 167552 12876  0.3 /sbin/init
    310 ?        S<s    0:00    196     0 56272 22020  0.5 /lib/systemd/systemd-journald
    351 ?        SLsl   0:00     20     0 289480 27520  0.6 /sbin/multipathd -d -s
    354 ?        Ss     0:00      3     0 25764  6144  0.1 /lib/systemd/systemd-udevd
    531 ?        Ssl    0:00      3     0 89360  7424  0.1 /lib/systemd/systemd-timesyncd
    629 ?        Ss     0:00     20     0 16124  8448  0.2 /lib/systemd/systemd-networkd
    631 ?        Ss     0:00     54     0 25668 13408  0.3 /lib/systemd/systemd-resolved
    662 ?        Ss     0:04     12     0  8736  5248  0.1 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
    667 ?        Ss     0:00     20     0 29072 19072  0.4 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
    671 ?        Ssl    0:13    149     0 1161636 22224  0.5 /usr/bin/prometheus-node-exporter
    674 ?        Ssl    0:01    312     0 1393516 30888  0.7 /usr/lib/snapd/snapd
    682 ?        Ssl    0:00    357     0 596632 27736  0.7 /usr/sbin/syslog-ng -F
    687 ?        Ss     0:00      6     0 15528  7680  0.1 /lib/systemd/systemd-logind
    714 ?        Ssl    0:34    697     0 230304 104844  2.6 /usr/lib/plexmediaserver/Plex Media Server
    756 ?        Ss     0:00      0     0 15432  9216  0.2 sshd: /usr/sbin/sshd -D 
[listener] 0 of 10-100 startups
    774 ?        Ssl    0:00     12     0 107220 21632  0.5 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
   1148 ?        SNl    0:05     73     0 64880 43728  1.1 Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-fb6452ebf/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.32.8.7639-fb6452ebf /usr/lib/plexmediaserver/Resources/Plug-ins-fb6452ebf/System.bundle
   1199 ?        Ss     0:00      2     0 41340 31320  0.7 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
   1229 ?        Sl     0:01     43     0 43496 16052  0.4 /usr/lib/plexmediaserver/Plex DLNA Server
   1230 ?        Sl     0:03     58     0 43936 13312  0.3 /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.32.8.7639-fb6452ebf 32600
   1308 ?        Ssl    0:02     13     0 1947432 61252  1.5 /usr/bin/crowdsec -c /etc/crowdsec/config.yaml
   1349 ?        S      0:00     16     0 162184 7808  0.1 journalctl --follow -n 0 _SYSTEMD_UNIT=smb.service
   1350 ?        S      0:00     43     0 252296 10496  0.2 journalctl --follow -n 0 _SYSTEMD_UNIT=ssh.service
   4104 ?        Ssl    0:00     78     0 292992 20096  0.5 /usr/libexec/packagekitd
   4108 ?        Ssl    0:00      4     0 234500 7424  0.1 /usr/libexec/polkitd --no-debug
   5641 ?        Sl     0:14      0     0 49256 19220  0.4 /usr/lib/plexmediaserver/Plex Transcoder -codec:0 h264 -codec:1 eac3_eae -eae_prefix:1 x1k1l2lq07t9easyo8es90n3_ -noaccurate_seek -analyzeduration 20000000 -probesize 20000000 -i  -map 0:0 -codec:0 copy -filter_complex [0:1] aresample=async=1:ochl='stereo':rematrix_maxval=0.000000dB:osr=48000[0] -map [0] -metadata:s:1 language=eng -codec:1 aac -b:1 256k -f dash -seg_duration 5 -dash_segment_type mp4 -init_seg_name init-stream$RepresentationID$.m4s -media_seg_name chunk-stream$RepresentationID$-$Number%05d$.m4s -window_size 5 -delete_removed false -skip_to_segment 1 -time_delta 0.0625 -manifest_name http://127.0.0.1:32400/video/:/transcode/session/x1k1l2lq07t9easyo8es90n3/049a67d6-1aef-40dd-9d84-3b0c17855dbf/manifest?X-Plex-Http-Pipeline=infinite -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 dash -map 0:2 -metadata:s:0 language=eng -codec:0 ass -strict_ts:0 0 -f segment -segment_format ass -segment_time 1 -segment_header_filename sub-header -segment_start_number 0 -segment_list http://127.0.0.1:32400/video/:/transcode/session/x1k1l2lq07t9easyo8es90n3/049a67d6-1aef-40dd-9d84-3b0c17855dbf/manifest?stream=subtitles&X-Plex-Http-Pipeline=infinite -segment_list_type csv -segment_list_size 5 -segment_list_separate_stream_times 1 -segment_format_options ignore_readorder=1 -segment_list_unfinished 1 -fflags +flush_packets sub-chunk-%05d -start_at_zero -copyts -vsync cfr -y -nostats -loglevel quiet -loglevel_plex error -progressurl http://127.0.0.1:32400/video/:/transcode/session/x1k1l2lq07t9easyo8es90n3/049a67d6-1aef-40dd-9d84-3b0c17855dbf/progress
   5755 ?        Ss     0:00      0     0 17184 10880  0.2 sshd: tokar86a [priv]
   5758 ?        Ss     0:00      0   893 16290  9728  0.2 /lib/systemd/systemd --user
   5759 ?        S      0:00      0     0 170464 6720  0.1 (sd-pam)
   5785 ?        S      0:00      0     0 17316  7932  0.2 sshd: tokar86a@pts/0
   5786 pts/0    Ss     0:00      0   891  4284  4096  0.1 -bash