Page 2 of 2 FirstFirst 12
Results 11 to 17 of 17

Thread: CPU load average spikes every hour with CPU 55% and memory 82% only

  1. #11
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    20,468
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: CPU load average spikes every hour with CPU 55% and memory 82% only

    Quote Originally Posted by azinity-tom View Post
    hi thefu,

    thanks for your suggestion!

    but it doesn't sound like a very good option at this point because we got 31 websites sitting on this DO droplet and many of the domain names are not even managed by us.
    What do domainnames have to do with this? Changing 1 line in 31 apache confs (the virtual domain name) wouldn't alter anything.
    The point is to simplify and test. And keep simplifying and testing until the root cause of the issue become clear. I'd rather not have a 100 guesses game, but it is your system.

  2. #12
    Join Date
    Jul 2020
    Beans
    8

    Re: CPU load average spikes every hour with CPU 55% and memory 82% only

    hi doug s, thanks for the note.

    i got a tip from someone on unix.com community forum who suggested some very aggressive, rogue, unidentified bots originating on Chinese networks! however, i don't find anything malicious or anomalous in the second session where the cpu load spiked and mysql was killed. am i missing something?? any idea?

    br,
    tom

  3. #13
    Join Date
    Jul 2020
    Beans
    8

    Re: CPU load average spikes every hour with CPU 55% and memory 82% only

    hi thefu,

    oh! sorry i misread that!

    migrating from mysql to mariadb takes time! i don't know if this is the route i want to take at this point.

    if the spike is due to malicious bots, migration to mariadb won't help, will it?

    br
    tom

  4. #14
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    2,852
    Distro
    Ubuntu Development Release

    Re: CPU load average spikes every hour with CPU 55% and memory 82% only

    I looked at the link you provided. All I can say is that this is an excerpt from my iptables rules set:

    Code:
    # Enough already. Just disallow the entire 185.0.0.0 sub-net
    $IPTABLES -A INPUT -i $EXTIF -s 185.0.0.0/8 -d $UNIVERSE -j DROP
    Which isn't China, by the way. I block all of China using ipset.

    And here are my packet/byte counts:

    Code:
       27621  1283626 DROP       all  --  enp4s0 *       185.0.0.0/8          0.0.0.0/0
    So, 27,621 attempts to create a session. My uptime is significant though, probably 1000 tries per day average. (and my site is just a pathetic little thing of no significance to anyone.)

    By the way, you got some really good advice over on that other forum. Just so you know, my iptables rules set has evolved over a period of well over a decade.
    Last edited by Doug S; 4 Weeks Ago at 07:42 PM.
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  5. #15
    Join Date
    Jul 2020
    Beans
    8

    Re: CPU load average spikes every hour with CPU 55% and memory 82% only

    Hey Doug,

    Thanks for sharing your invaluable experience with me. We ran some scripts on the server and came up with a summary of the activities as below.

    As far as IPs, we have offices in Norway, UK, HK, and China so some of those IPs are our IPs. Our China office uses VPN which means their IPs vary from time to time. So how can we find out which IPs from China are malicious? Shall we block those IPs?

    Any suggestion will be highly appreciated. Thank you so much in advance.

    P.S. We have yet to fix the mysterious hourly CPU load spike which cripples our 31 websites for almost a minute every hour!!



    Top 20 GET requests:
    76 GET /wp-content/themes/bridge/css/style_dynamic_responsive_callback.php
    80 GET /wp-includes/js/jquery/ui/tabs.min.js
    81 GET /wp-includes/js/comment-reply.min.js
    83 GET /wp-includes/js/jquery/ui/accordion.min.js
    83 GET /wp-includes/js/jquery/ui/sortable.min.js
    84 GET /wp-includes/js/jquery/ui/core.min.js
    84 GET /wp-includes/js/jquery/ui/widget.min.js
    85 GET /wp-includes/js/jquery/ui/mouse.min.js
    88 GET /wp-includes/css/dist/block-library/style.min.css
    92 GET /wp-content/uploads/2014/05/logo_urban.png
    93 GET /wp-includes/js/wp-emoji-release.min.js
    107 GET /wp-content/plugins/contact-form-7/includes/css/styles.css
    116 GET /wp-includes/js/wp-embed.min.js
    122 GET /wp-includes/js/jquery/jquery-migrate.min.js
    125 GET /wp-content/plugins/contact-form-7/includes/js/scripts.js
    126 GET /wp-includes/js/jquery/jquery.js
    308 GET /wp-json/yoast/v1/prominent_words
    396 GET /robots.txt
    574 GET /wp-login.php
    663 GET /

    Most Recent top 20 GET requests:
    2 GET /wp-content/plugins/cssigniter-shortcodes/src/fonts/fontawesome-webfont.woff2
    2 GET /wp-content/plugins/cssigniter-shortcodes/src/js/jquery.flexslider.js
    2 GET /wp-content/themes/technico/js/superfish.js
    2 GET /wp-includes/js/jquery/jquery.js
    2 GET /wp-includes/js/jquery/jquery-migrate.min.js
    2 GET /wp-includes/js/wp-embed.min.js
    2 GET /wp-json/oembed/1.0/embed
    2 GET ///wp-json/wp/v2/users/
    2 GET /zh-hant/
    2 GET /zh-hant/%E6%B2%96%E5%A3%93%E6%A9%9F/
    3 GET /contact/
    3 GET /dev/wp-admin/
    3 GET //install/
    3 GET /reborny/about/
    4 GET /&nbsp
    4 GET /website-design-service-blog/&nbsp
    12 GET /southchinastring/
    25 GET /wp-login.php
    50 GET /robots.txt
    74 GET /

    Top 20 POST requests for:
    4 POST /wp-json/contact-form-7/v1/contact-forms/5/feedback
    4 POST //wp-login.php
    5 POST //cate/dangdouqiao.ASp
    5 POST //xmlrpc.php
    6 POST /wp-admin/update-core.php
    6 POST /zh-hant/
    8 POST /hotcrafthobby/
    16 POST /wp-admin/post.php
    16 POST /wp-json/wp/v2/posts/4426
    18 POST /wp-json/yoast/v1/prominent_words_link/4426
    54 POST /wp-content/plugins/wp-phpmyadmin-extension/lib/phpMyAdmin_1JjuZITe0KGPznkN9D6l5dX/error_report.php
    65 POST /southchinastring/wp-login.php
    77 POST /
    99 POST /hotcrafthobby/wp-cron.php
    163 POST /hotcrafthobby/wp-admin/admin-ajax.php
    227 POST /wp-json/wp/v2/yst_prominent_words
    535 POST /xmlrpc.php
    575 POST /wp-login.php
    1359 POST /wp-cron.php
    1925 POST /wp-admin/admin-ajax.php

    Most Recent top 20 POST requests:
    1 POST /assets/images/blackhat.php
    1 POST /components/com_jce/editor/tiny_mce/plugins/imgmanager_ext/classes/image/imagick.php
    1 POST /contact/
    1 POST /error-logs.php
    1 POST /wp-admin/includes/lock46.php
    1 POST /wp-content/plugins/way2register.php
    1 POST /wp-json/contact-form-7/v1/contact-forms/5/feedback
    1 POST /wp-json/contact-form-7/v1/contact-forms/9/feedback
    2 POST /
    2 POST //cbvvc/mupiaowen.asp
    2 POST /hotcrafthobby/wp-cron.php
    2 POST /zh-hant/
    15 POST /xmlrpc.php
    20 POST /wp-login.php
    65 POST /southchinastring/wp-login.php
    109 POST /wp-cron.php
    217 POST /wp-admin/admin-ajax.php

    Top 20 IP addresses that have been accessing your site:
    Do you want geo location check for the IPs? [yes/no]
    yes
    3948 84.202.100.14 Norway
    1311 139.59.96.33 Singapore
    995 222.79.50.74 China
    837 110.167.93.145 China
    772 124.235.138.14 China
    722 223.166.74.9 China
    682 113.128.105.94 China
    628 27.211.56.183 China
    556 1.30.28.77 China
    549 222.94.212.104 China
    543 58.244.10.241 China
    488 185.69.144.24 United Kingdom
    445 222.94.195.46 China
    432 121.57.12.85 China
    361 121.57.229.55 China
    294 113.128.105.226 China
    283 114.33.16.191 Taiwan
    251 61.238.142.146 Hong Kong
    230 210.3.196.182 Hong Kong

    Most Recent top 20 IP addresses:
    185 84.202.100.14 Norway
    104 139.59.96.33 Singapore
    87 62.210.143.10 France
    73 183.89.212.199 Thailand
    69 36.237.55.63 Taiwan
    59 180.215.255.141 India
    31 110.235.33.135 India
    16 77.75.77.101 Czech Republic
    13 216.244.66.226 United States
    10 34.73.85.242 United States
    7 77.88.5.176 Russian Federation
    6 199.58.86.211 United States
    5 35.231.80.6 United States
    5 216.244.66.230 United States
    5 193.106.30.99 Ukraine
    4 66.249.65.134 United States
    4 64.202.185.246 United States
    4 62.84.58.177 Kazakhstan
    4 46.229.168.163 United States

  6. #16
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    2,852
    Distro
    Ubuntu Development Release

    Re: CPU load average spikes every hour with CPU 55% and memory 82% only

    Quote Originally Posted by azinity-tom View Post
    So how can we find out which IPs from China are malicious? Shall we block those IPs?
    My input here is exactly the same as you got from "Neo" on that other thread. I get the impression that you are looking for a short cut, but I am not aware of one. This stuff takes a lot of time and effort.
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  7. #17
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    20,468
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: CPU load average spikes every hour with CPU 55% and memory 82% only

    A few things to consider ...

    Why let any non-corporate user access /wp-admin/ on any request?
    Client requests to places they shouldn't go shouldn't be allowed.
    if you aren't using WP at all, then any request to /wp-* should immediately be added to a block list. Tools like fail2ban can do that.
    Subnets that aren't for the company or clients shouldn't be allowed at all.

    Have you enabled micro-caching at all? 95% of all requests should be for read-only data, so a delay of 2 seconds wouldn't matter at all, but it will drastically reduce the server and DBMS load.

    if your droplet is small perhaps spending $10/month more would make this issue go away. That is certainly cheaper than a highly paid developer wasting too much time on the problem. Or splitting the front-end from the DBMS?

    BTW, my 20 or so websites block over 8K subnets due to abuses. I'm pretty quick to block entire /8 networks from countries we don't do business inside. I also block a bunch of vps providers because they've become hideouts for illegal activities. I don't really care if some bot can't scan my sites. Blocking bots is part of what the reverse-proxy does.

Page 2 of 2 FirstFirst 12

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •