Page 12 of 13 FirstFirst ... 210111213 LastLast
Results 111 to 120 of 122

Thread: HOWTO: Set maximum CPU consumption in percentage by any process

  1. #111
    Join Date
    Mar 2007
    Beans
    680

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Hi,
    first for all I don't know why MySQL is crashing. According to your post it looks like regardless of variable you put in, ALL of the processes gets into limiting state (echo 40) and so this affects MySQL processes in some way - maybe MySQL recognizes that something has changed with its process and it crashes.

    When you change the code and run it one more time are you sure you have STOPPED previously executed daemon? You see in the code is endless while loop, so you have to kill existing running application (= daemon) before running another instance of it. Before running daemon again search if there is an already running process by:
    Code:
    ps -eo pid,args | grep "cpulimit_daemon.sh"
    If you get some output then there is already a daemon running and you have to kill it first before running another instance of daemon.

    The next thing you can do is to add "echos" into after all of the lines that some variable is specified and echo the variable content. I have prepared debugging code see bellow. Try to run this code (but make sure you don't have a daemon already running).

    Code:
    #!/bin/bash
    # ==============================================================
    # CPU limit daemon - set PID's max. percentage CPU consumptions
    # ==============================================================
    
    # Variables
    CPU_LIMIT=20       	# Maximum percentage CPU consumption by each PID
    DAEMON_INTERVAL=3  	# Daemon check interval in seconds
    BLACK_PROCESSES_LIST=   # Limit only processes defined in this variable. If variable is empty (default) all violating processes are limited.
    WHITE_PROCESSES_LIST=   # Limit all processes except processes defined in this variable. If variable is empty (default) all violating processes are limited.
    
    # Check if one of the variables BLACK_PROCESSES_LIST or WHITE_PROCESSES_LIST is defined.
    if [[ -n "$BLACK_PROCESSES_LIST" &&  -n "$WHITE_PROCESSES_LIST" ]] ; then    # If both variables are defined then error is produced.
       echo "At least one or both of the variables BLACK_PROCESSES_LIST or WHITE_PROCESSES_LIST must be empty."
       echo "1. BLACK_PROCESSES_LIST: " $BLACK_PROCESSES_LIST
       echo "2. WHITE_PROCESSES_LIST: " $WHITE_PROCESSES_LIST
       exit 1
    elif [[ -n "$BLACK_PROCESSES_LIST" ]] ; then                                 # If this variable is non-empty then set NEW_PIDS_COMMAND variable to bellow command
       NEW_PIDS_COMMAND="top -b -n1 -c | grep -E '$BLACK_PROCESSES_LIST' | gawk '\$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
       echo "3. CPU_LIMI: " $CPU_LIMIT
       echo "4. BLACK_PROCESSES_LIST: " $BLACK_PROCESSES_LIST
       echo "5. NEW_PIDS_COMMAND: " $NEW_PIDS_COMMAND
    elif [[ -n "$WHITE_PROCESSES_LIST" ]] ; then                                 # If this variable is non-empty then set NEW_PIDS_COMMAND variable to bellow command
       NEW_PIDS_COMMAND="top -b -n1 -c | gawk 'NR>6' | grep -E -v '$WHITE_PROCESSES_LIST' | gawk '\$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
       echo "6. CPU_LIMI: " $CPU_LIMIT
       echo "7. WHITE_PROCESSES_LIST: " $WHITE_PROCESSES_LIST
       echo "8. NEW_PIDS_COMMAND: " $NEW_PIDS_COMMAND
    else
       NEW_PIDS_COMMAND="top -b -n1 -c | gawk 'NR>6 && \$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
       echo "9. CPU_LIMI: " $CPU_LIMIT
       echo "10. NEW_PIDS_COMMAND: " $NEW_PIDS_COMMAND
    fi
    
    # Search and limit violating PIDs
    while sleep $DAEMON_INTERVAL
    do
       NEW_PIDS=$(eval "$NEW_PIDS_COMMAND")                                                                    # Violating PIDs
       LIMITED_PIDS=$(ps -eo args | gawk '$1=="cpulimit" {print $3}')                                          # Already limited PIDs
       QUEUE_PIDS=$(comm -23 <(echo "$NEW_PIDS" | sort -u) <(echo "$LIMITED_PIDS" | sort -u) | grep -v '^$')   # PIDs in queue
    
       echo "11. DAEMON_INTERVAL: " $DAEMON_INTERVAL
       echo "12. CPU_LIMI: " $CPU_LIMIT
       echo "13. NEW_PIDS: " $NEW_PIDS
       echo "14. LIMITED_PIDS: " $LIMITED_PIDS
       echo "15. QUEUE_PIDS: " $QUEUE_PIDS
    
       for i in $QUEUE_PIDS
       do
           cpulimit -p $i -l $CPU_LIMIT -z &   # Limit new violating processes
       done
    done
    Please post the outputs of echos to the forum.

    There is also possible that Debian has something different then Ubuntu. This code was tested on Ubuntu 10.04 (= April 2010) few years back. It is possible that Debian uses some never version of one of the programs and e.g. outputs something that script does not expects. For example there are "gawk" commands with searches the data in one of the columns, if some output has changed then script is not going to be working fine.

    One more thing to discuss is are you sure you need a cpulimit_daemon? How do you run tar and zip commands? Are they executed manually from shell by you or some other way? You see from my tutorial there is a note at the top of the tutorial: "Note: If you would like to omit only one process then you don't need this "cpulimit daemon". In this case you only need cpulimit program to be executed from terminal."

    So if you execute the code from shell manually then you can:
    1. Execute the 'tar' or 'zip' command.
    2. Check the process_pid: ps -eo pid,args | grep tar
    3. Limit process manually: cpulimit -p process_pid -l cpu_limit -z &
    Note: process_pid is from command 2, cpu_limit is for example 20 for 20%.

    Regards
    Last edited by abcuser; January 14th, 2014 at 08:05 AM.

  2. #112
    Join Date
    Mar 2007
    Beans
    680

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Quote Originally Posted by likudio View Post
    Code:
    BLACK_PROCESSES_LIST= ""   # Limit only processes defined in this variable. If variable is empty (default) all violating processes are limited.
    ...and the result:

    Code:
    root@hsh:~# /usr/bin/cpulimit_daemon.sh
    /usr/bin/cpulimit_daemon.sh: line 9: : command not found
    Process 23990 detected
    Process 23990 dead!
    There is a problem you didn't specify the variable in correct way! The variable should be empty, but you have defined it to have an empty string, but executing it in double quotes. See variable black_list should be defined as:
    Code:
    BLACK_PROCESSES_LIST=
    but you have defined it as:
    Code:
    BLACK_PROCESSES_LIST= ""
    Remove the double quotes from variable value. See my first post in this thread for original code.

    Also make sure there are no ADDITIONAL spaces in you code. For example if you add or remove some of the space in if statement then command can have completely different meaning and it will not work as designed or not work at all. Spaces are very important!!!

    Also how did you define black_process_list variable?
    Did you write:
    Code:
    BLACK_PROCESSES_LIST= "tar|zip"
    Note: So space after "=" character? If yes, then you code is not going to work correctly. In this case instead of defining a variable value you gonna execute tar command and pass the output into zip command... Strange thing can happen if you add additional space.
    Code:
    BLACK_PROCESSES_LIST="tar|zip"
    Note: You see there is no space after "=" character.

    Hope this helps.
    Last edited by abcuser; January 14th, 2014 at 08:26 AM.

  3. #113
    Join Date
    Mar 2007
    Beans
    680

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    It looks like old posts can be edited again. This was a limitation the last time I have looked at editing them. So in first post I have added the 'Attention' note on how to properly specify variables.

  4. #114
    Join Date
    May 2013
    Beans
    4

    Cool Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Sorry for the late answer, I didn't have time until now to dig the problem.

    Quote Originally Posted by abcuser View Post
    Spaces are very important!!!
    Unbelievable, but you were so right.
    That space before the equal sign was all the issue.
    After removing it, it finally goes in the right "if" section.
    And it limits processes as it should.

    On the other hand, for MySQL I have to say that it has the same behavior, but somehow can be explained.
    In Debian (perhaps in other distros too), if you install MySQL with aptitude, it comes with a monitoring package named "mysqld_safe" which constantly monitors the mysqld process. For some reason, when it sees it struggling (limited by cpulimit) it kills it and restarts it.

    To avoid that, you must disable mysqld_safe, but I wouldn't recommend that as it does some other things too in order to keep mysql up and running.
    For batch jobs like backups and so on, simply limit mysqldump or gzip (considering you export to a gzip archive) or so on.

    Anyway, thank you once again for your help, I wouldn't think in ages that those spaces can be the issue.

    Best regards,

  5. #115
    Join Date
    Mar 2007
    Beans
    680

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Quote Originally Posted by likudio View Post
    For some reason, when it sees it struggling (limited by cpulimit) it kills it and restarts it.
    Are you still having problem with MySQL? Can you try using the debug code from http://ubuntuforums.org/showthread.p...1#post12899891 and see if some of the mysql process gets limited? If yes then you should change the variable BLACK_PROCESSES_LIST to meet you requirements.
    Quote Originally Posted by likudio View Post
    To avoid that, you must disable mysqld_safe, but I wouldn't recommend that as it does some other things too in order to keep mysql up and running.
    For batch jobs like backups and so on, simply limit mysqldump or gzip (considering you export to a gzip archive) or so on.
    This is probably work around, but I can't suggest anything like this, because I am not an MySQL expert. Like written before, I suggest to trace down the process that gets limited and appropriatly change BLACK_PROCESSES_LIST variable.

  6. #116
    Join Date
    Dec 2010
    Beans
    4

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Hi abcuser, thank you for this daemon - I am finding it incredibly useful. I have found others that added the following to beginning of the cpulimit daemon:

    #!/bin/bash
    ### BEGIN INIT INFO
    # Provides: cpulimitd
    # Required-Start: $remote_fs $syslog
    # Required-Stop: $remote_fs $syslog
    # Default-Start: 2 3 4 5
    # Default-Stop: 0 1 6
    # Short-Description: Script to start CPU limit daemon
    # Description:
    #
    ### END INIT INFO

    Or Debian complains (insserv).

    I just need some input on a couple of things....

    I am running this on a Debian Wheezy system which for the most part seems to be working well. The only problem is that when the daemon is running, sometimes bash will background an apt-get process while it is running an apt-get update or install etc. and the output "[1]+ Stopped apt-get update" will be displayed... if I stop the cpulimit daemon, this problem goes away...

    Is this an issue that you have run into? I know you have posted this for Ubuntu and not Debian, but any help/adjustments you can offer would be very much appreciated.
    Last edited by bobmarley2021; January 21st, 2014 at 12:00 AM.

  7. #117
    Join Date
    Mar 2007
    Beans
    680

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Quote Originally Posted by bobmarley2021 View Post
    The only problem is that when the daemon is running, sometimes bash will background an apt-get process while it is running an apt-get update or install etc.
    Daemon searches for pattern in process names including the whole string it uses when run. When using update/install some of the processes that meets pattern! can be caught by daemon.

    Did you specify a BLACK_PROCESS_LIST or WHITE_PROCESSES_LIST variable? I suggest to define one of them and use proper regular expression to specify processes you need as exactly as you can. For example if you would like to limit only Gedit program, then run Gedit and monitor what is outputted by command:
    Code:
    top -b -n1 -c | grep -E 'gedit'
    Then try to add regular expressions like "^" for beginning of the command and "$" for end of the command. So example:
    Code:
    top -b -n1 -c | grep -E '^gedit$'
    Above command will most probably avoid the problem at apt-get update/install, because it will not contain any spaces that apt-get update/install produces. So in this case use:
    Code:
    BLACK_PROCESS_LIST="^gedit$"


    The other work-around is maybe you can stop limit daemon when updating the system. For example if you are updating your system at night when most probably system is not occupied too much, so you can write script:
    Code:
    [code to stop limit daemon]
    apt-get update...
    [code to start limit daemon]
    Hope this helps
    Last edited by abcuser; January 21st, 2014 at 07:51 AM.

  8. #118
    Join Date
    Dec 2010
    Beans
    4

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Thank you for your reply. After considering the options I have decided to add the list of offending processes to the black list, instead of white-listing apt-get etc and this has solved the issue. Thanks again for this great daemon.

  9. #119
    Join Date
    Jan 2014
    Location
    California
    Beans
    34
    Distro
    Xubuntu 15.04 Vivid Vervet

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    What are the benefits of controlling your cpu to such a lower % rather than just letting it run free? 0.0

  10. #120
    Join Date
    Mar 2007
    Beans
    680

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    @dale2, if you are asking this kind of question, then you don't need this daemon.

    The whole point is that one or more processes do not occupy whole your CPU and makes your computer unusable. For example you decided to compile some program that will take 30 minutes to compile, but when you execute this compile you can't work with your computer, because it is 100% CPU occupied. With this kind of restriction that daemon gives you are able to work normally like browsing the web, typing some text in office suite and similar while a background process does the compile work for you.

Page 12 of 13 FirstFirst ... 210111213 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •