Page 1 of 13 12311 ... LastLast
Results 1 to 10 of 122

Thread: HOWTO: Set maximum CPU consumption in percentage by any process

  1. #1
    Join Date
    Mar 2007
    Beans
    623

    HOWTO: Set maximum CPU consumption in percentage by any process

    EDIT: Tutorial updated on September 15, 2011. New features added to scripts and some bugs fixed.
    EDIT: 'Attention' info added for properly specifying the variables on January 14, 2014.

    Purpose of cpulimit daemon:
    Daemon runs in background and checks if some process is consuming more then 20% of CPU and if it does then daemon lowers CPU consumption of particular process to maximum of 20%. The goal is to have no single process that consumes more then 20% of CPU power.

    Note: If you would like to omit only one process then you don't need this "cpulimit daemon". In this case you only need cpulimit program to be executed from terminal.

    Tested environment:
    Cpulimit daemon was tested on Ubuntu 8.04 LTS and Ubuntu 10.04 LTS. However it should be running fine on other Ubuntu versions and also on other Linux distributions, because it does not uses any Ubuntu specific code.


    1. INSTALL PACKAGES

    1. Install cpulimit package.
    Code:
    sudo apt-get install cpulimit
    2. Install gawk package.
    Code:
    sudo apt-get install gawk

    2. CREATE CPULIMIT DAEMON FILE

    Open text editor with root privileges and save bellow daemon script text to new file /usr/bin/cpulimit_daemon.sh

    Code:
    #!/bin/bash
    # ==============================================================
    # CPU limit daemon - set PID's max. percentage CPU consumptions
    # ==============================================================
    
    # Variables
    CPU_LIMIT=20       	# Maximum percentage CPU consumption by each PID
    DAEMON_INTERVAL=3  	# Daemon check interval in seconds
    BLACK_PROCESSES_LIST=   # Limit only processes defined in this variable. If variable is empty (default) all violating processes are limited.
    WHITE_PROCESSES_LIST=   # Limit all processes except processes defined in this variable. If variable is empty (default) all violating processes are limited.
    
    # Check if one of the variables BLACK_PROCESSES_LIST or WHITE_PROCESSES_LIST is defined.
    if [[ -n "$BLACK_PROCESSES_LIST" &&  -n "$WHITE_PROCESSES_LIST" ]] ; then    # If both variables are defined then error is produced.
       echo "At least one or both of the variables BLACK_PROCESSES_LIST or WHITE_PROCESSES_LIST must be empty."
       exit 1
    elif [[ -n "$BLACK_PROCESSES_LIST" ]] ; then                                 # If this variable is non-empty then set NEW_PIDS_COMMAND variable to bellow command
       NEW_PIDS_COMMAND="top -b -n1 -c | grep -E '$BLACK_PROCESSES_LIST' | gawk '\$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
    elif [[ -n "$WHITE_PROCESSES_LIST" ]] ; then                                 # If this variable is non-empty then set NEW_PIDS_COMMAND variable to bellow command
       NEW_PIDS_COMMAND="top -b -n1 -c | gawk 'NR>6' | grep -E -v '$WHITE_PROCESSES_LIST' | gawk '\$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
    else
       NEW_PIDS_COMMAND="top -b -n1 -c | gawk 'NR>6 && \$9>CPU_LIMIT {print \$1}' CPU_LIMIT=$CPU_LIMIT"
    fi
    
    # Search and limit violating PIDs
    while sleep $DAEMON_INTERVAL
    do
       NEW_PIDS=$(eval "$NEW_PIDS_COMMAND")                                                                    # Violating PIDs
       LIMITED_PIDS=$(ps -eo args | gawk '$1=="cpulimit" {print $3}')                                          # Already limited PIDs
       QUEUE_PIDS=$(comm -23 <(echo "$NEW_PIDS" | sort -u) <(echo "$LIMITED_PIDS" | sort -u) | grep -v '^$')   # PIDs in queue
    
       for i in $QUEUE_PIDS
       do
           cpulimit -p $i -l $CPU_LIMIT -z &   # Limit new violating processes
       done
    done

    3. CHANGE VARIABLES TO YOUR ENVIRONMENT NEEDS

    CPU_LIMIT
    Change this variable in above script if you would like to omit CPU consumption for every process to any other percentage then 20%. Please read "If using SMP computer" chapter bellow if you have SMP computer (more then 1 CPU or CPU with more then 1 core).

    DAEMON_INTERVAL
    Change this variable in above script if you would like to have more/less regular checking. Interval is in seconds and default is set to 3 seconds.

    BLACK_PROCESS_LIST and WHITE_PROCESSES_LIST
    Variable BLACK_PROCESSES_LIST limits only specified processes. If variable is empty (default) all violating processes are limited.

    Variable WHITE_PROCESSES_LIST limits all processes except processes defined in this variable. If variable is empty (default) all violating processes are limited.

    One or both of the variables BLACK_PROCESSES_LIST and WHITE_PROCESSES_LIST has to be empty - it is not logical that both variables are defined.

    You can specify multiple processes in one of this two variables using delimiter characters "|" (without double quotes). Sample: if you would like to cpulimit all processes except mysql, firefox and gedit processes set variable: WHITE_PROCESSES_LIST="mysql|firefox|gedit"

    Attention: Make sure you don't put any spaces in variable, for example WHITE_PROCESS_LIST= "mysql|firefox|gedit" (notice space character after = character) is not going to work correctly. You need to specify the variable without spaces: WHITE_PROCESS_LIST="mysql|firefox|gedit"

    4. PROCEDURE TO AUTOMATICALLY START DAEMON AT BOOT TIME

    1. Set file permissions for root user:
    Code:
    sudo chmod 755 /usr/bin/cpulimit_daemon.sh
    2. Open text editor with root privileges and save bellow script to new file /etc/init.d/cpulimit
    Code:
    #!/bin/sh
    #
    # Script to start CPU limit daemon
    #
    set -e
    
    case "$1" in
    start)
    if [ $(ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print $1}' | wc -l) -eq 0 ]; then
        nohup /usr/bin/cpulimit_daemon.sh >/dev/null 2>&1 &
        ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print}' | wc -l | gawk '{ if ($1 == 1) print " * cpulimit daemon started successfully"; else print " * cpulimit daemon can not be started" }'
    else
        echo " * cpulimit daemon can't be started, because it is already running"
    fi
    ;;
    stop)
    CPULIMIT_DAEMON=$(ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print $1}' | wc -l)
    CPULIMIT_INSTANCE=$(ps -eo pid,args | gawk '$2=="cpulimit" {print $1}' | wc -l)
    CPULIMIT_ALL=$((CPULIMIT_DAEMON + CPULIMIT_INSTANCE))
    if [ $CPULIMIT_ALL -gt 0 ]; then
        if [ $CPULIMIT_DAEMON -gt 0 ]; then
            ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print $1}' | xargs kill -9   # kill cpulimit daemon
        fi
    
        if [ $CPULIMIT_INSTANCE -gt 0 ]; then
            ps -eo pid,args | gawk '$2=="cpulimit" {print $1}' | xargs kill -9                    # release cpulimited process to normal priority
        fi
        ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print}' | wc -l | gawk '{ if ($1 == 1) print " * cpulimit daemon can not be stopped"; else print " * cpulimit daemon stopped successfully" }'
    else
        echo " * cpulimit daemon can't be stopped, because it is not running"
    fi
    ;;
    restart)
    $0 stop
    sleep 3
    $0 start
    ;;
    status)
    ps -eo pid,args | gawk '$3=="/usr/bin/cpulimit_daemon.sh"  {print}' | wc -l | gawk '{ if ($1 == 1) print " * cpulimit daemon is running"; else print " * cpulimit daemon is not running" }'
    ;;
    esac
    exit 0
    3. Change file's owner to root:
    Code:
    sudo chown root:root /etc/init.d/cpulimit
    4. Change permissions:
    Code:
    sudo chmod 755 /etc/init.d/cpulimit
    5. Add script to boot-up procedure directories:
    Code:
    sudo update-rc.d cpulimit defaults
    6. Reboot to check if script starts cpulimit daemon at boot time:
    Code:
    sudo reboot

    5. MANUALLY CHECK, STOP, START AND RESTART DAEMON

    Note: Daemon and service in this tutorial has equal meaning.

    Note: For users using prior to Ubuntu 8.10 (like Ubuntu 8.04 LTS) instead of service command use "sudo /etc/init.d/cpulimit status/start/stop/restart" syntax or install sysvconfig package using command: sudo apt-get install sysvconfig

    Check if cpulimit service is running
    Check command returns: "cpulimit daemon is running" if service is running, or "cpulimit daemon is not running" if service is not running.
    Code:
    sudo service cpulimit status
    Start cpulimit service
    You can manually start cpulimit daemon which will start to omit CPU consumption.
    Code:
    sudo service cpulimit start
    Stop cpulimit service
    Stop command stops cpulimit daemon (so no new process will be limited) and also sets to all existing limited processes to have full access to CPU, just like it was before cpulimit was not running.
    Code:
    sudo service cpulimit stop
    Restart cpulimit service
    If you change some variables settings in /usr/bin/cpulimit_daemon.sh like CPU_LIMIT, DAEMON_INTERVAL, BLACK_PROCESSES_LIST or WHITE_PROCESSES_LIST, then after changing settings you must restart service.
    Code:
    sudo service cpulimit restart

    6. CHECK CPU CONSUMPTION WITH OR WITHOUT CPULIMIT DAEMON

    Without daemon
    1. stop cpulimit daemon (sudo service cpulimit stop)
    2. execute CPU intensive tasks in background
    3. execute command: top and check for %CPU column
    Result of %CPU is probably more then 20% for each process.

    With daemon turned on
    1. start cpulimit daemon (sudo service cpulimit start)
    2. execute the same CPU intensive tasks in background
    3. execute command: top and check for %CPU column
    Result of %CPU should be maximum 20% for each process.
    Note: Don't forget at beginning %CPU can be more then 20%, because daemon has to catch violating process in interval of 3 seconds (set in script by default)


    7. IF USING SMP COMPUTER

    I have tested this code on Intel dual-core CPU computer - that behaves like SMP computer. Don't forget that top command and also cpulimit by default behaves in Irix mode, where 20% means 20% of one CPU. If there are two CPUs (or dual-core) then total %CPU can be 200%. In top command Irix mode can be turned off with command I (pressing <Shift>+i when top command is running) and Solaris mode is turned on, where total amount of CPU is divided by number of CPUs, so %CPU can be no more then 100% on any number of CPU computer. Please read more info about top command in top man page (search for I command). Please also read more about how cpulimit is operating on SMP computer in cpulimit official page.

    But how does cpulimit daemon operates on SMP computer? Always in Irix mode. So if you would like to spend 20% of CPU power on 2-CPU computer then 40% should be used for CPU_LIMIT variable in cpulimit daemon script.


    8. UNINSTALL CPULIMIT DAEMON AND CPULIMIT PROGRAM

    If you would like to get rid of cpulimit daemon you can clean up your system by removing cpulimit daemon and uninstalling cpulimit program.

    1. Stop cpulimit daemon
    Code:
    sudo service cpulimit stop             # Stop cpulimit daemon and all cpulimited processes
    2. Remove daemon from boot-up procedure
    Code:
    sudo update-rc.d -f cpulimit remove    # Remove symbolic links
    3. Delete boot-up procedure
    Code:
    sudo rm /etc/init.d/cpulimit           # Delete cpulimit boot-up script
    4. Delete cpulimit daemon
    Code:
    sudo rm /usr/bin/cpulimit_daemon.sh    # Delete cpulimit daemon script
    5. Uninstall cpulimit program
    Code:
    sudo apt-get remove cpulimit
    6. Uninstall gawk program
    If you don't need this program for any other script, you can remote it.
    Code:
    sudo apt-get remove gawk

    9. NOTE ABOUT AUTHORS

    I have just written daemon for cpulimit (bash scripts above). I am not the author of cpulimit project. If you need more info about cpulimit program, please read official cpulimit web page: http://cpulimit.sourceforge.net/.

    Regards,
    Abcuser
    Last edited by abcuser; January 14th, 2014 at 08:35 AM.

  2. #2
    Join Date
    Sep 2006
    Location
    France.
    Beans
    Hidden!
    Distro
    Lubuntu Development Release

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Edited thread title

  3. #3

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    a very handy daemon to control the cpu usage in userspace. your instructions are comprehensive but you forgot to include one thing. you have to make the daemon executable. otherwise it will say something like "nohup: cannot run command `/root/cpulimit_daemon.sh': Permission denied".

    nice job!
    .

  4. #4
    Join Date
    Mar 2007
    Beans
    623

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    @bekirserifoglu, thanks I have updated the how-to.

  5. #5
    Join Date
    Dec 2005
    Location
    Western Australia
    Beans
    11,318
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Hang on a sec... can I just ask for clarification on something? Let's say you're playing a game or transcoding video, where the process uses nearly 100% CPU and you want it that way. Will it then run at 1.5th of the speed with your daemon turned on?

    If so, who wants that? I can understand limiting a specified process, but I can't understand limiting every process.
    I try to treat the cause, not the symptom. I avoid the terminal in instructions, unless it's easier or necessary. My instructions will work within the Ubuntu system, instead of breaking or subverting it. Those are the three guarantees to the helpee.

  6. #6
    Join Date
    Mar 2007
    Beans
    623

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Hi,
    @3rdalbum, thanks for sharing your comment.

    Let me explain. On PCs this daemon behaviour is probably not desired. Limiting specific process is more desired, like you described. But I have written this daemon for server environment and not every server environment is suitable for this daemon! I have virtual computers server environment.

    Lets see a sample. In my case I have an Intel server where I have multiple virtual machines (also Ubuntu and some Windows too) using VirtualBox virtualization tool. On Ubuntu virtual host computer I have several virtual machines. I can omit how much memory can be consumed by each application in VirtualBox GUI, but can't enforce CPU usage. So I have written a daemon to control maximum CPU consumption by each virtual machine. I know I could run every virtual machine with only cpulimit command, but I am not only administrator on this Linux boxes and new virtual machines are added on the fly and others are moved to other Intel computers. So if administrator forgets to set cpulimit command to one new specific virtual machine, this new machine would consume all CPU power and take down other very important virtual machines.

    Beside using cpulimit daemon, I can still set that some virtual machine can only use for example 10% of CPU power if desirable by executing cpulimit command by hand. But I can't allow one single process to make all others virtual systems down.
    Regards
    Last edited by abcuser; January 17th, 2009 at 07:17 PM.

  7. #7
    Join Date
    Mar 2007
    Beans
    20

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Thank you, your time and effort is much appreciated.

    I know how to run a single process through cpulimit,
    is there a way to limit a boot service with cpulimit?
    I see that there is a way to limit a service, but I want it
    limited from startup without me having to type anything...

    Thanks

  8. #8
    Join Date
    Mar 2007
    Beans
    623

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Hi,
    don't know if this is possible, it is strange because boot process starts deamon that handles CPU.

    Why would you like to omit CPU to boot process? I always like to boot up as soon as possible.
    Regards

  9. #9
    Join Date
    Mar 2007
    Beans
    20

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Quote Originally Posted by abcuser View Post
    Hi,
    don't know if this is possible, it is strange because boot process starts deamon that handles CPU.

    Why would you like to omit CPU to boot process? I always like to boot up as soon as possible.
    Regards
    An example is: Folding@Home, I have it set up to use a service, so at boot it starts working.
    I don't want to dedicate a full core to it though. I can manually cpulimit the PID, however
    I'd like a way to do that automatically. So, I didn't really mean a "boot process"
    so much as a process that starts on boot.

    Thanks.

  10. #10
    Join Date
    Mar 2007
    Beans
    623

    Re: HOWTO: Set maximum CPU consumption in percentage by any process

    Hi,
    OK, now I understand. The idea would probably go in this way. At boot time you start some script to start you program and after that some kind of cpulimit daemon has to start and try to catch PID of your desired program and lower the CPU of that program.

    I have written this script some time ago, so it should be something like this (instead of cpulimit daemon write this script)
    #!/bin/bash
    cpulimit -p $( ps -eo pid,args | grep "process name you would like to CPU limit" | grep -v grep | gawk '{print $1}' ) -l 5 -z &

    Note: instead of string "process name you would like to CPU limit" write your process name getting from "ps -eo pid, args" command. Above script assumes you would like to lower CPU to 5% (string: "-l 5" at the end of command), if you would like any other limit change this settings.

    After that you have to make sure that your program has to start first and then this new cpulimit script goes second. Assuming you scrip has 20 priority which is by default, run the following script:
    update-rc.d cpulimit-daemon defaults 21 19

    Note: cpulimit-daemon is your script file name (select appropriate name) to start cpulimit daemon.

    Hope this helps,
    Regards

Page 1 of 13 12311 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •