I have finished the storage check script and ended up using a single script instead of two.
The bulk of the script is error checking and avoiding problems...although I am sure it will not catch 100% of all potential problems but it "should" be sufficient.
I will share the code here but BE WARNED, this script will resize your File System under certain circumstances. Even though it should be safe, you need to make sure you designed your LVM for such a script and you have the correct references for any potential file systems as noted near line 152 that is commented "Check validity of File System names." I chose to use the LV names to keep it short but you can set it to whatever you like. I did it this way to avoid operator typos that run it manually or set it up wrong in the crontab. All the important stuff is pre-defined inside the script.
Here is how my system is designed:
Volume Group = LVG (Size=342 GB, 0 free)
Logical Volumes in LVG:
root (Size=8GB)
opt (Size=75GB)
bak (Size=125GB)
temp (Size=125GB)
File Systems:
/dev/mapper/LVG-root (Size=8GB), Mounted on /
/dev/mapper/LVG-opt (Size=50GB), Mounted on /opt
/dev/mapper/LVG-bak (Size=100GB), Mounted on /var/backup
/dev/mapper/LVG-temp (Size=100GB), Mounted on /var/temp
Here is a sample crontab file for checking 3 file systems that I expect will grow over time. Note that I am not including the root partition since I do not expect it to be growing over time...however, I do have Nagios monitoring it so I will get notified but by a different method. This process is meant to check AND grow the FS as needed. When they get within 5 GB of being full, it will try to increment the file system by 1 GB. Click on this link to read about my thoughts on how to maintain a crontab schedule.
Here is the script. Notice that the resize2fs command is currently commented out on line 64. That means this script will operate in a mode that will not affect the system as it is now. You uncomment the command by removing the pound characters from the beginning to make it actually perform a resize. Feel free to try it out on the commandline by issuing various threshold values which are below the current available space and above to see how it works without fear of it doing anything to your system.Code:######################################## # Name: Crontab Schedule for root user # Author: LHammonds ############# Update Log ############### # 2012-05-01 - LTH - Created schedule ######################################## SHELL=/bin/sh PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # m h dom mon dow command # # Get date/time from network time server every hour # 0 * * * * /etc/network/if-up.d/ntpdate # # Daily check for available space on /opt # 0 1 * * * /var/scripts/prod/check-storage.sh opt 5 1 > /dev/null 2>&1 # # Daily check for available space on /var/backup # 0 2 * * * /var/scripts/prod/check-storage.sh bak 5 1 > /dev/null 2>&1 # # Daily check for available space on /var/temp # 0 3 * * * /var/scripts/prod/check-storage.sh temp 5 1 > /dev/null 2>&1
/var/scripts/prod/check-storage.sh
Oh, one last thing...my code uses functions and variable I have pulled out into a common-use file which are specific to my environment (you need to change to suit your environment). Here are the pertinent bits:Code:#!/bin/bash ############################################# ## Name : check-storage.sh ## Version : 1.2 ## Date : 2012-05-01 ## Author : LHammonds ## Purpose : Check available space for a file system and expand if necessary. ## Compatibility : Verified on Ubuntu Server 10.04.4 LTS ## Requirements : None ## Run Frequency : Recommend once per day for each FS to monitor. ## Parameters : ## 1 = (Required) File System name (e.g. opt) ## 2 = (Required) File System Threshold (e.g. 2G) ## 3 = (Required) Amount to increase File System (e.g. 1G) ## Exit Codes : ## 0 = Success (either nothing was done or FS expanded without error) ## 1 = ERROR: Missing or incorrect parameter(s) ## 2 = ERROR: Invalid parameter value(s) ## 4 = ERROR: Lock file detected ## 8 = ERROR: Resize2fs error ## 16 = SEVERE: No room to expand ## 32 = ERROR: Script not run by root user ################ CHANGE LOG ################# ## DATE WHO WHAT WAS CHANGED ## ---------- --- ---------------------------- ## 2012-05-01 LTH Created script. ## 2012-05-02 LTH Improved log messages. ## 2012-05-02 LTH Improved email messages. ############################################# ## Import standard variables and functions. ## source /var/scripts/common/standard.conf ## Define local variables. LOGFILE="${TEMPDIR}/check-storage.log" LOCKFILE="${TEMPDIR}/check-storage.lock" ErrorFlag=0 ReturnCode=0 ####################################### ## FUNCTIONS ## ####################################### function f_cleanup() { if [ -f ${LOCKFILE} ];then ## Remove lock file so other check space jobs can run. rm ${LOCKFILE} 1>/dev/null 2>&1 fi exit ${ErrorFlag} } function f_showhelp() { echo -e "\nUsage : ${SCRIPTNAME} FileSystemName ThresholdSizeInGB AmountToIncreaseByInGB\n" echo -e "\nExample: ${SCRIPTNAME} opt 2 1\n" } function f_auto-increment() { let RoomInLV=${LVSize}-${FSSize} if [[ ${RoomInLV} -gt ${FSIncreaseBy} ]]; then ## There is room in the LV to increase space to the FS. #resize2fs ${FSVol} ${NewFSSize}G ReturnCode=$? echo "`date +%Y-%m-%d_%H:%M:%S` --- resize2fs ${FSVol} ${NewFSSize}G, ReturnCode=${ReturnCode}" | tee -a ${LOGFILE} if [[ ${ReturnCode} -ne 0 ]]; then ## There was an error in resize2fs. return ${ReturnCode} fi else ## There is not enough room in the LV to increase space in the FS. return 50 fi return 0 } ####################################### ## MAIN PROGRAM ## ####################################### if [ -f ${LOCKFILE} ]; then # Lock file detected. Abort script. echo "Check space script aborted" echo "This script tried to run but detected the lock file: ${LOCKFILE}" echo "Please check to make sure the file does not remain when check space is not actually running." f_sendmail "ERROR: check storage script aborted" "This script tried to run but detected the lock file: ${LOCKFILE}\n\nPlease check to make sure the file does not remain when check space is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LOCKFILE}'" ErrorFlag=4 f_cleanup else echo "`date +%Y-%m-%d_%H:%M:%S` ${SCRIPTNAME}" > ${LOCKFILE} fi ## Requirement Check: Script must run as root user. if [ "$(id -u)" != "0" ]; then ## FATAL ERROR DETECTED: Document problem and terminate script. echo "ERROR: Root user required to run this script." echo "" ErrorFlag=32 f_cleanup fi ## Check existance of required command-line parameters. case "$1" in "") f_showhelp ErrorFlag=1 f_cleanup ;; --help|-h|-?) f_showhelp ErrorFlag=1 f_cleanup ;; *) FSName=$1 ;; esac case "$2" in "") f_showhelp ErrorFlag=1 f_cleanup ;; --help|-h|-?) f_showhelp ErrorFlag=1 f_cleanup ;; *) FSThreshold=$2 ;; esac case "$3" in "") f_showhelp ErrorFlag=1 f_cleanup ;; --help|-h|-?) f_showhelp ErrorFlag=1 f_cleanup ;; *) FSIncreaseBy=$3 ;; esac ## Check validity of File System name. case "${FSName}" in "opt") FSVol="/dev/LVG/opt" FSMap="/dev/mapper/LVG-opt" ;; "bak") FSVol="/dev/LVG/bak" FSMap="/dev/mapper/LVG-bak" ;; "temp") FSVol="/dev/LVG/temp" FSMap="/dev/mapper/LVG-temp" ;; *) echo "ERROR: ${FSName} does not match a known file system defined in this script." f_showhelp ErrorFlag=2 f_cleanup ;; esac ## Check validity of threshold value. test ${FSThreshold} -eq 0 1>/dev/null 2>&1 if [[ $? -eq 2 ]]; then ## Threshold parameter is not an integer. echo "ERROR: ${FSThreshold} is not an integer." f_showhelp ErrorFlag=2 f_cleanup fi ## Check validity of increment value. test ${FSIncreaseBy} -eq 0 1>/dev/null 2>&1 if [[ $? -eq 2 ]]; then ## FSIncreaseBy parameter is not an integer. echo "ERROR: ${FSIncreaseBy} is not an integer." f_showhelp ErrorFlag=2 f_cleanup fi ## Get available space for the file system. FSAvailable="`df --block-size=g ${FSMap} | awk '{ print $4 }' | tail -n 1 | sed 's/G//'`" ## Get the current size of the File System. FSSize="`df --block-size=g ${FSMap} | awk '{ print $2 }' | tail -n 1 | sed 's/G//'`" ## Get the current size of the Logical Volume for the File System LVSize="`lvs --noheadings --nosuffix --units=g ${FSMap} | awk '{ print $4}' | sed 's/[.].*//'`" ## Calculate the new size of the FS in case we need it. let NewFSSize=${FSSize}+${FSIncreaseBy} if [[ ${FSAvailable} -lt ${FSThreshold} ]]; then echo "`date +%Y-%m-%d_%H:%M:%S` - Starting expansion of ${FSVol}" | tee -a ${LOGFILE} echo "`date +%Y-%m-%d_%H:%M:%S` --- LVSize=${LVSize}GB, FSSize=${FSSize}GB, FSAvail=${FSAvailable}GB, FSThreshold=${FSThreshold}GB, FSIncreaseBy=${FSIncreaseBy}GB" | tee -a ${LOGFILE} ## Run the auto-expansion function. f_auto-increment ReturnCode=$? case ${ReturnCode} in 0) f_sendmail "NOTICE: File System Expanded" "${FSVol} was expanded because it was nearing max capacity. Please review disk space usage and plan appropriately. LVSize=${LVSize}GB, FSSize=${FSSize}GB, FSAvailable=${FSAvailable}GB, FSThreshold=${FSThreshold}GB, FSIncreaseBy=${FSIncreaseBy}GB" ;; 50) echo "`date +%Y-%m-%d_%H:%M:%S` - SEVERE: No room to expand ${FSVol}" | tee -a ${LOGFILE} ErrorFlag=16 f_sendmail "SEVERE: No room to expand ${FSVol}" "There is not enough room in the Logical Volume to expand the ${FSVol} File System. Immediate action is required. Make sure there is free space in the Volume Group 'LVG' and then expand the Logical Volume...then expand the File System.\n\nLVSize=${LVSize}GB, FSSize=${FSSize}GB, FSAvailable=${FSAvailable}GB, FSThreshold=${FSThreshold}GB, FSIncreaseBy=${FSIncreaseBy}GB.\n\nType 'vgs' to see if there is any free space in the Volume Group which can be given to the Logical Volume.\n\nType 'lvs' to see the current sizes of the LVs.\n\nType 'lvdisplay' to see a list of Logical Volumes so you can get the LV Name which is used in the lvextend and resize2fs commands.\n\nType 'lvextend -L+1G /dev/LVG/opt' if you want to extend the opt Logical Volume by 1 gigabyte (assuming there is 1GB available in the Volume Group).\n\nType 'df --block-size=g' to see a list of file systems and their associated size and available space.\n\nType 'resize2fs /dev/LVG/opt ${NewFSSize}G' to set the size of opt to ${NewFSSize} gigabytes. Make sure you set the size to the desired end-result which should be LARGER than the current FS size so you do not lose data." ;; *) echo "`date +%Y-%m-%d_%H:%M:%S` - ERROR: Expansion failure for ${FSVol}" | tee -a ${LOGFILE} ErrorFlag=8 f_sendmail "ERROR: File System Expansion Failed" "${FSVol} Expansion failed with return code of ${ReturnCode}. LVSize=${LVSize}GB, FSSize=${FSSize}GB, FSAvailable=${FSAvailable}GB, FSThreshold=${FSThreshold}GB, FSIncreaseBy=${FSIncreaseBy}GB" ;; esac echo "`date +%Y-%m-%d_%H:%M:%S` - Finished expansion of ${FSVol}" | tee -a ${LOGFILE} else echo "`date +%Y-%m-%d_%H:%M:%S` - ${FSVol} ${FSAvailable}G>${FSThreshold}G No action required." | tee -a ${LOGFILE} fi ## Perform cleanup routine. f_cleanup
/var/scripts/common/standard.conf
Here is the typical output when it does not have to increase the FS:Code:## Global Variables ## TEMPDIR="/var/temp" MYDOMAIN="example.com" ADMINEMAIL="admin@${MYDOMAIN}" REPORTEMAIL="lhammonds@${MYDOMAIN}" HOSTNAME=$(hostname -s) SCRIPTNAME=$0 MAILFILE="${TEMPDIR}/mailfile.$$" ## Global Functions ## function f_sendmail() { ## Purpose: Send email message. ## Parameter #1 = Subject ## Parameter #2 = Body echo "From: ${ADMINEMAIL}" > ${MAILFILE} echo "To: ${REPORTEMAIL}" >> ${MAILFILE} echo "Subject: ${1}" >> ${MAILFILE} echo "" >> ${MAILFILE} echo -e ${2} >> ${MAILFILE} echo "" >> ${MAILFILE} echo -e "\n\nServer: ${HOSTNAME}\nProgram: ${SCRIPTNAME}\nLog: ${LOGFILE}" >> ${MAILFILE} sendmail -t < ${MAILFILE} rm ${MAILFILE} }
/var/temp/check-storage.log
Here is a sample of what the log will look like when it perform increases...notice how I had to increase the threshold in order for it to fire off the increase (but it still just issued a single gigabyte increase):Code:2012-05-01_01:00:00 - /dev/LVG/opt 44G>2G No action required. 2012-05-01_02:00:00 - /dev/LVG/bak 91G>2G No action required. 2012-05-01_03:00:00 - /dev/LVG/temp 93G>2G No action required. 2012-05-02_01:00:00 - /dev/LVG/opt 44G>2G No action required. 2012-05-02_02:00:00 - /dev/LVG/bak 91G>2G No action required. 2012-05-02_03:00:00 - /dev/LVG/temp 93G>2G No action required. 2012-05-03_01:00:00 - /dev/LVG/opt 44G>2G No action required. 2012-05-03_02:00:00 - /dev/LVG/bak 91G>2G No action required. 2012-05-03_03:00:00 - /dev/LVG/temp 93G>2G No action required.
/var/temp/check-storage.log
EDIT #1: The log file output is not all that great in 1.0. I spruced it up with more info in version 1.1. Have fun!Code:2012-05-02_01:00:00 - Starting expansion of /dev/LVG/opt 2012-05-02_01:00:00 --- LVSize=75GB, FSSize=50GB, FSAvail=44GB, FSThreshold=50GB, IncreaseBy=1GB 2012-05-02_01:00:00 --- resize2fs /dev/LVG/opt 51, ReturnCode=0 2012-05-02_01:00:00 - Finished expansion of /dev/LVG/opt 2012-05-02_02:00:00 - Starting expansion of /dev/LVG/bak 2012-05-02_02:00:00 --- LVSize=125GB, FSSize=99GB, FSAvail=91GB, FSThreshold=100GB, IncreaseBy=1GB 2012-05-02_02:00:00 --- resize2fs /dev/LVG/bak 100, ReturnCode=0 2012-05-02_02:00:00 - Finished expansion of /dev/LVG/bak 2012-05-02_03:00:00 - Starting expansion of /dev/LVG/temp 2012-05-02_03:00:00 --- LVSize=125GB, FSSize=99GB, FSAvail=93GB, FSThreshold=100GB, IncreaseBy=1GB 2012-05-02_03:00:00 --- resize2fs /dev/LVG/temp 100, ReturnCode=0 2012-05-02_03:00:00 - Finished expansion of /dev/LVG/temp
EDIT #2: Updated to version 1.2, improved email notifications and other minor changes.
Here is a sample email for the situation where there is not enough room available in the LV to extend the FS:
Code:From: admin@example.com To: dirkdiggler@example.com Sent: Wednesday, May 2, 2012 2:47:17 PM Subject: SEVERE: No room to expand /dev/LVG/opt There is not enough room in the Logical Volume to expand the /dev/LVG/opt File System. Immediate action is required. Make sure there is free space in the Volume Group 'LVG' and then expand the Logical Volume...then expand the File System. LVSize=75GB, FSSize=50GB, FSAvailable=44GB, FSThreshold=55GB, FSIncreaseBy=55GB. Type 'vgs' to see if there is any free space in the Volume Group which can be given to the Logical Volume. Type 'lvs' to see the current sizes of the LVs. Type 'lvdisplay' to see a list of Logical Volumes so you can get the LV Name which is used in the lvextend and resize2fs commands. Type 'lvextend -L+1G /dev/LVG/opt' if you want to extend the opt Logical Volume by 1 gigabyte (assuming there is 1GB available in the Volume Group). Type 'df --block-size=g' to see a list of file systems and their associated size and available space. Type 'resize2fs /dev/LVG/opt 105G' to set the size of opt to 105 gigabytes. Make sure you set the size to the desired end-result which should be LARGER than the current FS size so you do not lose data. Server: mail Program: ./check-storage.sh Log: /var/temp/check-storage.log
Adv Reply
Bookmarks