Check Storage Space
In favor of manage by exception, I wrote a script that can be scheduled to run daily to check the file systems to see if they are getting close to filling up and will automatically expand them a little bit and give you an email notice. Everything is done at the megabyte level. If you do not want the script to perform the increase, simply add a pound sign in front of the resize2fs command on line 64 to comment it out. Might also want to modify the log and email messages so it does not look like it actually "performed" the resize but instead is telling YOU how to perform the resize.
Here are the lines I added to my crontab schedule which will check each file system I expect will grow on a daily basis @ 1am, 2am and 3am. I have them checking to see if we have less than 50 MB available and if so, it will try to increase by 50 MB.
crontab
Code:
0 1 * * * /var/scripts/prod/check-storage.sh var 50 50 > /dev/null 2>&1
0 2 * * * /var/scripts/prod/check-storage.sh bak 50 50 > /dev/null 2>&1
0 3 * * * /var/scripts/prod/check-storage.sh tmp 50 50 > /dev/null 2>&1
/var/scripts/prod/check-storage.sh
Code:
#!/bin/bash
#############################################
## Name : check-storage.sh
## Version : 1.0
## Date : 2012-05-11
## Author : LHammonds
## Purpose : Check available space for a file system and expand if necessary.
## Compatibility : Verified on Ubuntu Server 12.04 LTS
## Requirements : None
## Run Frequency : Recommend once per day for each FS to monitor.
## Parameters :
## 1 = (Required) File System name (e.g. var)
## 2 = (Required) File System Threshold in MB (e.g. 50)
## 3 = (Required) Amount to increase File System in MB (e.g. 50)
## Exit Codes :
## 0 = Success (either nothing was done or FS expanded without error)
## 1 = ERROR: Missing or incorrect parameter(s)
## 2 = ERROR: Invalid parameter value(s)
## 4 = ERROR: Lock file detected
## 8 = ERROR: Resize2fs error
## 16 = SEVERE: No room to expand
## 32 = ERROR: Script not run by root user
################ CHANGE LOG #################
## DATE WHO WHAT WAS CHANGED
## ---------- --- ----------------------------
## 2012-05-11 LTH Created script.
#############################################
## Import standard variables and functions. ##
source /var/scripts/common/standard.conf
## Define local variables.
LOGFILE="${LOGDIR}/check-storage.log"
LOCKFILE="${TEMPDIR}/check-storage.lock"
ErrorFlag=0
ReturnCode=0
#######################################
## FUNCTIONS ##
#######################################
function f_cleanup()
{
if [ -f ${LOCKFILE} ];then
## Remove lock file so other check space jobs can run.
rm ${LOCKFILE} 1>/dev/null 2>&1
fi
exit ${ErrorFlag}
}
function f_showhelp()
{
echo -e "\nUsage : ${SCRIPTNAME} FileSystemName ThresholdSizeInMB AmountToIncreaseByInMB\n"
echo -e "\nExample: ${SCRIPTNAME} var 50 50\n"
}
function f_auto-increment()
{
let RoomInLV=${LVSize}-${FSSize}
if [[ ${RoomInLV} -gt ${FSIncreaseBy} ]]; then
## There is room in the LV to increase space to the FS.
resize2fs ${FSVol} ${NewFSSize}M
ReturnCode=$?
echo "`date +%Y-%m-%d_%H:%M:%S` --- resize2fs ${FSVol} ${NewFSSize}M, ReturnCode=${ReturnCode}" | tee -a ${LOGFILE}
if [[ ${ReturnCode} -ne 0 ]]; then
## There was an error in resize2fs.
return ${ReturnCode}
fi
else
## There is not enough room in the LV to increase space in the FS.
return 50
fi
return 0
}
#######################################
## MAIN PROGRAM ##
#######################################
if [ -f ${LOCKFILE} ]; then
# Lock file detected. Abort script.
echo "Check space script aborted"
echo "This script tried to run but detected the lock file: ${LOCKFILE}"
echo "Please check to make sure the file does not remain when check space is not actually running."
f_sendmail "ERROR: check storage script aborted" "This script tried to run but detected the lock file: ${LOCKFILE}\n\nPlease check to make sure the file does not remain when check space is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LOCKFILE}'"
ErrorFlag=4
f_cleanup
else
echo "`date +%Y-%m-%d_%H:%M:%S` ${SCRIPTNAME}" > ${LOCKFILE}
fi
## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
## FATAL ERROR DETECTED: Document problem and terminate script.
echo "ERROR: Root user required to run this script."
echo ""
ErrorFlag=32
f_cleanup
fi
## Check existance of required command-line parameters.
case "$1" in
"")
f_showhelp
ErrorFlag=1
f_cleanup
;;
--help|-h|-?)
f_showhelp
ErrorFlag=1
f_cleanup
;;
*)
FSName=$1
;;
esac
case "$2" in
"")
f_showhelp
ErrorFlag=1
f_cleanup
;;
--help|-h|-?)
f_showhelp
ErrorFlag=1
f_cleanup
;;
*)
FSThreshold=$2
;;
esac
case "$3" in
"")
f_showhelp
ErrorFlag=1
f_cleanup
;;
--help|-h|-?)
f_showhelp
ErrorFlag=1
f_cleanup
;;
*)
FSIncreaseBy=$3
;;
esac
## Check validity of File System name.
case "${FSName}" in
"var")
FSVol="/dev/LVG/var"
FSMap="/dev/mapper/LVG-var"
;;
"bak")
FSVol="/dev/LVG/bak"
FSMap="/dev/mapper/LVG-bak"
;;
"tmp")
FSVol="/dev/LVG/tmp"
FSMap="/dev/mapper/LVG-tmp"
;;
*)
echo "ERROR: ${FSName} does not match a known file system defined in this script."
f_showhelp
ErrorFlag=2
f_cleanup
;;
esac
## Check validity of threshold value.
test ${FSThreshold} -eq 0 1>/dev/null 2>&1
if [[ $? -eq 2 ]]; then
## Threshold parameter is not an integer.
echo "ERROR: ${FSThreshold} is not an integer."
f_showhelp
ErrorFlag=2
f_cleanup
fi
## Check validity of increment value.
test ${FSIncreaseBy} -eq 0 1>/dev/null 2>&1
if [[ $? -eq 2 ]]; then
## FSIncreaseBy parameter is not an integer.
echo "ERROR: ${FSIncreaseBy} is not an integer."
f_showhelp
ErrorFlag=2
f_cleanup
fi
## Get available space for the file system.
FSAvailable="`df --block-size=m ${FSMap} | awk '{ print $4 }' | tail -n 1 | sed 's/M//'`"
## Get the current size of the File System.
FSSize="`df --block-size=m ${FSMap} | awk '{ print $2 }' | tail -n 1 | sed 's/M//'`"
## Get the current size of the Logical Volume for the File System
LVSize="`lvs --noheadings --nosuffix --units=m ${FSMap} | awk '{ print $4}' | sed 's/[.].*//'`"
## Calculate the new size of the FS in case we need it.
let NewFSSize=${FSSize}+${FSIncreaseBy}
if [[ ${FSAvailable} -lt ${FSThreshold} ]]; then
echo "`date +%Y-%m-%d_%H:%M:%S` - Starting expansion of ${FSVol}" | tee -a ${LOGFILE}
echo "`date +%Y-%m-%d_%H:%M:%S` --- LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvail=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB" | tee -a ${LOGFILE}
## Run the auto-expansion function.
f_auto-increment
ReturnCode=$?
case ${ReturnCode} in
0)
f_sendmail "NOTICE: File System Expanded" "${FSVol} was expanded because it was nearing max capacity. Please review disk space usage and plan appropriately. LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB"
;;
50)
echo "`date +%Y-%m-%d_%H:%M:%S` - SEVERE: No room to expand ${FSVol}" | tee -a ${LOGFILE}
ErrorFlag=16
f_sendmail "SEVERE: No room to expand ${FSVol}" "There is not enough room in the Logical Volume to expand the ${FSVol} File System. Immediate action is required. Make sure there is free space in the Volume Group 'LVG' and then expand the Logical Volume...then expand the File System.\n\nLVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB.\n\nType 'vgs' to see if there is any free space in the Volume Group which can be given to the Logical Volume.\n\nType 'lvs' to see the current sizes of the LVs.\n\nType 'lvdisplay' to see a list of Logical Volumes so you can get the LV Name which is used in the lvextend and resize2fs commands.\n\nType 'lvextend -L+50M /dev/LVG/var' if you want to extend the var Logical Volume by 50 megabytes (assuming there is 50MB available in the Volume Group).\n\nType 'df --block-size=m' to see a list of file systems and their associated size and available space.\n\nType 'resize2fs /dev/LVG/var ${NewFSSize}M' to set the size of var to ${NewFSSize} megabytes. Make sure you set the size to the desired end-result which should be LARGER than the current FS size so you do not lose data."
;;
*)
echo "`date +%Y-%m-%d_%H:%M:%S` - ERROR: Expansion failure for ${FSVol}" | tee -a ${LOGFILE}
ErrorFlag=8
f_sendmail "ERROR: File System Expansion Failed" "${FSVol} Expansion failed with return code of ${ReturnCode}. LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB"
;;
esac
echo "`date +%Y-%m-%d_%H:%M:%S` - Finished expansion of ${FSVol}" | tee -a ${LOGFILE}
else
echo "`date +%Y-%m-%d_%H:%M:%S` - ${FSVol} ${FSAvailable}M>${FSThreshold}M No action required." | tee -a ${LOGFILE}
fi
## Perform cleanup routine.
f_cleanup
Here is the typical output when it does not have to increase the FS:
/var/log/check-storage.log
Code:
2012-05-01_01:00:00 - /dev/LVG/var 44M>5M No action required.
2012-05-01_02:00:00 - /dev/LVG/bak 91M>5M No action required.
2012-05-01_03:00:00 - /dev/LVG/tmp 93M>5M No action required.
2012-05-02_01:00:00 - /dev/LVG/var 44M>5M No action required.
2012-05-02_02:00:00 - /dev/LVG/bak 91M>5M No action required.
2012-05-02_03:00:00 - /dev/LVG/tmp 93M>5M No action required.
2012-05-03_01:00:00 - /dev/LVG/var 44M>5M No action required.
2012-05-03_02:00:00 - /dev/LVG/bak 91M>5M No action required.
2012-05-03_03:00:00 - /dev/LVG/tmp 93M>5M No action required.
Here is a sample of what the log will look like when it perform increases...notice how I had to increase the threshold in order for it to fire off the increase (but it still just issued a single megabyte increase):
/var/log/check-storage.log
Code:
2012-05-02_01:00:00 - Starting expansion of /dev/LVG/var
2012-05-02_01:00:00 --- LVSize=75MB, FSSize=50MB, FSAvail=44MB, FSThreshold=55MB, IncreaseBy=1MB
2012-05-02_01:00:00 --- resize2fs /dev/LVG/var 51, ReturnCode=0
2012-05-02_01:00:00 - Finished expansion of /dev/LVG/var
2012-05-02_02:00:00 - Starting expansion of /dev/LVG/bak
2012-05-02_02:00:00 --- LVSize=125MB, FSSize=99MB, FSAvail=91MB, FSThreshold=100MB, IncreaseBy=1MB
2012-05-02_02:00:00 --- resize2fs /dev/LVG/bak 100, ReturnCode=0
2012-05-02_02:00:00 - Finished expansion of /dev/LVG/bak
2012-05-02_03:00:00 - Starting expansion of /dev/LVG/tmp
2012-05-02_03:00:00 --- LVSize=125MB, FSSize=99MB, FSAvail=93MB, FSThreshold=100MB, IncreaseBy=1MB
2012-05-02_03:00:00 --- resize2fs /dev/LVG/temp 100, ReturnCode=0
2012-05-02_03:00:00 - Finished expansion of /dev/LVG/tmp
My Notes for Installing MySQL on Ubuntu Server 12.04 LTS
Adv Reply
Bookmarks