Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 25

Thread: How to install and configure an Ubuntu Server 18.04 LTS

  1. #11
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb Scripts - Check Storage

    Check Storage Space

    In favor of managing by exception, this script that can be scheduled to run daily to check the file systems to see if they are getting close to filling up and will automatically expand them a little bit and give you an email notice. Everything is done at the megabyte level. If you do not want the script to perform the increase, simply add a pound sign in front of the resize2fs command on line 62 to comment it out. Might also want to modify the log and email messages so it does not look like it actually "performed" the resize but instead is telling YOU how to perform the resize.

    Here are the lines added to the root crontab schedule which will check each file system.

    The script check the specified file system and see if the amount of space free is less than the threshold (e.g. 100 MB). If the file system has free space that is less than the threshold, it will attempt to add the specified amount (e.g. 50 MB).

    crontab
    Code:
    0 1 * * * /var/scripts/prod/check-storage.sh root 500 100 > /dev/null 2>&1
    15 1 * * * /var/scripts/prod/check-storage.sh home 100 50 > /dev/null 2>&1
    30 1 * * * /var/scripts/prod/check-storage.sh tmp 100 50 > /dev/null 2>&1
    45 1 * * * /var/scripts/prod/check-storage.sh usr 100 50 > /dev/null 2>&1
    0 2 * * * /var/scripts/prod/check-storage.sh var 100 50 > /dev/null 2>&1
    15 2 * * * /var/scripts/prod/check-storage.sh srv 100 50 > /dev/null 2>&1
    30 2 * * * /var/scripts/prod/check-storage.sh opt 100 50 > /dev/null 2>&1
    45 2 * * * /var/scripts/prod/check-storage.sh bak 100 50 > /dev/null 2>&1
    /var/scripts/prod/check-storage.sh
    Code:
    #!/bin/bash
    #############################################
    ## Name          : check-storage.sh
    ## Version       : 1.2
    ## Date          : 2017-03-17
    ## Author        : LHammonds
    ## Purpose       : Check available space for a file system and expand if necessary.
    ## Compatibility : Verified on Ubuntu Server 12.04 thru 18.04 LTS
    ## Requirements  : None
    ## Run Frequency : Recommend once per day for each FS to monitor.
    ## Parameters    :
    ##    1 = (Required) File System name (e.g. var)
    ##    2 = (Required) File System Threshold in MB (e.g. 50)
    ##    3 = (Required) Amount to increase File System in MB (e.g. 50)
    ## Exit Codes    :
    ##    0 = Success (either nothing was done or FS expanded without error)
    ##    1 = ERROR: Missing or incorrect parameter(s)
    ##    2 = ERROR: Invalid parameter value(s)
    ##    4 = ERROR: Lock file detected
    ##    8 = ERROR: Resize2fs error
    ##   16 = SEVERE: No room to expand
    ##   32 = ERROR: Script not run by root user
    ################ CHANGE LOG #################
    ## DATE       WHO WHAT WAS CHANGED
    ## ---------- --- ----------------------------
    ## 2012-05-11 LTH Created script.
    ## 2013-03-11 LTH Added company prefix to logs.
    ## 2017-03-17 LTH Adjusted variable casing.
    #############################################
    
    ## Import standard variables and functions. ##
    source /var/scripts/common/standard.conf
    
    ## Define local variables.
    LogFile="${LogDir}/${Company}-check-storage.log"
    LockFile="${TempDir}/${Company}-check-storage.lock"
    ErrorFlag=0
    ReturnCode=0
    
    #######################################
    ##            FUNCTIONS              ##
    #######################################
    
    function f_cleanup()
    {
      if [ -f ${LockFile} ];then
        ## Remove lock file so other check space jobs can run.
        rm ${LockFile} 1>/dev/null 2>&1
      fi
      exit ${ErrorFlag}
    }
    
    function f_showhelp()
    {
      echo -e "\nUsage : ${ScriptName} FileSystemName ThresholdSizeInMB AmountToIncreaseByInMB\n"
      echo -e "\nExample: ${ScriptName} var 50 50\n"
    }
    
    function f_auto-increment()
    {
      let RoomInLV=${LVSize}-${FSSize}
      if [[ ${RoomInLV} -gt ${FSIncreaseBy} ]]; then
        ## There is room in the LV to increase space to the FS.
        resize2fs ${FSVol} ${NewFSSize}M
        ReturnCode=$?
        echo "`date +%Y-%m-%d_%H:%M:%S` --- resize2fs ${FSVol} ${NewFSSize}M, ReturnCode=${ReturnCode}" | tee -a ${LogFile}
        if [[ ${ReturnCode} -ne 0 ]]; then
          ## There was an error in resize2fs.
          return ${ReturnCode}
        fi
      else
        ## There is not enough room in the LV to increase space in the FS.
        return 50
      fi
      return 0
    }
    
    #######################################
    ##           MAIN PROGRAM            ##
    #######################################
    
    if [ -f ${LockFile} ]; then
      # Lock file detected.  Abort script.
      echo "Check space script aborted"
      echo "This script tried to run but detected the lock file: ${LockFile}"
      echo "Please check to make sure the file does not remain when check space is not actually running."
      f_sendmail "ERROR: check storage script aborted" "This script tried to run but detected the lock file: ${LockFile}\n\nPlease check to make sure the file does not remain when check space is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LockFile}'"
      ErrorFlag=4
      f_cleanup
    else
      echo "`date +%Y-%m-%d_%H:%M:%S` ${ScriptName}" > ${LockFile}
    fi
    
    ## Requirement Check: Script must run as root user.
    if [ "$(id -u)" != "0" ]; then
      ## FATAL ERROR DETECTED: Document problem and terminate script.
      echo "ERROR: Root user required to run this script."
      echo ""
      ErrorFlag=32
      f_cleanup
    fi
    
    ## Check existence of required command-line parameters.
    case "$1" in
      "")
        f_showhelp
        ErrorFlag=1
        f_cleanup
        ;;
      --help|-h|-?)
        f_showhelp
        ErrorFlag=1
        f_cleanup
        ;;
      *)
        FSName=$1
        ;;
    esac
    case "$2" in
      "")
        f_showhelp
        ErrorFlag=1
        f_cleanup
        ;;
      --help|-h|-?)
        f_showhelp
        ErrorFlag=1
        f_cleanup
        ;;
      *)
        FSThreshold=$2
        ;;
    esac
    case "$3" in
      "")
        f_showhelp
        ErrorFlag=1
        f_cleanup
        ;;
      --help|-h|-?)
        f_showhelp
        ErrorFlag=1
        f_cleanup
        ;;
      *)
        FSIncreaseBy=$3
        ;;
    esac
    
    ## Check validity of File System name.
    case "${FSName}" in
      "root")
        FSVol="/dev/LVG/root"
        FSMap="/dev/mapper/LVG-root"
        ;;
      "home")
        FSVol="/dev/LVG/home"
        FSMap="/dev/mapper/LVG-home"
        ;;
      "tmp")
        FSVol="/dev/LVG/tmp"
        FSMap="/dev/mapper/LVG-tmp"
        ;;
      "opt")
        FSVol="/dev/LVG/opt"
        FSMap="/dev/mapper/LVG-opt"
        ;;
      "bak")
        FSVol="/dev/LVG/bak"
        FSMap="/dev/mapper/LVG-bak"
        ;;
      "usr")
        FSVol="/dev/LVG/usr"
        FSMap="/dev/mapper/LVG-usr"
        ;;
      "var")
        FSVol="/dev/LVG/var"
        FSMap="/dev/mapper/LVG-var"
        ;;
      "srv")
        FSVol="/dev/LVG/srv"
        FSMap="/dev/mapper/LVG-srv"
        ;;
      *)
        echo "ERROR: ${FSName} does not match a known file system defined in this script."
        f_showhelp
        ErrorFlag=2
        f_cleanup
        ;;
    esac
    
    ## Check validity of threshold value.
    test ${FSThreshold} -eq 0 1>/dev/null 2>&1
    if [[ $? -eq 2 ]]; then
      ## Threshold parameter is not an integer.
      echo "ERROR: ${FSThreshold} is not an integer."
      f_showhelp
      ErrorFlag=2
      f_cleanup
    fi
    
    ## Check validity of increment value.
    test ${FSIncreaseBy} -eq 0 1>/dev/null 2>&1
    if [[ $? -eq 2 ]]; then
      ## FSIncreaseBy parameter is not an integer.
      echo "ERROR: ${FSIncreaseBy} is not an integer."
      f_showhelp
      ErrorFlag=2
      f_cleanup
    fi
    
    ## Get available space for the file system.
    FSAvailable="`df --block-size=m ${FSMap} | awk '{ print $4 }' | tail -n 1 | sed 's/M//'`"
    
    ## Get the current size of the File System.
    FSSize="`df --block-size=m ${FSMap} | awk '{ print $2 }' | tail -n 1 | sed 's/M//'`"
    
    ## Get the current size of the Logical Volume for the File System
    LVSize="`lvs --noheadings --nosuffix --units=m ${FSMap} | awk '{ print $4}' | sed 's/[.].*//'`"
    
    ## Calculate the new size of the FS in case we need it.
    let NewFSSize=${FSSize}+${FSIncreaseBy}
    
    if [[ ${FSAvailable} -lt ${FSThreshold} ]]; then
      echo "`date +%Y-%m-%d_%H:%M:%S` - Starting expansion of ${FSVol}" | tee -a ${LogFile}
      echo "`date +%Y-%m-%d_%H:%M:%S` --- LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvail=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB" | tee -a ${LogFile}
      ## Run the auto-expansion function.
      f_auto-increment
      ReturnCode=$?
      case ${ReturnCode} in
      0)
        f_sendmail "NOTICE: File System Expanded" "${FSVol} was expanded because it was nearing max capacity.  Please review disk space usage and plan appropriately. LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB"
        ;;
      50)
        echo "`date +%Y-%m-%d_%H:%M:%S` - SEVERE: No room to expand ${FSVol}" | tee -a ${LogFile}
        ErrorFlag=16
        f_sendmail "SEVERE: No room to expand ${FSVol}" "There is not enough room in the Logical Volume to expand the ${FSVol} File System.  Immediate action is required.  Make sure there is free space in the Volume Group 'LVG' and then expand the Logical Volume...then expand the File System.\n\nLVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB.\n\nType 'vgs' to see if there is any free space in the Volume Group which can be given to the Logical Volume.\n\nType 'lvs' to see the current sizes of the LVs.\n\nType 'lvdisplay' to see a list of Logical Volumes so you can get the LV Name which is used in the lvextend and resize2fs commands.\n\nType 'lvextend -L+50M /dev/LVG/var' if you want to extend the var Logical Volume by 50 megabytes (assuming there is 50MB available in the Volume Group).\n\nType 'df --block-size=m' to see a list of file systems and their associated size and available space.\n\nType 'resize2fs /dev/LVG/var ${NewFSSize}M' to set the size of var to ${NewFSSize} megabytes. Make sure you set the size to the desired end-result which should be LARGER than the current FS size so you do not lose data."
        ;;
      *)
        echo "`date +%Y-%m-%d_%H:%M:%S` - ERROR: Expansion failure for ${FSVol}" | tee -a ${LogFile}
        ErrorFlag=8
        f_sendmail "ERROR: File System Expansion Failed" "${FSVol} Expansion failed with return code of ${ReturnCode}.  LVSize=${LVSize}MB, FSSize=${FSSize}MB, FSAvailable=${FSAvailable}MB, FSThreshold=${FSThreshold}MB, FSIncreaseBy=${FSIncreaseBy}MB"
        ;;
      esac
      echo "`date +%Y-%m-%d_%H:%M:%S` - Finished expansion of ${FSVol}" | tee -a ${LogFile}
    else
      echo "`date +%Y-%m-%d_%H:%M:%S` - ${FSVol} ${FSAvailable}M>${FSThreshold}M No action required." | tee -a ${LogFile}
    fi
    
    ## Perform cleanup routine.
    f_cleanup
    Here is the typical output when it does not have to increase the FS:

    /var/log/check-storage.log
    Code:
    2018-04-19_01:00:00 - /dev/LVG/root 1285M>500M No action required.
    2018-04-19_01:15:00 - /dev/LVG/home 453M>100M No action required.
    2018-04-19_01:30:00 - /dev/LVG/tmp 932M>100M No action required.
    2018-04-19_01:45:00 - /dev/LVG/usr 1985M>100M No action required.
    2018-04-19_02:00:00 - /dev/LVG/var 1646M>100M No action required.
    2018-04-19_02:15:00 - /dev/LVG/srv 939M>100M No action required.
    2018-04-19_02:30:00 - /dev/LVG/opt 924M>100M No action required.
    2018-04-19_02:45:00 - /dev/LVG/bak 2834M>100M No action required.
    2018-04-20_01:00:00 - /dev/LVG/root 1285M>500M No action required.
    2018-04-20_01:15:00 - /dev/LVG/home 453M>100M No action required.
    2018-04-20_01:30:00 - /dev/LVG/tmp 932M>100M No action required.
    2018-04-20_01:45:00 - /dev/LVG/usr 1985M>100M No action required.
    2018-04-20_02:00:00 - /dev/LVG/var 1646M>100M No action required.
    2018-04-20_02:15:00 - /dev/LVG/srv 939M>100M No action required.
    2018-04-20_02:30:00 - /dev/LVG/opt 924M>100M No action required.
    2018-04-20_02:45:00 - /dev/LVG/bak 2834M>100M No action required.
    Here is a sample of what the log will look like when it performs increases:

    /var/log/check-storage.log
    Code:
    2017-02-02_01:30:00 - Starting expansion of /dev/LVG/tmp
    2018-04-20_01:30:00 --- LVSize=2048MB, FSSize=1004MB, FSAvail=93MB, FSThreshold=100MB, IncreaseBy=50MB
    2018-04-20_01:30:00 --- resize2fs /dev/LVG/temp 1054, ReturnCode=0
    2018-04-20_01:30:00 - Finished expansion of /dev/LVG/tmp
    2018-04-20_02:00:00 - Starting expansion of /dev/LVG/var
    2018-04-20_02:00:00 --- LVSize=3072MB, FSSize=1901MB, FSAvail=95MB, FSThreshold=100MB, IncreaseBy=50MB
    2018-04-20_02:00:00 --- resize2fs /dev/LVG/var 1951, ReturnCode=0
    2018-04-20_02:00:00 - Finished expansion of /dev/LVG/var
    2018-04-20_02:45:00 - Starting expansion of /dev/LVG/bak
    2018-04-20_02:45:00 --- LVSize=4096MB, FSSize=1996MB, FSAvail=91MB, FSThreshold=100MB, IncreaseBy=50MB
    2018-04-20_02:45:00 --- resize2fs /dev/LVG/bak 2044, ReturnCode=0
    2018-04-20_02:45:00 - Finished expansion of /dev/LVG/bak

  2. #12
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb Scripts - Operator Menu

    Operator Menu

    To make my servers easier to administer within my group (who are not all *NIX guys), I made an operator script that can be used to manipulate the server.

    The main features I wanted to make easier for other members of the IT group are:

    1. Update the operating system (manually)
    2. Disk Status
    3. Memory Status
    4. Stop/Restart primary services
    5. Cleanly shutdown services and reboot server
    6. Cleanly shutdown services and power off server

    I accomplished this by creating a script and making a shortcut to it so the only thing they need to type on each server is "opm" which is short for "Operator Menu"

    Here is how to do something similar for your servers.

    Create the script file and set the permissions so that only the root user can run the script and make a shortcut for it:

    Code:
    touch /var/scripts/prod/opm.sh
    chmod 0700 /var/scripts/prod/opm.sh
    ln -s /var/scripts/prod/opm.sh /usr/sbin/opm
    Now open the script in your favorite editor and add the following:
    /var/scripts/prod/opm.sh
    Code:
    #!/bin/bash
    #############################################
    ## Name          : opm.sh
    ## Version       : 1.2
    ## Date          : 2018-04-19
    ## Author        : LHammonds
    ## Compatibility : Ubuntu Server 12.04 thru 18.04 LTS
    ## Requirements  : dialog (apt-get dialog) and root privileges
    ## Purpose       : Display menu to control the server
    ## Run Frequency : As needed
    ## Exit Codes    : None
    ## SymLink Cmd   : ln -s /var/scripts/prod/opm.sh /usr/sbin/opm
    ################ CHANGE LOG #################
    ## DATE       WHO  WHAT WAS CHANGED
    ## ---------- ---- ----------------------------
    ## 2013-01-07 LTH  Created script.
    ## 2017-03-17 LTH  Updated variable standards.
    ## 2018-04-19 LTH  Various minor changes.
    #############################################
    ## Store menu options selected by the user.
    TempDir="/tmp"
    ScriptDir="/var/scripts/prod"
    InputFile="${TempDir}/opm-input.$$"
    
    ## Storage file for displaying cal and date command output.
    OutputFile="${TempDir}/opm-output.$$"
    
    ## Get text editor or fall back to vi_editor.
    vi_editor=${EDITOR-vi}
    
    ## Requirement Check: Script must run as root user.
    if [ "$(id -u)" != "0" ]; then
      ## FATAL ERROR DETECTED: Document problem and terminate script.
      echo -e "\nERROR: Root user required to run this script.\n"
      echo -e "Type 'sudo su' to temporarily become root user.\n"
      exit
    fi
    
    ## Trap and delete temp files.
    trap "rm $OutputFile; rm $InputFile; exit" SIGHUP SIGINT SIGTERM
    
    function f_display_output(){
      ## Purpose - display output using msgbox
      ##  $1 -> set msgbox height
      ##  $2 -> set msgbox width
      ##  $3 -> set msgbox title
      local h=${1-10}     ## box height default 10
      local w=${2-41}     ## box width default 41
      local t=${3-Output} ## box title
      dialog --backtitle "Operator Menu for $(hostname -f)" --title "${t}" --clear --msgbox "$(<$OutputFile)" ${h} ${w}
    } ## f_display_output
    
    function f_showdate(){
      ## Purpose - display current system date & time
      echo "Today is $(date) @ $(hostname -f)." >$OutputFile
      f_display_output 6 60 "Date and Time"
    } ## f_showdate
    
    function f_checkdisk(){
      ## Purpose: Display disk status.
      clear
      echo -e "df --block-size=M\n"
      df --block-size=M
      echo ""
      read -p "Press [Enter] key to continue..."
    } ## f_checkdisk
    
    ## Loop the menu display.
    while true
    do
      ## Display main menu.
      dialog --clear  --no-cancel --backtitle "Operator Menu for $(hostname -f)" \
      --title "[ M A I N - M E N U ]" \
      --menu "You can use the UP/DOWN arrow keys, the first \n\
      letter of the choice as a hot key, or the \n\
      number keys 1-9 to choose an option.\n\
      Choose the TASK" 19 50 7 \
      Exit "Exit menu" \
      OSUpdate "Update Operating System" \
      CheckDisk "Check Disk Status" \
      MEMCheck "Look at running processes" \
      ServiceRestart "Stop/Start Main Services" \
      RebootServer "Cleanly reboot server" \
      PoweroffServer "Cleanly Power-off server" \
      Date/time "Displays date and time" 2>"${InputFile}"
    
      menuitem=$(<"${InputFile}")
    
      ## Make decision.
      case $menuitem in
        OSUpdate) ${ScriptDir}/apt-upgrade.sh;;
        CheckDisk) f_checkdisk;;
        MEMCheck) htop;;
        ServiceRestart) ${ScriptDir}/servicerestart.sh;;
        RebootServer) ${ScriptDir}/reboot.sh;;
        PoweroffServer) ${ScriptDir}/shutdown.sh;;
        Date/time) f_showdate;;
        Exit) clear; echo "Clean menu exit."; break;;
      esac
    done
    
    ## Delete temp files.
    [ -f $OutputFile ] && rm $OutputFile
    [ -f $InputFile ] && rm $InputFile



    Customizing Operator Menu

    Let's say you have "mytop" installed on your MySQL/MariaDB server and you want that added to your menu. This is how you would do that.

    Edit the opm.sh file and find this section:

    Code:
      OSUpdate "Update Operating System" \
      CheckDisk "Check Disk Status" \
      MEMCheck "Look at running processes" \
      ServiceRestart "Stop/Start Main Services" \
      Reboot "Cleanly reboot server" \
      Date/time "Displays date and time" 2>"${Input}"
    Now add the following line so it shows up in your menu:

    Code:
      MySQLCheck "Look at MySQL processes" \
    End result looking like this:

    Code:
      OSUpdate "Update Operating System" \
      CheckDisk "Check Disk Status" \
      MEMCheck "Look at running processes" \
      MySQLCheck "Look at MySQL processes" \
      ServiceRestart "Stop/Start Main Services" \
      Reboot "Cleanly reboot server" \
      Date/time "Displays date and time" 2>"${Input}"
    Now that the menu entry is there, we need the menu item to actually do something so find this section below:

    Code:
        OSUpdate) ${ScriptDir}/abc-apt-upgrade.sh;;
        CheckDisk) f_checkdisk;;
        MEMCheck) htop;;
        Reboot) ${ScriptDir}/reboot.sh;;
        ServiceRestart) ${ScriptDir}/servicerestart.sh;;
        Date/time) f_showdate;;
        Exit) clear; echo "Clean menu exit."; break;;
    Add the following line:

    Code:
        MySQLCheck) mytop;;
    The end result looking like this:

    Code:
        OSUpdate) ${ScriptDir}/abc-apt-upgrade.sh;;
        CheckDisk) f_checkdisk;;
        MEMCheck) htop;;
        MySQLCheck) mytop;;
        Reboot) ${ScriptDir}/reboot.sh;;
        ServiceRestart) ${ScriptDir}/servicerestart.sh;;
        Date/time) f_showdate;;
        Exit) clear; echo "Clean menu exit."; break;;
    Save and close the script.

    Now when you run opm, you will see the MySQL/MariaDB entry which will run mytop when you select it.

    With this menu example, I have given you 3 example types of how you can do something. You can call the program directly, like mytop or htop, or you can call a function to process several lines of script like f_checkdisk or f_showdate or you can call an external script like reboot.sh or servicerestart.sh which can further isolate individual server needs in other scripts.

  3. #13
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb Security - Firewall

    Firewall

    Linux uses IPTables as its firewall but learning the ins and outs of the syntax can be daunting. Luckily for us, Ubuntu has a front-end for it called Uncomplicated Firewall (UFW for short).

    If you are not using IPv6 (most networks are still just using IPv4), you can comment out the IPv6 setting so you don't get doubles of each rule (one for IPv4 and one for IPv6).

    To disable IPv6 rules:
    Code:
    vi /etc/default/ufw
    Find:
    Code:
    IPV6=yes
    Change to:
    Code:
    #IPV6=yes
    Here is a script I use to configure the firewall rules and turn it on...which will remain on even during a reboot.

    You can copy this script and use it for your own purposes and tweak it for your environment. Enough commands are used/documented in it that you should be able to modify it to fit your particular server. For example, if running a Minecraft server, you could add a command to allow TCP port 25565. If you are running a web server, you could remove the comment from the TCP ports 80, 8080 and 443.

    I NEVER simply copy this script and run it. Each and every server requires a custom variation of this script tailored for it. The application section has examples for commonly used services such as web, database, etc. Feel free to uncomment lines you can use and modify to suit your needs.

    /var/scripts/prod/en-firewall.sh
    Code:
    #!/bin/bash
    #############################################
    ## Name          : enable-firewall.sh
    ## Version       : 1.1
    ## Date          : 2017-04-13
    ## Author        : LHammonds
    ## Compatibility : Ubuntu Server 14.04 thru 18.04 LTS
    ## Requirements  : Run as root
    ## Purpose       : Restore and enable firewall.
    ## Run Frequency : As needed
    ## Exit Codes    : None
    ################ CHANGE LOG #################
    ## DATE       WHO  WHAT WAS CHANGED
    ## ---------- ---- ----------------------------
    ## 2015-08-28 LTH  Created script.
    ## 2017-04-13 LTH  Added comments in rules.
    #############################################
    
    ## Requirement Check: Script must run as root user.
    if [ "$(id -u)" != "0" ]; then
      ## FATAL ERROR DETECTED: Document problem and terminate script.
      echo -e "\nERROR: Root user required to run this script.\n"
      echo -e "Type 'sudo su' to temporarily become root user.\n"
      exit
    fi
    
    clear
    echo ""
    echo "Resetting Firewall to factory default"
    echo y | ufw reset 1>/dev/null 2>&1
    ufw default deny incoming 1>/dev/null 2>&1
    ufw default allow outgoing 1>/dev/null 2>&1
    echo "Allowing SSH from only LAN connections"
    ufw allow from 192.168.107.0/24 to any port 22 comment 'SSH via LAN' 1>/dev/null 2>&1
    ufw allow from 192.168.108.0/24 to any port 22 comment 'SSH via LAN' 1>/dev/null 2>&1
    echo "Allowing Samba file sharing connections"
    #ufw allow proto tcp to any port 135,139,445 comment 'Samba Share' 1>/dev/null 2>&1
    #ufw allow proto udp to any port 137,138 comment 'Samba Share' 1>/dev/null 2>&1
    echo "Allowing Nagios connections"
    #ufw allow from 192.168.107.21 to any port 12489 comment 'Nagios' 1>/dev/null 2>&1
    #ufw allow from 192.168.107.21 proto tcp to any port 5666 comment 'Nagios' 1>/dev/null 2>&1
    echo "Adding Application-specific rules"
    echo "Adding MySQL/MariaDB rules"
    #ufw allow from 192.168.107.0/24 proto tcp to any port 3306 comment 'MariaDB via LAN' 1>/dev/null 2>&1
    #ufw allow from 192.168.108.0/24 proto tcp to any port 3306 comment 'MariaDB via LAN' 1>/dev/null 2>&1
    echo "Adding FTP/FTPS rules"
    #ufw allow proto tcp to any port 990 comment 'FTPS' 1>/dev/null 2>&1
    #ufw allow proto tcp to any port 21 comment 'FTP' 1>/dev/null 2>&1
    #ufw allow proto tcp to any port 2000:2020 comment 'FTP Passive' 1>/dev/null 2>&1
    echo "Adding Web Server rules"
    #ufw allow proto tcp to any port 80 comment 'Web Service' 1>/dev/null 2>&1
    #ufw allow proto tcp to any port 8080 comment 'Web Service' 1>/dev/null 2>&1
    #ufw allow proto tcp to any port 443 comment 'Web Service' 1>/dev/null 2>&1
    echo "Enabling firewall"
    echo y | ufw enable 1>/dev/null 2>&1
    echo "Firewall enabled and all rules have been configured."

  4. #14
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb Security - Fail2Ban

    Fail2Ban

    Fail2Ban is an intrusion prevention system that can be used to protect servers from different kinds of attacks.

    Fail2ban scans log files for various services, such as SSH, FTP, SMTP, Apache and block the IP address that makes too many password failures.

    Install Fail2Ban
    Code:
    apt-get install fail2ban
    Configure Fail2Ban
    The default configuration file should not be altered since upgrades tend to overwrite this file with new settings.

    Instead, use a custom configuration file that will simply override or append various settings to the default.

    Fail2Ban reads .conf files first and then .local files last which can override any settings initially set in .conf files.

    First, look at the default file:
    Code:
    vi /etc/fail2ban/jail.conf
    Notice the following partial list of default values. These are based on the OS and version at the time of this writing:
    Code:
    [DEFAULT]
    ignoreip = 127.0.0.1/8
    bantime  = 600
    findtime  = 600
    maxretry = 5
    filter = %(__name__)s
    destemail = root@localhost
    sender = root@localhost
    mta = sendmail
    action = %(action_)s
    [sshd]
    port    = ssh
    logpath = %(sshd_log)s
    Now create a local jail file that will be used to override any settings in the default that we want changed or added.

    For this example, we will ignore all failed login attempts from our local server subnet (192.168.107.xxx) and a specific admin workstation (192.168.1.69)

    Code:
    vi /etc/fail2ban/jail.local
    Let's make some changes that fit our server as it stands right now:

    Code:
    [DEFAULT]
    ## 127.0.0.1/8 = ignore login failures on the local machine
    ## 192.168.107.0/24 = ignore login failures on an entire subnet
    ## 192.168.1.69 = ignore login failures for this specific IP
    ignoreip = 127.0.0.1/8 192.168.107.0/24 192.168.1.69
    
    ## "bantime" is the number of seconds that a host is banned.
    ##  300 =  5 minutes
    ##  600 = 10 minutes
    ##  900 = 15 minutes
    ## 1800 = 30 minutes
    ## 3600 = 60 minutes
    bantime = 1800
    ## "findtime" is the length of time between login attempts before a ban is set.
    findtime = 600
    ## "maxretry" is how many attempts can be made to access the server from a single IP before a ban is imposed.
    maxretry = 7
    
    ## "destemail" is the email address where you would like to receive the emails.
    destemail = webmaster@mydomain.com
    ## "sender" is the FROM: address when it arrives in your mailbox.
    sender = myserver@mydomain.com
    ## Use the lightweight sendemail instead of sendmail
    mta = sendemail
    ## Email notify with whois report and relevant log lines when a ban occurs
    action = %(action_mwl)s
    We are using sendemail for our scripts so we will also use it for fail2ban. We will need to make a configuration file that tells fail2ban how to make use of it though.

    Code:
    vi /etc/fail2ban/action.d/sendemail-whois-lines.conf
    Code:
    [Definition]
    actionstart =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: started" -m "The jail <name> has been started successfully.\n\nFail2Ban"
    actionstop =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: stopped" -m "The jail <name> has been stopped.\n\nFail2Ban"
    actioncheck =
    actionban =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: banned <ip>" -m "The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>.\n\nHere is more information about <ip>:\n `/usr/bin/whois <ip>`\n\n Lines containing IP:<ip> in <logpath>\n`/bin/grep '\<<ip>\>' <logpath>`\n\n\n\nFail2Ban"
    actionunban =
    
    [Init]
    ## Amended to be the same as the SMTP user
    sender = fail2ban@mydomain.com
    ## SMTP password for user
    #password = XXXXXXX
    ## SMTP server - use port 587 for Google rather than 25 (times out too often) or 465 (crashes sendemail)
    #smtp = smtp.googlemail.com:587
    smtp = srv-mail
    
    ## Name for this server - handy when there are lots of servers sending emails to the destemail
    servername = srv-ubuntu
    Useful commands

    Here are a few of the most-used commands. Reference: Fail2Ban commands.
    Stop, start or restart the fail2ban service:
    Code:
    service fail2ban stop
    service fail2ban start
    service fail2ban restart
    This command will show which services are being watched (sshd in this example):
    Code:
    fail2ban-client status
    Example output:
    Code:
    Status
    |- Number of jail:      1
    `- Jail list:   sshd
    This command gets the status on the sshd service being watched:
    Code:
    fail2ban-client status sshd
    Example output:
    Code:
    Status for the jail: sshd
    |- Filter
    |  |- Currently failed: 0
    |  |- Total failed:     0
    |  `- File list:        /var/log/auth.log
    `- Actions
       |- Currently banned: 0
       |- Total banned:     0
       `- Banned IP list:
    Firewall Rules

    Installing Fail2Ban will create the following firewall rules which can be seen with the following command:

    Code:
    iptables -S
    Code:
    -N f2b-sshd
    -A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
    -A f2b-sshd -j RETURN
    If you have already enabled Uncomplicated Firewall (UFW), you will see several ufw-prefixed rules as well.

    Test Fail2Ban

    To manually ban an IP:
    Code:
    fail2ban-client set sshd banip 192.168.1.69
    To manually unban an IP:
    Code:
    fail2ban-client set sshd unbanip 192.168.1.69
    On the server, run the following command to watch the Fail2Ban log file:
    Code:
    tail -f /var/log/fail2ban.log
    On a different machine (that is not on your ignoreip list), try to ssh into your server with the incorrect ID or password multiple times until you have reached the "maxretry" attempts and you get locked out...thus causing an entry in the log.

    References

    Fail2Ban Manual

  5. #15
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb Security - SSH Public and Private Keys

    SSH Public and Private Keys

    If you have more than one Ubuntu server, you probably will want to setup a trusted SSH authentication between the two servers so you can securely and automatically transfer files between them with commands such as scp or sftp.

    NOTE: You probably have the root account locked (by default) so you cannot login with root directly. In order to initially copy the public key, you will need to enable the root account temporarily on all servers.

    Temporarily enable login with the root account on each server:

    1. Connect to a server using PuTTY.
    2. At the login prompt, login with your administrator account (administrator / myadminpass)
    3. At the $ prompt, temporarily grant yourself super user privileges by typing sudo su {ENTER} and then provide the administrator password (myadminpass).
    4. Type the following and set the password to the same as your administrator account (myadminpass):
      Code:
      passwd root
    5. Edit /etc/ssh/sshd_config, and comment out the following from:
      Code:
      PermitRootLogin prohibit-password
      to:
      Code:
      #PermitRootLogin prohibit-password
    6. Just below that line, add the following and then save/exit:
      Code:
      PermitRootLogin yes
    7. Restart SSH:
      Code:
      service ssh restart
    8. Do the same for each of your servers



    Generate the private/public key on each server:

    1. Generate a key by typing the following and accept the default values on everything (even the blank passphrase):
      Code:
      mkdir ~/.ssh
      chmod 700 ~/.ssh
      ssh-keygen -t rsa -b 4096
    2. Do the same thing on all your other servers.



    Transfer the public key to your trusted servers:

    1. Type the following command for each server you have (use your own server name or IP instead of my example names):
      Code:
      ssh-copy-id root@srv-mysql
      ssh-copy-id root@srv-wiki
      ssh-copy-id root@srv-ftp
      ssh-copy-id root@srv-nagios
    2. Do the same thing on all your other servers.



    Lock the root account on each server:

    1. Type the following:
      Code:
      passwd -l root
    2. Do the same for each of your servers



    Set root login to SSH only on each server:

    1. Edit /etc/ssh/sshd_config, and uncomment the following from:
      Code:
      #PermitRootLogin prohibit-password
      to:
      Code:
      PermitRootLogin prohibit-password
    2. Just below that line, delete the following and then save/exit:
      Code:
      PermitRootLogin yes
    3. Restart SSH:
      Code:
      service ssh restart
    4. Do the same for each of your servers



    If you have other accounts you want to use across the servers (such as a low-rights account), you can do the same thing as the above except you probably won't need to do the unlock/lock steps for the account if you can already login with that account.

    Now you should be able to securely copy files back and forth between servers without having to type a password.

    Example:
    Code:
    scp /etc/hosts root@srv-mysql:/tmp/hosts.txt

  6. #16
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb Security - Messages

    SUDO Warning Message


    1. Enable the default warning message whenever "sudo" is used. Edit sudoers file by typing:
      Code:
      sudo EDITOR=vim visudo
      Add the following line near the other defaults (instead of "always" you can use "once" too):
      Code:
      Defaults lecture=always
    2. To make sure the syntax of the edited file is not incorrect, check it with the following command and look for "parsed: OK"
      NOTE: If you mess up the syntax in this file, it can cause you problems using the "sudo" command.
      Code:
      visudo -c
    3. To customize the message, you can create a custom message file. Edit the lecture file by typing:
      Code:
      touch /etc/lecture
      chmod 644 /etc/lecture
      vi /etc/lecture
      Add the following text (which is default) or whatever you want:
      Code:
      We trust you have received the usual lecture from the local System
      Administrator. It usually boils down to these three things:
      
          #1) Respect the privacy of others.
          #2) Think before you type.
          #3) With great power comes great responsibility.
    4. Now point to the newly-created file in the sudoers config file.
      Code:
      sudo EDITOR=vim visudo
      Code:
      Defaults lecture_file=/etc/lecture



    SSH Login Message

    1. To enable login banners that are displayed before entering a password, edit the following:
      Code:
      vi /etc/issue.net
    2. Add your message however you like to word it and save/close the file:
      Code:
      *********************************************************************
        ALERT! You are entering into a secured area!
        Your IP, Login Time, Username have been recorded.
        This service is restricted to authorized users only.
        All activities on this system are logged.
        Unauthorized access will be reported to the law enforcement agencies.
      *********************************************************************
    3. Enable banners to be displayed by editing the SSH config file:
      Code:
      vi /etc/ssh/sshd_config
      Code:
      Banner /etc/issue.net
    4. Restart the SSH service:
      Code:
      service sshd restart



    Reboot the server by typing reboot

    Custom Login Logo

    If you want an Ubuntu Logo and additional info added to your login, visit this thread.

    Last edited by LHammonds; May 3rd, 2018 at 04:43 PM. Reason: Added Custom Login Logo section

  7. #17
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb File Sharing via Samba

    Configure Ubuntu for File Sharing

    This file sharing section is optional but can be handy if you need to swap files between the Linux server and a Windows machine.

    This documentation will utilize this share for passing pre-configured files (configs, scripts, etc.) to make it faster/easier during installation.


    1. Start the Ubuntu server and connect using PuTTY.
    2. At the login prompt, login with your administrator account (administrator / myadminpass)
    3. At the $ prompt, temporarily grant yourself super user privileges by typing sudo su {ENTER} and then provide the administrator password (myadminpass).
    4. Install Samba by typing the following:
      Code:
      apt -y install samba cifs-utils
      NOTE: To share a folder with Windows, you just need the samba package, to connect to a Windows share, you need both samba and cifs-utils
    5. Type the following commands:
      Code:
      cp /etc/samba/smb.conf /etc/samba/smb.conf.bak
      mkdir -p /srv/samba/share
      chown nobody:nogroup /srv/samba/share/
      chmod 0777 /srv/samba
      chmod 0777 /srv/samba/share
    6. Edit the configuration file:
      Code:
      vi /etc/samba/smb.conf
    7. Change:
      Code:
      workgroup = WORKGROUP
      to:
      Code:
      workgroup = work
      (you are using the domain alias)
    8. Add the following section to the end of the file:
      Code:
      [share]
      comment = Ubuntu File Server Share
      path = /srv/samba/share
      browsable = yes
      guest ok = yes
      read only = no
      create mask = 0755
    9. Save and exit the file.
    10. Restart the samba services to utilize the new configuration by typing:
      Code:
      service smbd restart
      service nmbd restart
    11. Update your firewall rules to allow Samba file sharing:
      Code:
      ufw allow proto tcp to any port 135,139,445 comment 'Samba Share'
      ufw allow proto udp to any port 137,138 comment 'Samba Share'
    12. You should now be able to click Start --> Run and type \\srv-ubuntu or \\192.168.107.2 {ENTER} and see an explorer window with a Share folder. Drag-n-drop a file into the Share folder. If it worked, it will not display an error message and you should be able to view it from the server by typing ls -l /srv/samba/share/
    13. Shutdown and power off the server by typing shutdown -P now {ENTER}
    14. In VM menu, select VM --> Snapshot --> Take Snapshot. Give it a name like STEP 4 and description of Ubuntu Server 18.04 LTS, File share configured, Static IP: 192.168.107.2. The Snapshot Manager should now have a nice hierarchy of snapshots (STEP 1 --> STEP 2 --> STEP 3 --> STEP 4 --> You are here)
    Last edited by LHammonds; May 3rd, 2018 at 06:16 PM. Reason: Additional permission settings

  8. #18
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb File Sharing via NFS (Linux to Linux)

    NFS Mount (Sharing among Linux)

    If you want to share files between Linux servers, you can use NFS. Here is how you can do that.

    On the machine that will host the files to be shared:


    1. Install the required software:
      Code:
      apt install nfs-kernel-server
    2. Configure a folder to be shared:
      Code:
      vi /etc/exports
    3. Add the following line, adjust as necessary and then save and close the file:
      Code:
      /srv/samba/share        *(rw,sync,no_root_squash,no_subtree_check)
    4. Make the folder and configure ownership and permissions:
      Code:
      mkdir -p /srv/samba/share
      chown nobody:nogroup /srv/samba/share
      chmod 777 /srv/samba/share
    5. Restart services for changes to take effect:
      Code:
      systemctl restart nfs-kernel-server
    6. If your UFW firewall is enabled, you will need to open port 2049 TCP and UDP. Example:
      Code:
      ufw allow proto tcp to any port 2049 comment 'NFS Share'
      ufw allow proto udp to any port 2049 comment 'NFS Share'



    On the machine that will connect to the other remotely:

    1. Install the required software:
      Code:
      apt install nfs-common
    2. Create the mount point folder:
      Code:
      mkdir -p /mnt/nfs
      touch /mnt/nfs/offline.txt
    3. Now connect to the server. We will assume the server's IP doing the sharing is 192.168.107.100 in this example.
      Code:
      mount 192.168.107.100:/srv/samba/share /mnt/nfs
    4. If the mount was successful, you will not see the "offline.txt" file we created at the mount point.
    5. To disconnect from the server, type the following:
      Code:
      umount /mnt/nfs
    6. If the unmount was successful, you will see the "offline.txt" file.

  9. #19
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb File Sharing via Windows Shares

    Configure Windows Server as a Remote Mount Point

    If you have a Windows 2008 server that serves as your offsite backup repository, this section describes how to configure and mount the remote server for storage use.

    Part of the backup process involves copying the backup files to an offsite storage server.

    First, let's document the variables for this solution below and highlight them in red throughout the document for easy identification.

    The values below are merely samples which you need to change in order to match your environment:

    Windows AD Domain Name: work
    Windows AD Share ID: ubuntushare
    Windows AD Share Password: ubuntupassword
    Windows Server Name: SRV-Backup
    Windows Server IP: 192.168.107.218
    Windows Share Name: ubuntu
    Windows Physical Share Location: D:\Ubuntu\

    Create a share on a Windows 2008 server


    1. In Windows Explorer, right-click on the D:\Ubuntu folder and select Properties
    2. Click the Sharing tab
    3. Click the Advanced Sharing button
    4. Place a checkmark beside Share this folder
    5. Change the Share name to ubuntu
    6. Set the Comment to Ubuntu Backup
    7. Click the Permissions button
    8. Select Everyone and click the Remove button
    9. Click the Add button
    10. Type in your Ubuntu share account: work\ubuntushare and click the Check Names button, click OK
    11. Place a checkmark for Allow Full Control and click OK, click OK, click OK
    12. Create a text file in the root of the shared folder called "online.txt" and you might want to add some text inside saying to never delete this file because it is used (will be) by a backup script. Probably a good idea to make it read-only as well. Example: D:\Ubuntu\online.txt


    Create an NFS mount to the Windows 2008 server

    Connecting to a Windows share requires the samba and cifs-utils packages to be installed. If you did not install them (from a prior section), type apt -y install samba cifs-utils


    1. At the login prompt, login with your administrator account (administrator / myadminpass)
    2. At the $ prompt, temporarily grant yourself super user privileges by typing sudo su {ENTER} and then provide the administrator password (myadminpass).
    3. Type the following commands:
      Code:
      mkdir -p /mnt/backup
      chown root:root /mnt/backup
      chmod 0755 /mnt/backup
      echo "This file is used to tell if the mount is active or not" > /mnt/backup/offline.txt
      chown root:root /mnt/backup/offline.txt
      chmod 0444 /mnt/backup/offline.txt
      touch /etc/cifspw
      chmod 0600 /etc/cifspw
    4. Type vi /etc/cifspw and add the following text, save and exit the file:
      Code:
      username=ubuntushare
      domain=work
      password=ubuntupassword
    5. Type vi /etc/hosts and add the following line anywhere in the file:
      Code:
      192.168.107.218    srv-backup
    6. At this point, you might want to type ping srv-backup to make sure you typed the right IP address as well as seeing a good response.
    7. To mount this system for backups, type the following command:
      Code:
      mount -t cifs //srv-backup/ubuntu /mnt/backup --options nouser,rw,nofail,noexec,credentials=/etc/cifspw
    8. To test it, type cp /etc/hosts /mnt/backup/hosts.txt and look on the Windows server and see if the file shows up. Then type rm /mnt/backup/hosts.txt and verify that the file was deleted on the windows server.
    9. This would also be a good time to verify that you can see the "online.txt" file that will be used by the backup script. Type ls -l /mnt/backup/*.txt
    10. To dismount the windows share, type the following command:
      Code:
      umount /mnt/backup


    The scripts will call a common mount and unmount function to connect to this share only when needed.

    However, if you would rather have it mounted all the time (even after a reboot), do the following (but remember to not use the mount/umount functions in the scripts later):


    1. Type vi /etc/fstab and add the following line at the bottom of the file:
      Code:
      //srv-backup/ubuntu    /mnt/backup    cifs nouser,rw,nofail,noexec,credentials=/etc/cifspw    0    0
    2. Type mount -a and if it does not spew out any error messages, it will quietly mount the share.
    3. To test it, type cp /etc/hosts /mnt/backup/hosts.txt and look on the Windows server and see if the file shows up. Then type rm /mnt/backup/hosts.txt and verify that the file was deleted on the windows server.
    4. If you need to unmount it, simply type umount /mnt/backup and it will remain unmounted until you reboot. To make it permanent, you need to remove the line you added in the /etc/fstab file.


    Sometimes it is helpful during tests to manually toggle the mount on or off so here is a script you might find helpful. You can find the contents of "standard.conf" file in the scripting section.

    togglemount.sh
    Code:
    #!/bin/bash
    #############################################
    ## Name          : togglemount.sh
    ## Version       : 1.1
    ## Date          : 2017-03-17
    ## Author        : LHammonds
    ## Compatibility : Ubuntu Server 10.04 thru 18.04 LTS
    ## Purpose       : Toggle the mount status of a pre-configured backup mount.
    ## Run Frequency : Manual as needed.
    ## Exit Codes    :
    ##   0 = success
    ##   1 = failure
    ################ CHANGE LOG #################
    ## DATE       WHO WHAT WAS CHANGED
    ## ---------- --- ----------------------------
    ## 2011-11-05 LTH Created script.
    ## 2017-03-17 LTH Updated variable standards.
    #############################################
    
    ## Import common variables and functions. ##
    source /var/scripts/common/standard.conf
    ErrorFlag=0
    
    if [ -f ${OffsiteDir}/offline.txt ]; then
      echo "Windows share is not mounted.  Mounting share now..."
      f_mount
      sleep 2
      if [ -f ${OffsiteDir}/online.txt ]; then
        echo "Mount successful.  Listing contents:"
      else
        echo "Mount failed.  Listing contents:"
        ErrorFlag=1
      fi
    else
      echo "Windows share is mounted.  Dismounting share now..."
      f_umount
      sleep 2
      if [ -f ${OffsiteDir}/offline.txt ]; then
        echo "Dismount successful.  Listing contents:"
      else
        echo "Dismount failed.  Listing contents:"
        ErrorFlag=1
      fi
    fi
    ls -l ${OffsiteDir}
    exit ${ErrorFlag}

  10. #20
    Join Date
    Sep 2011
    Location
    Behind you!
    Beans
    1,079
    Distro
    Ubuntu 16.04 Xenial Xerus

    Lightbulb Partition-Level Backups

    Backup Partitions Using LVM Snapshots and FSArchiver

    This method will allow online backup of the server at the partition level. It is designed to run via crontab schedule but can also be run manually.

    This should be considered a full backup which means you will probably need to rely on other methods for granular backups and restores such as using rsync at the file level.

    This method is great for backing up a system just prior to and just after a major upgrade of the OS or application. It is not very helpful for retrieving individual files although it could be done but would require a bit of work by temporarily restoring to an unused area, retrieving the file(s) and then destroying the temporary partition.

    The /bak partition is skipped because that is where the archives are being stored.

    The /tmp partition is skipped because there should not be anything in there that needs to be restored...but feel free to include it if you like.

    The script below was built around a few very basic commands that do the bulk of the work but most of the code is for error handling.

    Here are examples of the commands:
    Code:
    ## Create the snapshot volume of the partition to be backed up.
    lvcreate --size=5G --snapshot --name="tempsnap" /dev/LVG/root
    
    ## Create the compressed and encrypted archive of the snapshot.
    fsarchiver savefs --compress=7 --jobs=1 --cryptpass="abc123" --label="insert comment here" /bak/root.fsa /dev/LVG/tempsnap
    
    ## Create an informational text file about the archive.
    fsarchiver archinfo --cryptpass="abc123" /bak/root.fsa > /bak/root.txt 2>&1
    
    ## Remove the snapshot.
    lvremove --force /dev/LVG/tempsnap
    
    ## Create a checksum file about the archive.
    md5sum /bak/root.fsa > /bak/root.md5
    
    ## Verify that the checksum file can validate against the archive.
    md5sum --check /bak/root.md5
    Here is an example of a crontab entry to run the script once a day.

    /var/scripts/data/crontab.root
    Code:
    0 4 * * * /var/scripts/prod/back-parts.sh > /dev/null 2>&1
    Here is the script.

    /var/scripts/prod/back-parts.sh
    Code:
    #!/bin/bash
    #############################################################
    ## Name : back-parts.sh (Backup Partitions)
    ## Version : 1.3
    ## Date : 2017-10-04
    ## Author : LHammonds
    ## Purpose : Backup partitions
    ## Compatibility : Verified on Ubuntu Server 12.04 thru 18.04 LTS
    ##                 Verified with fsarchiver 0.8.4)
    ## Requirements : Fsarchiver, Sendemail, run as root
    ## Run Frequency : Once per day or as often as desired.
    ## Parameters : None
    ## Exit Codes :
    ## 0  = Success
    ## 1  = ERROR: Lock file detected
    ## 2  = ERROR: Must be root user
    ## 4  = ERROR: Missing software
    ## 8  = ERROR: LVM problems
    ## 16 = ERROR: File creation problems
    ## 32 = ERROR: Mount/Unmount problems
    ###################### CHANGE LOG ###########################
    ## DATE       VER WHO WHAT WAS CHANGED
    ## ---------- --- --- ---------------------------------------
    ## 2013-01-09 1.0 LTH Created script.
    ## 2017-03-16 1.1 LTH Updated variable standards.
    ## 2017-08-31 1.2 LTH Added create folder if not exist.
    ## 2017-10-04 1.3 LTH Set file permissions.
    #############################################################
    
    ## Import standard variables and functions. ##
    source /var/scripts/common/standard.conf
    
    ## Define local variables.
    LogFile="${LogDir}/${Company}-back-parts.log"
    LockFile="${TempDir}/${Company}-back-parts.lock"
    LVG="/dev/LVG"
    TempLV="${LVG}/tempsnap"
    MaxTempVolSize=1G
    ErrorFlag=0
    ReturnCode=0
    CryptPass="abc123"
    
    #######################################
    ##            FUNCTIONS              ##
    #######################################
    
    function f_cleanup()
    {
      if [ -f ${LockFile} ];then
        ## Remove lock file so other check space jobs can run.
        rm ${LockFile} 1>/dev/null 2>&1
      fi
      if [ ${ErrorFlag} != 0 ]; then
        f_sendmail "ERROR: Script Failure" "Please review the log file on ${Hostname}${LogFile}"
        echo "`date +%Y-%m-%d_%H:%M:%S` - Backup aborted." >> ${LogFile}
      fi
      exit ${ErrorFlag}
    }
    
    function f_archive_fs()
    {
      FSName=$1
      FSPath=$2
    
      ## Purge old backup files.
      if [ -f ${BackupDir}/${Hostname}-${FSName}.fsa ]; then
        rm ${BackupDir}/${Hostname}-${FSName}.fsa
      fi
      if [ -f ${BackupDir}/${Hostname}-${FSName}.txt ]; then
        rm ${BackupDir}/${Hostname}-${FSName}.txt
      fi
      if [ -f ${BackupDir}/${Hostname}-${FSName}.md5 ]; then
        rm ${BackupDir}/${Hostname}-${FSName}.md5
      fi
    
      ## Unmount FileSystem.
      umount /${FSName}
    
      LVLabel="${Hostname}:${FSPath}->/${FSName}"
      ## Create the compressed and encrypted archive of the snapshot.
      fsarchiver savefs --compress=7 --jobs=1 --cryptpass="${CryptPass}" --label="${LVLabel}" ${BackupDir}/${Hostname}-${FSName}.fsa ${FSPath} > /dev/null 2>&1
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Creation of the archive failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of ${BackupDir}/${FSName}.fsa failed, Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=16
        f_cleanup
      fi
    
      ## Create an informational text file about the archive.
      fsarchiver archinfo --cryptpass="${CryptPass}" ${BackupDir}/${Hostname}-${FSName}.fsa > ${BackupDir}/${Hostname}-${FSName}.txt 2>&1
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Creation of info text failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of info file failed for ${BackupDir}/${FSName}.txt, Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=16
        f_cleanup
      fi
    
      ## Create a checksum file about the archive.
      md5sum ${BackupDir}/${Hostname}-${FSName}.fsa > ${BackupDir}/${Hostname}-${FSName}.md5
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Creation of md5 checksum failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of checksum failed for ${BackupDir}/${FSName}.md5, Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=16
        f_cleanup
      fi
    
      ## Verify that the checksum file can validate against the archive.
      md5sum --check --status ${BackupDir}/${Hostname}-${FSName}.md5
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Verification failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: md5 validation check failed for ${BackupDir}/${FSName}.md5. Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=16
        f_cleanup
      fi
    
      ## Set file permissions.
      chmod 0600 ${BackupDir}/${Hostname}-${FSName}.*
    
      BackupSize=`ls -lak --block-size=m "${BackupDir}/${Hostname}-${FSName}.fsa" | awk '{ print $5 }'`
    
      echo "`date +%Y-%m-%d_%H:%M:%S` --- Created: ${BackupDir}/${Hostname}-${FSName}.fsa, ${BackupSize}" >> ${LogFile}
    
      ## Make sure target folder exists.
      if [ ! -d ${OffsiteDir}/${Hostname} ]; then
        ## Create folder.  This is typical of 1st time use.
        mkdir -p ${OffsiteDir}/${Hostname}
      fi
      ## Copy the backup to an offsite storage location.
      echo "`date +%Y-%m-%d_%H:%M:%S` --- Copying archive file to offsite location." >> ${LogFile}
      cp ${BackupDir}/${Hostname}-${FSName}.* ${OffsiteDir}/${Hostname}/. 1>/dev/null 2>&1
      if [ ! -f ${OffsiteDir}/${Hostname}/${Hostname}-${FSName}.fsa ]; then
        ## NON-FATAL ERROR: Copy command did not work.  Send email notification.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- WARNING: Remote copy failed. ${OffsiteDir}/${Hostname}/${Hostname}-${FSName}.fsa does not exist!" >> ${LogFile}
        f_sendmail "Backup Failure - Remote Copy" "Remote copy failed. ${OffsiteDir}/${Hostname}/${Hostname}-${FSName}.fsa does not exist\n\nBackup file still remains in this location: ${Hostname}:${BackupDir}/${Hostname}-${FSName}.fsa"
      fi
    
      ## Remount FileSystem.
      mount /${FSName}
    }
    
    function f_archive_vol()
    {
      LVName=$1
      LVPath=${LVG}/${LVName}
    
      ## Purge old backup files.
      if [ -f ${BackupDir}/${Hostname}-${LVName}.fsa ]; then
        rm ${BackupDir}/${Hostname}-${LVName}.fsa
      fi
      if [ -f ${BackupDir}/${Hostname}-${LVName}.txt ]; then
        rm ${BackupDir}/${Hostname}-${LVName}.txt
      fi
      if [ -f ${BackupDir}/${Hostname}-${LVName}.md5 ]; then
        rm ${BackupDir}/${Hostname}-${LVName}.md5
      fi
    
      ## Create the snapshot volume of the partition to be backed up.
      lvcreate --size=${MaxTempVolSize} --snapshot --name="tempsnap" ${LVPath} > /dev/null 2>&1
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Creation of temporary volume failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of temp volume failed for ${LVPath}, size=${MaxTempVolSize}, Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=8
        f_cleanup
      fi
    
      ## Give the OS a moment to let the LV create command do its thing.
      sleep 2
    
      LVLabel="${Hostname}:${LVPath}"
      ## Create the compressed and encrypted archive of the snapshot.
      fsarchiver savefs --compress=7 --jobs=1 --cryptpass="${CryptPass}" --label="${LVLabel}" ${BackupDir}/${Hostname}-${LVName}.fsa ${TempLV} > /dev/null 2>&1
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Creation of the archive failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of ${BackupDir}/${Hostname}-${LVName}.fsa failed, Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=16
        f_cleanup
      fi
    
      ## Create an informational text file about the archive.
      fsarchiver archinfo --cryptpass="${CryptPass}" ${BackupDir}/${Hostname}-${LVName}.fsa > ${BackupDir}/${Hostname}-${LVName}.txt 2>&1
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Creation of info text failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of info file failed for ${BackupDir}/${Hostname}-${LVName}.txt, Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=16
        f_cleanup
      fi
    
      ## Create a checksum file about the archive.
      md5sum ${BackupDir}/${Hostname}-${LVName}.fsa > ${BackupDir}/${Hostname}-${LVName}.md5
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Creation of md5 checksum failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Creation of checksum failed for ${BackupDir}/${Hostname}-${LVName}.md5, Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=16
        f_cleanup
      fi
    
      ## Set file permissions.
      chmod 0600 ${BackupDir}/${Hostname}-${LVName}.*
    
      ## Remove the snapshot.
      lvremove --force ${TempLV} > /dev/null 2>&1
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Removal of temporary volume failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Removal of temp volume failed. ${TempLV}. Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=8
        f_cleanup
      fi
    
      ## Give the OS a moment to let the LV remove command do its thing.
      sleep 2
    
      ## Verify that the checksum file can validate against the archive.
      md5sum --check --status ${BackupDir}/${Hostname}-${LVName}.md5
      ReturnCode=$?
      if [ ${ReturnCode} != 0 ]; then
        ## Verification failed.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: md5 validation check failed for ${BackupDir}/${Hostname}-${LVName}.md5. Return Code = ${ReturnCode}" >> ${LogFile}
        ErrorFlag=16
        f_cleanup
      fi
    
      BackupSize=`ls -lak --block-size=m "${BackupDir}/${Hostname}-${LVName}.fsa" | awk '{ print $5 }'`
    
      echo "`date +%Y-%m-%d_%H:%M:%S` --- Created: ${BackupDir}/${Hostname}-${LVName}.fsa, ${BackupSize}" >> ${LogFile}
    
      ## Make sure target folder exists.
      if [ ! -d ${OffsiteDir}/${Hostname} ]; then
        ## Create folder.  This is typical of 1st time use.
        mkdir -p ${OffsiteDir}/${Hostname}
      fi
      ## Copy the backup to an offsite storage location.
      echo "`date +%Y-%m-%d_%H:%M:%S` --- Copying archive file to offsite location." >> ${LogFile}
      cp ${BackupDir}/${Hostname}-${LVName}.* ${OffsiteDir}/${Hostname}/. 1>/dev/null 2>&1
      if [ ! -f ${OffsiteDir}/${Hostname}/${Hostname}-${LVName}.fsa ]; then
        ## NON-FATAL ERROR: Copy command did not work.  Send email notification.
        echo "`date +%Y-%m-%d_%H:%M:%S` --- WARNING: Remote copy failed. ${OffsiteDir}/${Hostname}/${Hostname}-${LVName}.fsa does not exist!" >> ${LogFile}
        f_sendmail "Backup Failure - Remote Copy" "Remote copy failed. ${OffsiteDir}/${Hostname}/${Hostname}-${LVName}.fsa does not exist\n\nBackup file still remains in this location: ${Hostname}:${BackupDir}/${Hostname}-${LVName}.fsa"
      fi
    }
    
    #######################################
    ##           MAIN PROGRAM            ##
    #######################################
    
    if [ -f ${LockFile} ]; then
      # Lock file detected.  Abort script.
      echo "Backup partitions script aborted"
      echo "This script tried to run but detected the lock file: ${LockFile}"
      echo "Please check to make sure the file does not remain when backup partitions is not actually running."
      f_sendmail "ERROR: Backup partitions script aborted" "This script tried to run but detected the lock file: ${LockFile}\n\nPlease check to make sure the file does not remain when backup partitions is not actually running.\n\nIf you find that the script is not running/hung, you can remove it by typing 'rm ${LockFile}'"
      ErrorFlag=1
      exit ${ErrorFlag}
    else
      echo "`date +%Y-%m-%d_%H:%M:%S` ${ScriptName}" > ${LockFile}
    fi
    
    ## Requirement Check: Script must run as root user.
    if [ "$(id -u)" != "0" ]; then
      ## FATAL ERROR DETECTED: Document problem and terminate script.
      echo "`date +%Y-%m-%d_%H:%M:%S` ERROR: Root user required to run this script." >> ${LogFile}
      ErrorFlag=2
      f_cleanup
    fi
    
    ## Requirement Check: Software
    command -v fsarchiver > /dev/null 2>&1 && ReturnCode=0 || ReturnCode=1
    if [ ${ReturnCode} = 1 ]; then
      ## Required program not installed.
      echo "`date +%Y-%m-%d_%H:%M:%S` ERROR: fsarchiver not installed." >> ${LogFile}
      ErrorFlag=4
      f_cleanup
    fi
    
    ## Mount the remote folder. ##
    f_mount
    
    if [ -f ${OffsiteTestFile} ]; then
      ## Found local file.  Assuming failed mount.
      echo "`date +%Y-%m-%d_%H:%M:%S` --- ERROR: Cannot detect remote location: ${OffsiteTestFile}" >> ${LogFile}
      ErrorFlag=32
      f_cleanup
    fi
    
    StartTime="$(date +%s)"
    echo "`date +%Y-%m-%d_%H:%M:%S` - Backup started." >> ${LogFile}
    
    f_archive_fs boot /dev/sda1
    f_archive_vol root
    f_archive_vol var
    f_archive_vol tmp
    f_archive_vol home
    f_archive_vol usr
    f_archive_vol srv
    f_archive_vol opt
    #f_archive_vol swap
    
    ## Unmount the Windows shared folder.
    f_umount
    
    ## Calculate total time for backup.
    FinishTime="$(date +%s)"
    ElapsedTime="$(expr ${FinishTime} - ${StartTime})"
    Hours=$((${ElapsedTime} / 3600))
    ElapsedTime=$((${ElapsedTime} - ${Hours} * 3600))
    Minutes=$((${ElapsedTime} / 60))
    Seconds=$((${ElapsedTime} - ${Minutes} * 60))
    
    echo "`date +%Y-%m-%d_%H:%M:%S` --- Total backup time: ${Hours} hour(s) ${Minutes} minute(s) ${Seconds} second(s)" >> ${LogFile}
    
    echo "`date +%Y-%m-%d_%H:%M:%S` - Backup Finished." >> ${LogFile}
    f_cleanup
    Here is an example of the log file:

    /var/log/abc-back-parts.log
    Code:
    2018-04-20_17:11:14 - Backup started.
    2018-04-20_17:11:25 --- Created: /bak/srv-ubuntu-boot.fsa, 63M
    2018-04-20_17:11:25 --- Copying archive file to offsite location.
    2018-04-20_17:12:29 --- Created: /bak/srv-ubuntu-root.fsa, 254M
    2018-04-20_17:12:29 --- Copying archive file to offsite location.
    2018-04-20_17:13:20 --- Created: /bak/srv-ubuntu-var.fsa, 74M
    2018-04-20_17:13:20 --- Copying archive file to offsite location.
    2018-04-20_17:13:32 --- Created: /bak/srv-ubuntu-tmp.fsa, 1M
    2018-04-20_17:13:32 --- Copying archive file to offsite location.
    2018-04-20_17:13:37 --- Created: /bak/srv-ubuntu-home.fsa, 1M
    2018-04-20_17:13:37 --- Copying archive file to offsite location.
    2018-04-20_17:15:11 --- Created: /bak/srv-ubuntu-usr.fsa, 259M
    2018-04-20_17:15:11 --- Copying archive file to offsite location.
    2018-04-20_17:15:40 --- Created: /bak/srv-ubuntu-srv.fsa, 1M
    2018-04-20_17:15:40 --- Copying archive file to offsite location.
    2018-04-20_17:15:46 --- Created: /bak/srv-ubuntu-opt.fsa, 4M
    2018-04-20_17:15:46 --- Copying archive file to offsite location.
    2018-04-20_17:15:47 --- Total backup time: 0 hour(s) 4 minute(s) 33 second(s)
    2018-04-20_17:15:47 - Backup Finished.
    An example email notification when a fatal error occurred:
    Code:
    From: admin@mydomain.com
    Sent: Friday, April 20, 2018 5:20:45 PM
    To: lhammonds@mydomain.com
    Subject: ERROR: Backup partitions script aborted
    
    This script tried to run but detected the lock file: /tmp/abc-back-parts.lock
    
    Please check to make sure the file does not remain when backup partitions is not actually running.
    
    If you find that the script is not running/hung, you can remove it by typing 'rm /tmp/abc-back-parts.lock'
    
    Server: srv-ubuntu
    Program: /var/scripts/prod/back-parts.sh
    Log: /var/log/abc-back-parts.log
    An example email notification when no errors occur and email notifications turned on:
    Code:
    From: admin@mydomain.com
    To: lhammonds@mydomain.com
    Sent: Friday, April 20, 2018 5:20:45 PM
    Subject: Backup Completed
    
    INFO: The partition backup job has completed without any errors.
    
    Server: srv-ubuntu
    Program: /var/scripts/prod/back-parts.sh
    Log: /var/log/abc-back-parts.log
    Troubleshooting Snapshot Failures

    If the snapshot volume could not be automatically removed, here is how you do it:

    Code:
    dmsetup ls
    Code:
    LVG-srv (252, 6)
    LVG-tempsnap    (252, 9)
    LVG-opt (252, 7)
    LVG-swap        (252, 1)
    LVG-root        (252, 0)
    LVG-opt-real    (252, 10)
    LVG-bak (252, 8)
    LVG-tmp (252, 3)
    LVG-tempsnap-cow        (252, 11)
    LVG-usr (252, 4)
    LVG-var (252, 5)
    LVG-home        (252, 2)
    Code:
    dmsetup remove LVG-tempsnap
    dmsetup remove LVG-tempsnap-cow
    The script can fail for multiple reasons...mostly due to lack of space.

    Use "vgdisplay" to make sure you have enough free/unallocated space in the volume group. Your max tempsnap size cannot exceed the amount of unallocated/unused space in the volume group.

    Make sure you have enough room in /bak to hold the partitions as well as your remote location. A new server without any software/data installed will take up at least 652 MB of space when archived.

    Backup Test

    Before the partitions are backed on your server, create a couple of empty test files to verify that the restore in the next section will work.

    Type the following commands:

    Code:
    touch /important.txt
    touch /srv/samba/share/important.txt
    Make sure the above files are included in your backup before testing the restore in the next section.

Page 2 of 3 FirstFirst 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •