How to Install and Configure an Ubuntu Server 16.04 LTS

[LHammonds]

Crontab Schedule

The crontab schedule can be edited directly by typing "crontab -e" but that can be a bit dangerous. It would be safer to edit a file and then load that file into the schedule. This will allow backups of the schedule to be made. If there is ever a problem with the schedule, it can be re-loaded with a known-good schedule or at least back to the way it was before the last change. This requires the person doing the editing to always work with a copy of the schedule 1st.

Here is an example crontab scheduling file for the root user:

/var/scripts/data/crontab.root

########################################
# Name: Crontab Schedule for root user
# Author: LHammonds
############# Update Log ###############
# 2012-05-20 - LTH - Created schedule
########################################

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Crontab SYNTAX:
# minute(0-59) hour(0-23) day-of-month(1-31) month(1-12) day-of-week(0-6) command-to-execute
#
# Adjust the time clock
#
0 1-23 * * * /usr/sbin/ntpdate ntp.ubuntu.com > /dev/null 2>&1
#
# Backup MySQL Server
#
0 23 * * * /var/scripts/prod/mysql-backup.sh > /dev/null 2>&1
#
# Backup MySQL Database On Demand
#
0-59 * * * * /var/scripts/prod/mysql-db-backup.sh > /dev/null 2>&1
#
# Daily checks for available space
#
0 1 * * * /var/scripts/prod/check-storage.sh root 500 100 > /dev/null 2>&1
15 1 * * * /var/scripts/prod/check-storage.sh home 100 50 > /dev/null 2>&1
30 1 * * * /var/scripts/prod/check-storage.sh tmp 100 50 > /dev/null 2>&1
45 1 * * * /var/scripts/prod/check-storage.sh usr 100 50 > /dev/null 2>&1
0 2 * * * /var/scripts/prod/check-storage.sh var 100 50 > /dev/null 2>&1
15 2 * * * /var/scripts/prod/check-storage.sh srv 100 50 > /dev/null 2>&1
30 2 * * * /var/scripts/prod/check-storage.sh opt 100 50 > /dev/null 2>&1
45 2 * * * /var/scripts/prod/check-storage.sh bak 100 50 > /dev/null 2>&1
#
# Daily software upgrade check
#
0 3 * * * /var/scripts/prod/apt-upgrade.sh > /dev/null 2>&1

Once the file is created, make sure appropriate permissions are set by typing the following:

chown root:root /var/scripts/data/crontab.root
chmod 0600 /var/scripts/data/crontab.root

To enable the root schedule using this file, type the following:

crontab -u root /var/scripts/data/crontab.root

To disable the root schedule, type the following:

touch /tmp/deleteme
crontab -u root /tmp/deleteme
rm /tmp/deleteme

If you need to modify the schedule, make a backup copy 1st. For example:

cp /var/scripts/data/crontab.root /var/scripts/data/2012-11-28-crontab.root
vi /var/scripts/data/crontab.root
(make your changes)
crontab -u root /var/scripts/data/crontab.root

[LHammonds]

Configure Windows Server as a Remote Mount Point

If you have a Windows 2008 server that serves as your offsite backup repository, this section describes how to configure and mount the remote server for storage use.

Part of the backup process involves copying the backup files to an offsite storage server.

First, let's document the variables for this solution below and highlight them in red throughout the document for easy identification.

The values below are merely samples which you need to change in order to match your environment:

Windows AD Domain Name: work
Windows AD Share ID: ubuntushare
Windows AD Share Password: ubuntupassword
Windows Server Name: SRV-Backup
Windows Server IP: 192.168.107.218
Windows Share Name: ubuntu
Windows Physical Share Location: D:\Ubuntu\

Create a share on a Windows 2008 server

1. In Windows Explorer, right-click on the D:\Ubuntu folder and select Properties
2. Click the Sharing tab
3. Click the Advanced Sharing button
4. Place a checkmark beside Share this folder
5. Change the Share name to ubuntu
6. Set the Comment to Ubuntu Backup
7. Click the Permissions button
8. Select Everyone and click the Remove button
9. Click the Add button
10. Type in your Ubuntu share account: work\ubuntushare and click the Check Names button, click OK
11. Place a checkmark for Allow Full Control and click OK, click OK, click OK
12. Create a text file in the root of the shared folder called "online.txt" and you might want to add some text inside saying to never delete this file because it is used (will be) by a backup script. Probably a good idea to make it read-only as well. Example: D:\Ubuntu\online.txt

Create an NFS mount to the Windows 2008 server

Connecting to a Windows share requires the samba and cifs-utils packages to be installed. If you did not install them (from a prior section), type apt -y install samba cifs-utils

1. At the login prompt, login with your administrator account (administrator / myadminpass)
2. At the $ prompt, temporarily grant yourself super user privileges by typing sudo su {ENTER} and then provide the administrator password (myadminpass).
3. Type the following commands:
   

   mkdir -p /mnt/backup
   chown root:root /mnt/backup
   chmod 0755 /mnt/backup
   echo "This file is used to tell if the mount is active or not" > /mnt/backup/offline.txt
   chown root:root /mnt/backup/offline.txt
   chmod 0444 /mnt/backup/offline.txt
   touch /etc/cifspw
   chmod 0600 /etc/cifspw

4. Type vi /etc/cifspw and add the following text, save and exit the file:
   

   username=ubuntushare
   domain=work
   password=ubuntupassword

5. Type vi /etc/hosts and add the following line anywhere in the file:
   

   192.168.107.218    srv-backup

6. At this point, you might want to type ping srv-backup to make sure you typed the right IP address as well as seeing a good response.
7. To mount this system for backups, type the following command:
   

   mount -t cifs //srv-backup/ubuntu /mnt/backup --options nouser,rw,nofail,noexec,credentials=/etc/cifspw

8. To test it, type cp /etc/hosts /mnt/backup/hosts.txt and look on the Windows server and see if the file shows up. Then type rm /mnt/backup/hosts.txt and verify that the file was deleted on the windows server.
9. This would also be a good time to verify that you can see the "online.txt" file that will be used by the backup script. Type ls -l /mnt/backup/*.txt
10. To dismount the windows share, type the following command:
    

    umount /mnt/backup

The scripts will call a common mount and unmount function to connect to this share only when needed.

However, if you would rather have it mounted all the time (even after a reboot), do the following (but remember to not use the mount/umount functions in the scripts later):

1. Type vi /etc/fstab and add the following line at the bottom of the file:
   

   //srv-backup/ubuntu    /mnt/backup    cifs nouser,rw,nofail,noexec,credentials=/etc/cifspw    0    0

2. Type mount -a and if it does not spew out any error messages, it will quietly mount the share.
3. To test it, type cp /etc/hosts /mnt/backup/hosts.txt and look on the Windows server and see if the file shows up. Then type rm /mnt/backup/hosts.txt and verify that the file was deleted on the windows server.
4. If you need to unmount it, simply type umount /mnt/backup and it will remain unmounted until you reboot. To make it permanent, you need to remove the line you added in the /etc/fstab file.

Sometimes it is helpful during tests to manually toggle the mount on or off so here is a script you might find helpful. You can find the contents of "standard.conf" file in the scripting section.

togglemount.sh

#!/bin/bash
#############################################
## Name          : togglemount.sh
## Version       : 1.1
## Date          : 2017-03-17
## Author        : LHammonds
## Compatibility : Ubuntu Server 10.04 - 16.04 LTS
## Purpose       : Toggle the mount status of a pre-configured backup mount.
## Run Frequency : Manual as needed.
## Exit Codes    :
##   0 = success
##   1 = failure
################ CHANGE LOG #################
## DATE       WHO WHAT WAS CHANGED
## ---------- --- ----------------------------
## 2011-11-05 LTH Created script.
## 2017-03-17 LTH Updated variable standards.
#############################################

## Import common variables and functions. ##
source /var/scripts/common/standard.conf
ErrorFlag=0

if [ -f ${OffsiteDir}/offline.txt ]; then
  echo "Windows share is not mounted.  Mounting share now..."
  f_mount
  sleep 2
  if [ -f ${OffsiteDir}/online.txt ]; then
    echo "Mount successful.  Listing contents:"
  else
    echo "Mount failed.  Listing contents:"
    ErrorFlag=1
  fi
else
  echo "Windows share is mounted.  Dismounting share now..."
  f_umount
  sleep 2
  if [ -f ${OffsiteDir}/offline.txt ]; then
    echo "Dismount successful.  Listing contents:"
  else
    echo "Dismount failed.  Listing contents:"
    ErrorFlag=1
  fi
fi
ls -l ${OffsiteDir}
exit ${ErrorFlag}

[LHammonds]

SSH Public and Private Keys

If you have more than one Ubuntu server, you probably will want to setup a trusted SSH authentication between the two servers so you can securely and automatically transfer files between them with commands such as scp or sftp.

NOTE: You probably have the root account locked (by default) so you cannot login with root directly. In order to initially copy the public key, you will need to enable the root account temporarily on all servers.

Temporarily enable login with the root account on each server:

1. Connect to a server using PuTTY.
2. At the login prompt, login with your administrator account (administrator / myadminpass)
3. At the $ prompt, temporarily grant yourself super user privileges by typing sudo su {ENTER} and then provide the administrator password (myadminpass).
4. Type the following and set the password to the same as your administrator account (myadminpass):
   

   passwd root

5. Edit /etc/ssh/sshd_config, and comment out the following from:
   

   PermitRootLogin prohibit-password

   to:
   

   #PermitRootLogin prohibit-password

6. Just below that line, add the following and then save/exit:
   

   PermitRootLogin yes

7. Restart SSH:
   

   service ssh restart

8. Do the same for each of your servers

Generate the private/public key on each server:

1. Generate a key by typing the following and accept the default values on everything (even the blank passphrase):
   

   mkdir ~/.ssh
   chmod 700 ~/.ssh
   ssh-keygen -t rsa -b 4096

2. Do the same thing on all your other servers.

Transfer the public key to your trusted servers:

1. Type the following command for each server you have (use your own server name or IP instead of my example names):
   

   ssh-copy-id root@srv-mysql
   ssh-copy-id root@srv-wiki
   ssh-copy-id root@srv-ftp
   ssh-copy-id root@srv-nagios

2. Do the same thing on all your other servers.

Lock the root account on each server:

1. Type the following:
   

   passwd -l root

2. Do the same for each of your servers

Set root login to SSH only on each server:

1. Edit /etc/ssh/sshd_config, and uncomment the following from:
   

   #PermitRootLogin prohibit-password

   to:
   

   PermitRootLogin prohibit-password

2. Just below that line, delete the following and then save/exit:
   

   PermitRootLogin yes

3. Restart SSH:
   

   service ssh restart

4. Do the same for each of your servers

If you have other accounts you want to use across the servers (such as a low-rights account), you can do the same thing as the above except you probably won't need to do the unlock/lock steps for the account if you can already login with that account.

Now you should be able to securely copy files back and forth between servers without having to type a password.

Example:

scp /etc/hosts root@srv-mysql:/tmp/hosts.txt

[LHammonds]

Operator Menu

To make my servers easier to administer within my group (who are not all *NIX guys), I made an operator script that can be used to manipulate the server.

The main features I wanted to make easier for other members of the IT group are:

1. Update the operating system (manually)
2. Disk Status
3. Memory Status
4. Stop/Restart primary services
5. Cleanly shutdown services and reboot server

I accomplished this by creating a script and making a shortcut to it so the only thing they need to type on each server is "opm" which is short for "Operator Menu"

Here is how to do something similar for your servers.

Create the script file and set the permissions so that only the root user can run the script and make a shortcut for it:

touch /var/scripts/prod/opm.sh
chmod 0700 /var/scripts/prod/opm.sh
ln -s /var/scripts/prod/opm.sh /usr/sbin/opm

You will need a program called "dialog" installed for this menu to work.

apt install dialog

Now open the script in your favorite editor and add the following:
/var/scripts/prod/opm.sh

#!/bin/bash
#############################################
## Name          : opm.sh
## Version       : 1.1
## Date          : 2017-03-17
## Author        : LHammonds
## Compatibility : Ubuntu Server 12.04-16.04 LTS
## Requirements  : dialog (apt-get dialog) and root privileges
## Purpose       : Display menu to control the server
## Run Frequency : As needed
## Exit Codes    : None
## SymLink Cmd   : ln -s /var/scripts/prod/opm.sh /usr/sbin/opm
################ CHANGE LOG #################
## DATE       WHO  WHAT WAS CHANGED
## ---------- ---- ----------------------------
## 2013-01-07 LTH  Created script.
## 2017-03-17 LTH  Updated variable standards.
#############################################

## Store menu options selected by the user.
TempDir="/tmp"
ScriptDir="/var/scripts/prod"
Input="${TempDir}/opm-input.$$"

## Storage file for displaying cal and date command output.
Output="${TempDir}/opm-output.$$"

## Get text editor or fall back to vi_editor.
vi_editor=${EDITOR-vi}

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  echo -e "\nERROR: Root user required to run this script.\n"
  echo -e "Type 'sudo su' to temporarily become root user.\n"
  exit
fi

## Trap and delete temp files.
trap "rm $Output; rm $Input; exit" SIGHUP SIGINT SIGTERM

function f_display_output(){
  ## Purpose - display output using msgbox
  ##  $1 -> set msgbox height
  ##  $2 -> set msgbox width
  ##  $3 -> set msgbox title
  local h=${1-10}     ## box height default 10
  local w=${2-41}     ## box width default 41
  local t=${3-Output} ## box title
  dialog --backtitle "KDC Operator Menu for $(hostname -f)" --title "${t}" --clear --msgbox "$(<$Output)" ${h} ${w}
}

function f_showdate(){
  ## Purpose - display current system date & time
  echo "Today is $(date) @ $(hostname -f)." >$Output
  f_display_output 6 60 "Date and Time"
}

function f_checkdisk(){
  ## Purpose: Display disk status.
  clear
  echo -e "df --block-size=M\n"
  df --block-size=M
  echo ""
  read -p "Press [Enter] key to continue..."
}

## Loop the menu display.
while true
do
  ## Display main menu.
  dialog --clear  --no-cancel --backtitle "Operator Menu for $(hostname -f)" \
  --title "[ M A I N - M E N U ]" \
  --menu "You can use the UP/DOWN arrow keys, the first \n\
  letter of the choice as a hot key, or the \n\
  number keys 1-9 to choose an option.\n\
  Choose the TASK" 19 50 7 \
  Exit "Exit menu" \
  OSUpdate "Update Operating System" \
  CheckDisk "Check Disk Status" \
  MEMCheck "Look at running processes" \
  ServiceRestart "Stop/Start Main Services" \
  Reboot "Cleanly reboot server" \
  Date/time "Displays date and time" 2>"${Input}"

  menuitem=$(<"${Input}")

  ## Make decision.
  case $menuitem in
    OSUpdate) ${ScriptDir}/abc-apt-upgrade.sh;;
    CheckDisk) f_checkdisk;;
    MEMCheck) htop;;
    Reboot) ${ScriptDir}/reboot.sh;;
    ServiceRestart) ${ScriptDir}/servicerestart.sh;;
    Date/time) f_showdate;;
    Exit) clear; echo "Clean menu exit."; break;;
  esac
done

## Delete temp files.
[ -f $Output ] && rm $Output
[ -f $Input ] && rm $Input

I like the opm script to as static as possible when copying among the many servers I have so I push the reboot and service restart "logic" into their own scripts which can be unique to each server.

On a MySQL server, the service restart script would look something like this:

/var/scripts/prod/servicerestart.sh

#!/bin/bash
#############################################
## Name          : servicerestart.sh
## Version       : 1.0
## Date          : 2013-01-08
## Author        : LHammonds
## Compatibility : Ubuntu Server 12.04-16.04 LTS
## Requirements  : None
## Purpose       : Stop/Start primary services.
## Run Frequency : As needed
## Exit Codes    : None
################ CHANGE LOG #################
## DATE       WHO  WHAT WAS CHANGED
## ---------- ---- ----------------------------
## 2013-01-08 LTH  Created script.
#############################################

clear
echo ""
echo "Stopping services..."
service mysql stop
sleep 1
echo "Starting services..."
service mysql start
sleep 1

On a MySQL server, the reboot script would look something like this:

/var/scripts/prod/reboot.sh

#!/bin/bash
#############################################
## Name          : reboot.sh
## Version       : 1.0
## Date          : 2013-01-08
## Author        : LHammonds
## Compatibility : Ubuntu Server 12.04-16.04 LTS
## Requirements  : Run as root
## Purpose       : Stop services and reboot server.
## Run Frequency : As needed
## Exit Codes    : None
################ CHANGE LOG #################
## DATE       WHO  WHAT WAS CHANGED
## ---------- ---- ----------------------------
## 2013-01-07 LTH  Created script.
#############################################

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  echo -e "\nERROR: Root user required to run this script.\n"
  echo -e "Type 'sudo su' to temporarily become root user.\n"
  exit
fi

clear
echo ""
echo "Stopping MySQL service..."
service mysql stop
echo "Rebooting..."
echo "3"
sleep 1
echo "2"
sleep 1
echo "1"
sleep 1
shutdown -r now

---

Customizing Operator Menu

Let's say you have "mytop" installed on your MySQL server and you want that added to your menu. This is how you would do that.

Edit the opm.sh file and find this section:

  OSUpdate "Update Operating System" \
  CheckDisk "Check Disk Status" \
  MEMCheck "Look at running processes" \
  ServiceRestart "Stop/Start Main Services" \
  Reboot "Cleanly reboot server" \
  Date/time "Displays date and time" 2>"${Input}"

Now add the following line so it shows up in your menu:

  MySQLCheck "Look at MySQL processes" \

End result looking like this:

  OSUpdate "Update Operating System" \
  CheckDisk "Check Disk Status" \
  MEMCheck "Look at running processes" \
  MySQLCheck "Look at MySQL processes" \
  ServiceRestart "Stop/Start Main Services" \
  Reboot "Cleanly reboot server" \
  Date/time "Displays date and time" 2>"${Input}"

Now that the menu entry is there, we need the menu item to actually do something so find this section below:

    OSUpdate) ${ScriptDir}/abc-apt-upgrade.sh;;
    CheckDisk) f_checkdisk;;
    MEMCheck) htop;;
    Reboot) ${ScriptDir}/reboot.sh;;
    ServiceRestart) ${ScriptDir}/servicerestart.sh;;
    Date/time) f_showdate;;
    Exit) clear; echo "Clean menu exit."; break;;

Add the following line:

    MySQLCheck) mytop;;

The end result looking like this:

    OSUpdate) ${ScriptDir}/abc-apt-upgrade.sh;;
    CheckDisk) f_checkdisk;;
    MEMCheck) htop;;
    MySQLCheck) mytop;;
    Reboot) ${ScriptDir}/reboot.sh;;
    ServiceRestart) ${ScriptDir}/servicerestart.sh;;
    Date/time) f_showdate;;
    Exit) clear; echo "Clean menu exit."; break;;

Save and close the script.

Now when you run opm, you will see the MySQL entry which will run mytop when you select it.

With this menu example, I have given you 3 example types of how you can do something. You can call the program directly, like mytop or htop, or you can call a function to process several lines of script like f_checkdisk or f_showdate or you can call an external script like reboot.sh or servicerestart.sh which can further isolate individual server needs in other scripts.

[LHammonds]

Firewall

Linux uses IPTables as its firewall but learning the ins and outs of the syntax can be daunting. Luckily for us, Ubuntu has a front-end for it called Uncomplicated Firewall (UFW for short).

If you are not using IPv6 (most networks are still just using IPv4), you can comment out the IPv6 setting so you don't get doubles of each rule (one for IPv4 and one for IPv6).

To disable IPv6 rules:

vi /etc/default/ufw

Find:

IPV6=yes

Change to:

#IPV6=yes

Here is a script I use to configure the firewall rules and turn it on...which will remain on even during a reboot.

You can copy this script and use it for your own purposes and tweak it for your environment. Enough commands are used/documented in it that you should be able to modify it to fit your particular server. For example, if running a Minecraft server, you could add a command to allow TCP port 25565. If you are not running a web server, you could remove the TCP ports 80, 8080 and 443 from the application-specific section.

I "NEVER" simply copy this script and run it. Each and every server requires a custom variation of this script tailored for it. The application section is much larger than needed because its much faster to simply remove or edit what is not needed than adding.

enable-firewall.sh

#!/bin/bash
#############################################
## Name          : enable-firewall.sh
## Version       : 1.1
## Date          : 2017-04-13
## Author        : LHammonds
## Compatibility : Ubuntu Server 14.04-16.04 LTS
## Requirements  : Run as root
## Purpose       : Restore and enable firewall.
## Run Frequency : As needed
## Exit Codes    : None
################ CHANGE LOG #################
## DATE       WHO  WHAT WAS CHANGED
## ---------- ---- ----------------------------
## 2015-08-28 LTH  Created script.
## 2017-04-13 LTH  Added comments in rules.
#############################################

## Requirement Check: Script must run as root user.
if [ "$(id -u)" != "0" ]; then
  ## FATAL ERROR DETECTED: Document problem and terminate script.
  echo -e "\nERROR: Root user required to run this script.\n"
  echo -e "Type 'sudo su' to temporarily become root user.\n"
  exit
fi

clear
echo ""
echo "Resetting Firewall to factory default"
echo y | ufw reset 1>/dev/null 2>&1
ufw default deny incoming 1>/dev/null 2>&1
ufw default allow outgoing 1>/dev/null 2>&1
echo "Allowing SSH from only LAN connections"
ufw allow from 192.168.1.0/24 to any port 22 comment 'SSH via LAN' 1>/dev/null 2>&1
echo "Allowing Samba file sharing connections"
ufw allow proto tcp to any port 135,139,445 comment 'Samba Share' 1>/dev/null 2>&1
ufw allow proto udp to any port 137,138 comment 'Samba Share' 1>/dev/null 2>&1
echo "Allowing Nagios connections"
ufw allow from 192.168.107.21 to any port 12489 comment 'Nagios' 1>/dev/null 2>&1
ufw allow from 192.168.107.21 proto tcp to any port 5666 comment 'Nagios' 1>/dev/null 2>&1
echo "Adding Application-specific rules"
echo "Adding MySQL/MariaDB rules"
ufw allow from 192.168.107.0/24 proto tcp to any port 3306 comment 'MariaDB via LAN' 1>/dev/null 2>&1
ufw allow from 192.168.2.0/24 proto tcp to any port 3306 comment 'MariaDB via LAN' 1>/dev/null 2>&1
echo "Adding FTP/FTPS rules"
ufw allow proto tcp to any port 990 comment 'FTPS' 1>/dev/null 2>&1
ufw allow proto tcp to any port 21 comment 'FTP' 1>/dev/null 2>&1
ufw allow proto tcp to any port 2000:2020 comment 'FTP Passive' 1>/dev/null 2>&1
echo "Adding Web Server rules"
ufw allow proto tcp to any port 80 comment 'Web Service' 1>/dev/null 2>&1
ufw allow proto tcp to any port 8080 comment 'Web Service' 1>/dev/null 2>&1
ufw allow proto tcp to any port 443 comment 'Web Service' 1>/dev/null 2>&1
echo "Enabling firewall"
echo y | ufw enable 1>/dev/null 2>&1
echo "Firewall enabled and all rules have been configured."

[LHammonds]

Boot Issues

systemd-analyze plot > /srv/samba/share/boot.xml

Open boot.xml in a web browser to examine the boot process to see if anything is taking longer than it should.

systemd-analyze blame

This shows a list of all running processes ordered by the time it took to initialize. Be careful not to judge a process too harshly at 1st sight...it may have been waiting on another process to initialize 1st.

dmesg > /tmp/boot.txt

Open boot.txt and observe the order of events from the moment it starts its boot all the way to the end. The events are ordered by time and it shows how much time has passed since the machine was turned on for that particular event to occur.

[LHammonds]

NFS Mount (Linux to Linux)

If you want to share files between Linux servers, you can use NFS. Here is how you can do that.

On the machine that will host the files to be shared:

1. Install the required software:
   

   apt install nfs-kernel-server

2. Configure a folder to be shared:
   

   vi /etc/exports

3. Add the following line, adjust as necessary and then save and close the file:
   

   /srv/samba/share        *(rw,sync,no_root_squash,no_subtree_check)

4. Make the folder and configure ownership and permissions:
   

   mkdir -p /srv/samba/share
   chown nobody:nogroup /srv/samba/share
   chmod 777 /srv/samba/share

5. Restart services for changes to take effect:
   

   systemctl restart nfs-kernel-server

6. If your UFW firewall is enabled, you will need to open port 2049 TCP and UDP. Example:
   

   ufw allow proto tcp to any port 2049
   ufw allow proto udp to any port 2049

On the machine that will connect to the other remotely:

1. Install the required software:
   

   apt install nfs-common

2. Create the mount point folder:
   

   mkdir -p /mnt/nfs
   touch /mnt/nfs/offline.txt

3. Now connect to the server. We will assume the server's IP doing the sharing is 192.168.107.100 in this example.
   

   mount 192.168.107.100:/srv/samba/share /mnt/nfs

4. If the mount was successful, you will not see the "offline.txt" file we created at the mount point.
5. To disconnect from the server, type the following:
   

   umount /mnt/nfs

6. If the unmount was successful, you will see the "offline.txt" file.

[LHammonds]

Migrating users/groups from one server to another

WARNING: These migration commands are completely untested by me...just jotting down these research notes for future testing.

Commands to run on old server

Copy accounts to passwd.mig using awk to filter out system accounts (i.e. only copy user accounts)

awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /tmp/passwd.mig

Copy groups to group.mig

awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/group > /tmp/group.mig

Copy shadow to shadow.mig

awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534) {print $1}' /etc/passwd | tee - |egrep -f - /etc/shadow > /tmp/shadow.mig

Copy gshadow to gshadow.mig (rarely used):

cp /etc/gshadow /tmp/gshadow.mig

Backup the home directories.

tar -zcvpf /tmp/home.tar.gz /home

If you have Samba users...

cp /etc/samba/smb.conf /tmp/smb.mig
cp /etc/samba/smbusers /tmp/smbusers.mig
cp /etc/samba/smbpasswd /tmp/smbpasswd.mig

Transfer the migration files to new server.

scp /tmp/*.mig administrator@newserver:/tmp/.
scp /tmp/home.tar.gz administrator@newserver:/tmp/.

Commands to run on new server

Backup current files (and their backups).

mkdir /root/save
cp /etc/passwd /etc/passwd- /etc/shadow /etc/shadow- /etc/group /etc/group- /etc/gshadow /etc/gshadow- /root/save

If you have Samba users, backup your current files.

cp /etc/samba/smb.conf /etc/samba/smbusers /etc/samba/smbpasswd /root/save

Merge/Overwrite files.

cat /tmp/passwd.mig >> /etc/passwd
cat /tmp/group.mig >> /etc/group
cat /tmp/shadow.mig >> /etc/shadow
cp /tmp/gshadow.mig /etc/gshadow

Ensure file ownership and permissions are correct.

chown root:shadow /etc/shadow /etc/gshadow
chmod 640 /etc/shadow /etc/gshadow
chown root:root /etc/passwd /etc/group
chmod 644 /etc/passwd /etc/group

If you have Samba users...

cp /tmp/smb.mig /etc/samba/smb.conf
cp /tmp/smbusers.mig /etc/samba/smbusers
cp /tmp/smbpasswd.mig /etc/samba/smbpasswd

Extract home directories.

cd /
tar -zxvf /tmp/home.tar.gz

Reboot system for new user accounts to take effect.

reboot

Replacing a server with a new one

If you want to keep SSH keys and the fingerprint from changing when swapping out an old server for a new one (e.g. old FTPS server to new FTPS), these are the steps needed to keep the system looking the same. (I think...but will know when I upgrade my FTPS server)

The old server name and the new server name must batch. Example: srv-ftps

Make sure the sshd configuration files on both servers are pointing to the same files (filenames and paths)

vi /etc/ssh/sshd_config

Look for section like this:

# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

If there are differences, figure out if they matter and if so, make sure the transferred files on the new server are being referenced correctly in the config file on the new server.

Commands to run on old server

Transfer files to new server

scp /etc/ssh/ssh_host* administrator@newserver:/tmp/.

Commands to run on new server

Backup existing files

mkdir /root/ssh-backup
cp /etc/ssh/ssh_host* /root/ssh-backup/.

Set correct file ownership and permissions

chown root:root /tmp/ssh_host*
chmod 600 /tmp/ssh_host*
chmod 644 /tmp/ssh_host*.pub

Move and overwrite files to correct location

mv /tmp/ssh_host* /etc/ssh/.

Restart SSH daemon

service ssh restart

You will also want to copy over individual user's public keys in their ~/.ssh/authorized_keys but if you transferred the contents of the /home folder, this was already done.

[LHammonds]

Fail2Ban

Fail2Ban is an intrusion prevention system that can be used to protect servers from different kinds of attacks.

Fail2ban scans log files for various services, such as SSH, FTP, SMTP, Apache and block the IP address that makes too many password failures.

Install Fail2Ban

apt-get install fail2ban

Configure Fail2Ban
The default configuration file should not be altered since upgrades tend to overwrite this file with new settings.

Instead, use a custom configuration file that will simply override or append various settings to the default.

Fail2Ban reads .conf files first and then .local files last which can override any settings initially set in .conf files.

First, look at the default file:

vi /etc/fail2ban/jail.conf

Notice the following partial list of default values. These are based on the OS and version at the time of this writing:

[DEFAULT]
ignoreip = 127.0.0.1/8
bantime  = 600
findtime  = 600
maxretry = 5
filter = %(__name__)s
destemail = root@localhost
sender = root@localhost
mta = sendmail
action = %(action_)s
[sshd]
port    = ssh
logpath = %(sshd_log)s

Now create a local jail file that will be used to override any settings in the default that we want changed or added.

For this example, we will ignore all failed login attempts from our local server subnet (192.168.107.xxx) and a specific admin workstation (192.168.1.69)

vi /etc/fail2ban/jail.local

Let's make some changes that fit our server as it stands right now:

[DEFAULT]
## 127.0.0.1/8 = ignore login failures on the local machine
## 192.168.107.0/24 = ignore login failures on an entire subnet
## 192.168.1.69 = ignore login failures for this specific IP
ignoreip = 127.0.0.1/8 192.168.107.0/24 192.168.1.69

## "bantime" is the number of seconds that a host is banned. 600=10 minutes, 1800=30 minutes.
bantime = 1800
## "findtime" is the length of time between login attempts before a ban is set.
findtime = 600
## "maxretry" is how many attempts can be made to access the server from a single IP before a ban is imposed.
maxretry = 7

## "destemail" is the email address where you would like to receive the emails.
destemail = webmaster@mydomain.com
## "sender" is the FROM: address when it arrives in your mailbox.
sender = myserver@mydomain.com
## Use the lightweight sendemail instead of sendmail
mta = sendemail
## Email notify with whois report and relevant log lines when a ban occurs
action = %(action_mwl)s

We are using sendemail for our scripts so we will also use it for fail2ban. We will need to make a configuration file that tells fail2ban how to make use of it though.

vi /etc/fail2ban/action.d/sendemail-whois-lines.conf

[Definition]
actionstart =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: started" -m "The jail <name> has been started successfully.\n\nFail2Ban"
actionstop =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: stopped" -m "The jail <name> has been stopped.\n\nFail2Ban"
actioncheck =
actionban =  /usr/bin/sendemail -f <sender> -t <dest> -s <smtp> -xu <sender> -xp <password> -u "[Fail2Ban] <servername> <name>: banned <ip>" -m "The IP <ip> has just been banned by Fail2Ban after <failures> attempts against <name>.\n\nHere is more information about <ip>:\n `/usr/bin/whois <ip>`\n\n Lines containing IP:<ip> in <logpath>\n`/bin/grep '\<<ip>\>' <logpath>`\n\n\n\nFail2Ban"
actionunban =

[Init]
## Amended to be the same as the SMTP user
sender = fail2ban@mydomain.com
## SMTP password for user
#password = XXXXXXX
## SMTP server - use port 587 for Google rather than 25 (times out too often) or 465 (crashes sendemail)
#smtp = smtp.googlemail.com:587
smtp = srv-mail

## Name for this server - handy when there are lots of servers sending emails to the destemail
servername = srv-ubuntu

Useful commands

Here are a few of the most-used commands. Reference: Fail2Ban commands.
Stop, start or restart the fail2ban service:

service fail2ban stop
service fail2ban start
service fail2ban restart

This command will show which services are being watched (sshd in this example):

fail2ban-client status

Example output:

Status
|- Number of jail:      1
`- Jail list:   sshd

This command gets the status on the sshd service being watched:

fail2ban-client status sshd

Example output:

Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

Firewall Rules

Installing Fail2Ban will create the following firewall rules which can be seen with the following command:

iptables -S

-N f2b-sshd
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A f2b-sshd -j RETURN

If you have already enabled Uncomplicated Firewall (UFW), you will see several ufw-prefixed rules as well.

Test Fail2Ban

To manually ban an IP:

fail2ban-client set sshd banip 192.168.1.69

To manually unban an IP:

fail2ban-client set sshd unbanip 192.168.1.69

On the server, run the following command to watch the Fail2Ban log file:

tail -f /var/log/fail2ban.log

On a different machine (that is not on your ignoreip list), try to ssh into your server with the incorrect ID or password multiple times until you have reached the "maxretry" attempts and you get locked out...thus causing an entry in the log.

References

Fail2Ban Manual

[mörgæs]

Whoa, impressive thread. Thanks for posting.

Feel free to ask for this post to be deleted when you have read it. Just wanted to thank you in public.