How to install and configure an Ubuntu Server 20.04 LTS

**LHammonds** · May 4th, 2020

Restore Partitions Using SystemRescueCD and FSArchiver

Partitions cannot be mounted when restoring to them. If services can be stopped that use files on a specific partition, it can be unmounted and restored. However, the root partition can never be restored while the server is online so these instructions will cover the common denominator which requires taking the server offline.

The server needs to be booted up with a CD but not just any CD will do because it needs to have FSArchiver on it. For this document, the ISO image from www.sysresccd.org will be used.

Once downloaded, the ISO can be burned to a CD-ROM disc or uploaded to your ISO repository such as a LUN. VMware and VirtualBox can attach an ISO image and mount it in the CD-ROM device to allow the virtual machine to boot the ISO image.

For this example, files on the root and home partitions will be restored using this summarized procedure:

Boot the system using the RescueCD.
Mount the bak partition which contains the backup archives.
Verify that the backup archives are in good condition.
Restore the partitions.
Mount the root and home partitions.
Verify the restore worked.
Unmount the partitions.
Reboot the system.

Be sure the partitions have been backed up and the files are sitting in the /bak volume.

Code:

ls -l /bak

Code:

total 1035280
drwx------ 2 root root     16384 Apr 30 13:31 lost+found
-rw------- 1 root root 156167681 May  3 15:33 srv-ubuntu-boot.fsa
-rw------- 1 root root        59 May  3 15:33 srv-ubuntu-boot.md5
-rw------- 1 root root       747 May  3 15:33 srv-ubuntu-boot.txt
-rw------- 1 root root     11113 May  3 15:40 srv-ubuntu-home.fsa
-rw------- 1 root root        59 May  3 15:40 srv-ubuntu-home.md5
-rw------- 1 root root       749 May  3 15:40 srv-ubuntu-home.txt
-rw------- 1 root root 837941885 May  3 15:38 srv-ubuntu-root.fsa
-rw------- 1 root root        59 May  3 15:38 srv-ubuntu-root.md5
-rw------- 1 root root       750 May  3 15:38 srv-ubuntu-root.txt
-rw------- 1 root root      7142 May  3 15:40 srv-ubuntu-tmp.fsa
-rw------- 1 root root        58 May  3 15:40 srv-ubuntu-tmp.md5
-rw------- 1 root root       745 May  3 15:40 srv-ubuntu-tmp.txt
-rw------- 1 root root  65930756 May  3 15:40 srv-ubuntu-var.fsa
-rw------- 1 root root        58 May  3 15:40 srv-ubuntu-var.md5
-rw------- 1 root root       748 May  3 15:40 srv-ubuntu-var.txt

As a test of the restore, let's delete the two text files created in the previous section:

Code:

rm /important.txt
rm /home/administrator/important.txt

These files should have been included in the backup image. When the restore is complete, these files should return.

Insert the SystemRescueCD CDROM (or mount the ISO image) and boot the server with it.

Here is the 1st screen:

Option #1 was chosen (SystemRescueCd: default boot options)

The next screen takes you to the command prompt:

Show what partitions the system has by using fsarchiver.

root@sysresccd /root % fsarchiver probe simple

Code:

[======DISK======] [=============NAME==============] [====SIZE====] [MAJ] [MIN]
[sda             ] [VBOX HARDDISK                  ] [    25.00 GB] [  8] [  0]
[sr0             ] [CD-ROM                         ] [   675.00 MB] [ 11] [  0]

[=====DEVICE=====] [==FILESYS==] [======LABEL======] [====SIZE====] [MAJ] [MIN]
[loop0           ] [squashfs   ] [<unknown>        ] [   620.12 MB] [  7] [  0]
[sda1            ] [ext4       ] [boot             ] [   953.00 MB] [  8] [  1]
[sda5            ] [LVM2_member] [<unknown>        ] [    24.87 GB] [  8] [  5]
[dm-0            ] [ext4       ] [root             ] [     6.00 GB] [254] [  0]
[dm-1            ] [ext4       ] [var              ] [     3.00 GB] [254] [  1]
[dm-2            ] [ext4       ] [tmp              ] [     1.00 GB] [254] [  2]
[dm-3            ] [ext4       ] [bak              ] [     4.00 GB] [254] [  3]
[dm-4            ] [ext4       ] [home             ] [     1.00 GB] [254] [  4]

Mount the bak partition which contains the backup archives.

root@sysresccd /root % mkdir /bak
root@sysresccd /root % mount --read-only /dev/dm-3 /bak
root@sysresccd /root % ls -l /bak

Code:

total 1035280
drwx------ 2 root root     16384 Apr 30 13:31 lost+found
-rw------- 1 root root 156167681 May  3 15:33 srv-ubuntu-boot.fsa
-rw------- 1 root root        59 May  3 15:33 srv-ubuntu-boot.md5
-rw------- 1 root root       747 May  3 15:33 srv-ubuntu-boot.txt
-rw------- 1 root root     11113 May  3 15:40 srv-ubuntu-home.fsa
-rw------- 1 root root        59 May  3 15:40 srv-ubuntu-home.md5
-rw------- 1 root root       749 May  3 15:40 srv-ubuntu-home.txt
-rw------- 1 root root 837941885 May  3 15:38 srv-ubuntu-root.fsa
-rw------- 1 root root        59 May  3 15:38 srv-ubuntu-root.md5
-rw------- 1 root root       750 May  3 15:38 srv-ubuntu-root.txt
-rw------- 1 root root      7142 May  3 15:40 srv-ubuntu-tmp.fsa
-rw------- 1 root root        58 May  3 15:40 srv-ubuntu-tmp.md5
-rw------- 1 root root       745 May  3 15:40 srv-ubuntu-tmp.txt
-rw------- 1 root root  65930756 May  3 15:40 srv-ubuntu-var.fsa
-rw------- 1 root root        58 May  3 15:40 srv-ubuntu-var.md5
-rw------- 1 root root       748 May  3 15:40 srv-ubuntu-var.txt

Verify that the backup archives are in good condition:

root@sysresccd /root % md5sum --check /bak/srv-ubuntu-root.md5

Code:

/bak/srv-ubuntu-root.fsa: OK

root@sysresccd /root % md5sum --check /bak/srv-ubuntu-home.md5

Code:

/bak/srv-ubuntu-home.fsa: OK

Restore the root partition:

root@sysresccd /root % fsarchiver restfs --cryptpass="abc123" /bak/srv-ubuntu-root.fsa id=0,dest=/dev/dm-0

Code:

Statistics for filesystem 0
* files successfully processed:....regfiles=96032, directories=19993, symlinks=7266, hardlinks=9, specials=8
* files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0

Restore the home partition:

root@sysresccd /root % fsarchiver restfs --cryptpass="abc123" /bak/srv-ubuntu-home.fsa id=0,dest=/dev/dm-4

Code:

Statistics for filesystem 0
* files successfully processed:....regfiles=11, directories=5, symlinks=0, hardlinks=0, specials=0
* files with errors:...............regfiles=0, directories=0, symlinks=0, hardlinks=0, specials=0

Mount the root and home partitions:

root@sysresccd /root % mkdir /mnt/root
root@sysresccd /root % mkdir /mnt/home
root@sysresccd /root % mount --read-only /dev/dm-0 /mnt/root
root@sysresccd /root % mount --read-only /dev/dm-4 /mnt/home

Verify the restore worked:

root@sysresccd /root % ls -l /mnt/root/import*

Code:

-rw-r--r-- 1 root root 0 May  3 17:00 important.txt

root@sysresccd /root % ls -l /mnt/home/administrator/import*

Code:

-rw-r--r-- 1 root root 0 May  3 17:00 important.txt

Unmount the partitions:

root@sysresccd /root % umount /mnt/root
root@sysresccd /root % umount /mnt/home
root@sysresccd /root % umount /bak

Eject the CDROM/ISO and reboot the server.

If it worked right, the server will boot up (we restored the root file system) and the test files should have been restored.

NOTE: If you noticed any ext file systems that had <unknown> labels, you can update them using the tune2fs command which makes identification easier when restoring.

Example 1:

Code:

sudo tune2fs /dev/sda1 -L boot

Example 2:

Code:

sudo tune2fs /dev/dm-0 root

**LHammonds** · May 4th, 2020

Migrating users/groups from one server to another

WARNING: These migration commands are thoroughly tested by me...just jotting down these research notes for future testing.

Commands to run on old server

Copy accounts to passwd.mig using awk to filter out system accounts (i.e. only copy user accounts)

Code:

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/passwd > /tmp/passwd.mig

Copy groups to group.mig

Code:

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534)' /etc/group > /tmp/group.mig

Copy shadow to shadow.mig

Code:

sudo awk -v LIMIT=1000 -F: '($3>=LIMIT) && ($3!=65534) {print $1}' /etc/passwd | tee - |egrep -f - /etc/shadow > /tmp/shadow.mig

Copy gshadow to gshadow.mig (rarely used):

Code:

sudo cp /etc/gshadow /tmp/gshadow.mig

Backup the home directories.

Code:

sudo tar -zcvpf /tmp/home.tar.gz /home

If you have Samba users...

Code:

sudo cp /etc/samba/smb.conf /tmp/smb.mig
sudo cp /etc/samba/smbusers /tmp/smbusers.mig
sudo cp /etc/samba/smbpasswd /tmp/smbpasswd.mig

Transfer the migration files to new server.

Code:

sudo scp /tmp/*.mig administrator@newserver:/tmp/.
sudo scp /tmp/home.tar.gz administrator@newserver:/tmp/.

Commands to run on new server

Backup current files (and their backups).

Code:

sudo mkdir /root/save
sudo cp /etc/passwd /etc/passwd- /etc/shadow /etc/shadow- /etc/group /etc/group- /etc/gshadow /etc/gshadow- /root/save

If you have Samba users, backup your current files.

Code:

sudo cp /etc/samba/smb.conf /etc/samba/smbusers /etc/samba/smbpasswd /root/save

Merge/Overwrite files.

Code:

sudo cat /tmp/passwd.mig >> /etc/passwd
sudo cat /tmp/group.mig >> /etc/group
sudo cat /tmp/shadow.mig >> /etc/shadow
sudo cp /tmp/gshadow.mig /etc/gshadow

Ensure file ownership and permissions are correct.

Code:

sudo chown root:shadow /etc/shadow /etc/gshadow
sudo chmod 640 /etc/shadow /etc/gshadow
sudo chown root:root /etc/passwd /etc/group
sudo chmod 644 /etc/passwd /etc/group

If you have Samba users...

Code:

sudo cp /tmp/smb.mig /etc/samba/smb.conf
sudo cp /tmp/smbusers.mig /etc/samba/smbusers
sudo cp /tmp/smbpasswd.mig /etc/samba/smbpasswd

Extract home directories.

Code:

cd /
sudo tar -zxvf /tmp/home.tar.gz

Reboot system for new user accounts to take effect.

Code:

sudo reboot

**LHammonds** · May 4th, 2020

Replacing a server with a new one

If you want to keep SSH keys and the fingerprint from changing when swapping out an old server for a new one (e.g. old FTPS server to new FTPS which interfaces with many 3rd-party vendors), these are the steps needed to keep the system looking the same.

The old server name and the new server name must match. Example: srv-ftps

Make sure the sshd configuration files on both servers are pointing to the same files (filenames and paths)

Code:

sudo vi /etc/ssh/sshd_config

Look for section like this:

Code:

# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

If there are differences, figure out if they matter and if so, make sure the transferred files on the new server are being referenced correctly in the config file on the new server.

Commands to run on old server

Transfer files to new server

Code:

sudo scp /etc/ssh/ssh_host* administrator@newserver:/tmp/.

Commands to run on new server

Backup existing files

Code:

sudo mkdir /root/ssh-backup
sudo cp /etc/ssh/ssh_host* /root/ssh-backup/.

Set correct file ownership and permissions

Code:

sudo chown root:root /tmp/ssh_host*
sudo chmod 600 /tmp/ssh_host*
sudo chmod 644 /tmp/ssh_host*.pub

Move and overwrite files to correct location

Code:

sudo mv /tmp/ssh_host* /etc/ssh/.

Restart SSH daemon

Code:

sudo systemctl restart ssh

You will also want to copy over individual user's public keys in their ~/.ssh/authorized_keys but if you transferred the contents of the /home folder, this was already done.

**LHammonds** · May 4th, 2020

Boot Issues

Code:

systemd-analyze plot > /srv/samba/share/boot.xml

Open boot.xml in a web browser to examine the boot process to see if anything is taking longer than it should.

Code:

systemd-analyze blame

This shows a list of all running processes ordered by the time it took to initialize. Be careful not to judge a process too harshly at 1st sight...it may have been waiting on another process to initialize 1st.

Code:

dmesg > /tmp/boot.txt

Open boot.txt and observe the order of events from the moment it starts its boot all the way to the end. The events are ordered by time and it shows how much time has passed since the machine was turned on for that particular event to occur.

Disk Space Issues

To quickly find out status on partition sizes, use this command:

Code:

df -h

To help find out what is eating up your space in a particular folder, you can use this command:

Code:

du -sh /path/to/folder

To make faster use of du, use the ncdu utility I recommended to install in an earlier section. If you start at the root like the example below, you can use your arrow keys to select the problematic folders and keep drilling down until you find the culprit you are looking for. It automatically sorts the results so you are typically already on the biggest folder and just need to press ENTER to drill down to the next folder.

Code: