Updates

ROLE common, upgrades.yml, logwatch.yml

We set up the unattended upgrades package to install security patches once a day. The unattended-upgrades package installs the updates, the needrestart package bounces services with changed libraries, and the apt-listchanges package emails changelogs to the admin_email address.

We configure the logwatch package to send a daily summary of activity to the admin_email address. The report contains basic information about SSH logins, disk usage, mail traffic, web traffic, and fail2ban activity.

Occasionally a kernel update or other change will require a reboot. The email report will note this. Log in and run sudo shutdown -r now at your convenience. Patches to postfix or dovecot may not restart the servers properly, so if you don't see the normal backula and logwatch messages on a particular morning, you may need to manually restart them (or just reboot and then run mailboot).

OS Update

Eventually, the distribution that you are using will fall out of long term support, and will no longer get security updates. Debian versions are typically released every two years, with full support for three years, and long term support out to five years. It is wise to update before your machine falls out of long term support.

Most distributions have a process for an in-place update to the next major version. Instructions can be found online. I prefer to use a clean OS install, but this requires a bit of preparation.

Ideally, you have separated your system and data disks onto different partitions. Then you can simply replace the system partition with a fresh OS image, then configure with Ansible as usual. The small amount of data on the system disk (like certbot keys) can be handled by the snapshot role.

On Linode, I create /dev/sda from an OS image, swap as /dev/sdb, then create /dev/sdc for a LUKS encrypted data disk. Using their online manager, you can delete the sda system disk, create a new disk from any given OS image, then use the configuration manager to assign it to sda.

The general process looks like this:

# system on /dev/sda
# swap on /dev/sdb
# data (plain or crypt) on /dev/sdc

# make snapshot/machine.name directory
$ ansible-playbook -K --ask-vault-pass -i ../myhosts site.yml --tags snapshot

# manually cache anything this missed to the data disk

# REPLACE /dev/sda WITH A NEW OS IMAGE.

# boot and configure the new OS image
$ ansible-playbook -k --ask-vault-pass -i ../myhosts first.yml
$ ansible-playbook -K --ask-vault-pass -i ../myhosts site.yml

# manually restore anything that needs it
# delete snapshot/machine.name directory

I will typically test out everything on a scrap machine before updating the live one, There may still be some manual operations when packages have changed how they do things, like rspamd retiring support for sqlite.

• Debian 12 Notes

If you put everything on a single disk, and do not want to in-place upgrade, you may have to reimage the entire machine and restore from backup.

Other Manual Updates

Very rarely, an update may need operator input. These will be deferred and you will get a message about it. To manually install an update to a package:

% sudo apt update 	     	# fetch latest package index
% sudo apt list --upgradable	# list available upgrades

% sudo apt install <pkg>	# update just foobar pkg
% sudo apt --only-upgrade install <pkg>   # same, won't install new packages

This is specific to Debian and its derivitives. For Red Hat, Gentoo, etc. you'll need to figure out how they handle automatic security updates.