Background

A recent move of some mail and web from an in-house machine, where they had been for 25 years, to a VPS in the cloud, gave me a chance to look at the setup with fresh eyes. The result is a familiar, modern pattern: virtual users, Postfix, Dovecot, and Apache running on Debian, but with some differences in the details. I also wanted to do the configuration entirely with Ansible.

All who lived through the dark times, when bits were hewn from the living rock and 640k was enough for anyone, developed a natural aversion to waste. Each package omitted means fewer things to understand, a smaller attack surface, and lower machine requirements. The LowCloud name is a reflection of this.

For example, many people install MySQL, with 8.5 million lines of code, to return account names from a single static table with a dozen rows. SQL does make the job easier, so we use SQLite, which is fast, has zero moving parts, and is easily secured with file permissions.

Rather than a mail administration client like postfixadmin, I put together a perl script to handle the common tasks, like adding new accounts and changing passwords, from the command line.

Not having physical posession of a cloud machine motivated me to spend some time on several options for disk encryption, mostly for the mail spool and user database, although you could apply it more broadly.

Backups also differ from common examples. We use Bacula, and control the backups from a separate machine, ideally isolated behind a NATed firewall. This was more difficult to develop Ansible roles for, but keeps your backups safe, even if the cloud machine is compromised.

Following a recipe is quick, but does not lead to knowledge. I've tried to document my choices and priorities here, so that you can evaluate them for your own situation, and perhaps make other choices that reflect your priorities.

Why Ansible?

Back in the day, you deployed a new machine by hand editing an endless series of config files while sitting at the console, or perhaps via telnet. A few hours or days later, the machine was declared good and left alone.

After doing it a few times, you might copy config files from other machines as a starting point. After many times, you might write a few shell scripts. Big sites had enterprise deployment tools. But after a long gap, you forgot much and had to relearn, iterating as you missed things. Over time, machines accumulated unique tweaks and extra packages.

That is no way to run a railroad.

In software development, we stopped manually calling the C compiler years ago. Today we describe the desired state - source files, object files, executables - and let a tool like make or msbuild figure out which actions to use to place the code into that state.

Ansible is make for machines. Playbooks and roles are the Makefiles that describe "idempotent" tasks that leave the machine in the same state no matter how many times you run them. For example, a config file needs a line "foo=bar". If present, it is left alone. If missing, it is added. If "foo=zap" is present, it is changed to "foo=bar". At the end, that config file has one copy of the value and it is "foo=bar". Other tasks say that a package should be present or absent, and it is installed or uninstalled as needed.

The end result is a machine in a known state. You can describe the state in great detail, put it under source control, and reproduce it as needed. Ansible works remotely over ssh and can operate in parallel on lists of hosts, so you can configure or reproduce whole networks of machines.

This is a wonderful way to run a railroad!

Why Debian?

I have happily used Gentoo, Centos, Mint, and other distros going back to boot floppies. I've also used every iteration of Windows and MacOS, old school picks like SunOS, Solaris, HP-UX, AIX, IRIX, Ultrix and even VMS. I don't hold strong opinions on operating systems (but editors are a different story - I didn't grow that extra meta-finger for nothing!)

I wanted to target a single distro, just to keep the playbooks clean and understandable. Debian was the winner simply because of numbers — 80% of the examples that I ran across used Debian, and it is the origin of a hundred other downstream distros, like Ubuntu and Mint. I started with Debian 9 (Stretch) but moved to Debian 10 (Buster) shortly after it was released.

It shouldn't take too much work to adapt this to other distros. In most places, the Ansible playbooks use the package: module rather than apt: for installing, so it can work with emerge or yum, but there are bound to be some differences in package details or file locations.

Debian Notes

The Debian website has a catalog of all available packages, and a backports catalog with newer versions of certain packages.

I advise using Ansible for any additions, but while prototyping, occasional calls to the apt package manager come in handy:

apt update                 # fetch latest package index
apt list --installed       # all installed packages
apt list <pkg>
apt list <pkg> -a    # list all versions in the repo

# to completely uninstall and reinstall a package.
apt-get --purge remove <pkg>
apt-get install <pkg>