LowCloud Mail, Web, and Backup

These Ansible roles set up mail and web service in the cloud, with backups from a separate machine. Requirements are low, with few extra frills, and should appeal to anyone comfortable with stone knives and bearskins. My three goals for this exercise were:

1. Basic mail and web for one or more domains,
2. Built securely with modern packages, and proper backups,
3. A clear description of what is on the machine and why.

These Ansible playbooks assume Debian 10 and configure the following areas. For each one, I describe what is there, how to configure it, and why I chose it, with some fun pictures of my old hardware at the bottom of each page.

• Background
• The Machines

• SSH
• Firewall
• Mail Server
• Mail Relay
• Web Server
• Backup
• Encryption
• DNS
• Webmail
• Updates

I hope this can serve as a starting point for your own project. If these details don't interest you, a push-button mail server like Modoboa, mail-in-a-box, or homebox is a fine alternative.

Getting Started

Check out the lowcloud repo. Make a separate directory for your settings by recursively copying hosts.example to someplace outside of the repo. We call it myhosts here, but any name is fine.

$ cp -r lowcloud/hosts.example myhosts

$ ls
lowcloud  myhosts

The new myhosts directory contains a hosts file and a group_vars directory. After customizing these as described below, I recommend putting this directory under source control. You can do that however you like because it is not connected to the lowcloud repo.

As described in Going Further, you can expand myhosts with your own roles and playbooks, so you only need to fork lowcloud to change a role in that. It would be nice to star the lowcloud repo though, so that I have an sense of how many people find it useful.

Hosts

The myhosts/hosts file identifies your machines, and organizes them into groups. We use two groups: cloud and backup_server. The playbooks apply tasks to the machines in each group.

Machines are listed by fully qualified domain name (FQDN) and IP address. Ansible uses the ansible_host to connect, and sets the inventory_hostname variable to the FQDN for use in the scripts.

[cloud]
# mail and webserver, needs a public IP address from a business class
# ISP connection or VPS provider like AWS, Linode, or DigitalOcean
example.com		ansible_host=1.2.3.4

[backup_server]
# connects to other machines to fetch and store backups, best if you
# have physical control of the machine.  It does not need a public IP
# and can be behind a residential ISP connection.
guardian.example.com	ansible_host=5.6.7.8

Once you have machines, create DNS entries for them using your registrar or other service. We may eventually run Bind on the cloud machine, but still getting that sorted out.

Variables

The myhosts/group_vars directory contains settings for the machine groups in the hosts file. Ansible looks for a <group-name>.yml file or a <group-name>/ directory containing YML files. Every machine belongs to the special all group.

Most settings are in all/all.yml. Look through this file and customize the values as appropriate. A few things unique to each machine are kept in cloud.yml and backup_server.yml.

Sensitive values like passwords are in all/vault.yml, which will be an encrypted Ansible vault file. The vault.yml that you copied is not yet encrypted and contains placeholders. Edit it to contain real values, generate some ssh keys, then encrypt it with your own unique password:

$ ansible-vault encrypt vault.yml	# encrypt plain file

The file stays encrypted. Use the edit command to open a clear-text version temporarily in $EDITOR to look at or modify the contents. You can also change the password with rekey.

$ ansible-vault edit vault.yml 		# run EDITOR on file
$ ansible-vault rekey vault.yml		# change password on file

To make it easier to find variables, we prefix each value in vault.yml with "vault_" and then declare a version without the prefix in the other files. For example, a password might show up as follows:

something_pw: "{{vault_something_pw}}"   # vault_* is in vault.yml

Ansible vault files are encrypted as AES256 and stored as a block of ASCII text, similar to ssh keys. Ansible will decrypt on-the-fly when running playbooks. There are many ways to get the password to Ansible, but we use the --ask-vault-pass flag to type it in every time.

You shouldn't need anything beyond these files for this application, but if you extend these playbooks for other things, be aware that Ansible variables have an absurd amount of flexibility. Each role also has settings in <ROLE>/defaults/main.yml that you can override.

First Login Playbook

The first thing to do after provisioning a machine is run the first.yml playbook to set up a deployment account, install keys, and then lock down SSH.

Tasks are executed remotely as root. The lowercase -k flag is a shorthand for --ask-pass. This is the only time a password-based login ever happens. Ansible needs the sshpass package on your local machine to do password-based login to a remote machine. If you can pre-install an ssh key for root, you could use the --private-key flag instead.

$ ansible-playbook -k --ask-vault-pass -i ../myhosts first.yml --limit cloud
$ ansible-playbook -k --ask-vault-pass -i ../myhosts first.yml --limit backup_server

After this, remote password login is disabled and everything requires ssh keys. All further work uses the deployment account. It is a good idea to change or star the root password because the original passed through your VPS provider's UI.

Site Setup Playbook

The site.yml file is the main playbook for our cloud and backup server machines. Tasks are executed remotely as the deploy user, so Ansible needs passwords for the deploy user (for sudo) and the vault (for decrypting). The uppercase -K flag is a shorthand for --ask-become-pass

Ansible playbooks are meant to be idempotent, which means that they are supposed to leave a machine in a given state, no matter how many times you run it or what state it was in before.

$ ansible-playbook -K --ask-vault-pass -i ../myhosts site.yml

By default, Ansible runs the playbook on all machines in the inventory file. We can limit it to a certain group or machine with the --limit flag. We can also select just parts of the playbook with the --tags flag.

# run the playbook only on the cloud group
$ ansible-playbook -K --ask-vault-pass -i ../myhosts site.yml --limit cloud

# run apache tagged tasks only on the cloud group
$ ansible-playbook -K --ask-vault-pass -i ../myhosts site.yml \
    --limit cloud  --tags apache

Most roles have their own tags, so you can call them selectively. There is another snapshot tag that copies back certain system data like keys and certs for easy transfer when wiping or migrating a machine.

Adding the --check option does a dry-run that does not change anything. This can be combined with the --diff option to report the actual text changes of templated files.

Going Further

You can add your own roles directory and playbook YML files to the myhosts directory and build more extensive actions into your own setup. Set your role search path to include the lowcloud roles to build upon then in your playbook. Create an ansible.cfg file in myhosts that contains:

[defaults]
roles_path=../lowcloud/roles

The Ansible Galaxy site contains roles that people have written for many different things, and is a great place to see worked examples and get ideas.

Rebooting

At boot, everything will start normally except mail, which waits until we log in and unlock the encrypted mail spool by running the mailboot script. Any incoming mail should remain queued by the sending system until the mail daemon is up again.