Encryption

When talking about encryption, people usually distinguish between data in transit and data at rest. To protect data in transit, we encrypt communication links for mail (STARTTLS and IMAPS) and web (HTTPS). To give the other party confidence that we are who we claim to be, we identify ourself using certificates issued by Certbot / LetsEncrypt.

To protect data at rest, we place sensitive data like the mail spool, DKIM keys, and mail user database into a central encrypted directory. When the machine is rebooted, everything starts normally except mail. We must ssh in and run the mailboot script and type in the encryption password to unlock the directories and start the mail daemon. Any incoming mail should remain queued by the sending system until the mail daemon is up again.

Keeping the communication secure is an ongoing effort, as various protocols are strengthened and deprecated in response to ever more clever attacks. The mozilla ssl-config generator is an ideal first step for any configuration. Tools like testssl.sh and sites like immuniweb.com are also useful for periodically scanning for new vulnerabilities. The ssh-audit script looks for weaknesses in the SSH setup, this ssllabs test looks at https, and Lynis test many different aspects of a system.

Certbot / LetsEncrypt

ROLE certbot

We use certbot to acquire LetsEncrypt SSL certificates. We request one certificate for each web domain and one for the mail server, to simplify moving web or mail across machines as needed. Each certificate matches any aliases for that domain (www., etc). We use the http challenge and run certbot standalone, manually starting and stopping any web server.

The /etc/letsencrypt directory holds a history of all past keys under the archive directory and symlinks to the current one under the live directory. These are mode 700 so only apps that run as root can see them. This works for apache, postfix and dovecot, but if a non-root app needs access, you might want to create a ssl-cert group and change it to mode 750.

Below is an example of certs for the example.com and foobar.com domains. When archiving the letsencrypt directory, it is important to use a method that preserves the symlinks in the live directory or certbot may not renew properly. Bacula preserves the links and the snapshot tag makes a compressed tar file of this directory.

root@localhost:/etc/letsencrypt# ls -l
total 40
drwxr-xr-x 3 root root 4096 Dec  2 23:08 accounts
drwx------ 4 root root 4096 Dec  2 23:08 archive
-rw-r--r-- 1 root root  121 May 26  2018 cli.ini
drwxr-xr-x 2 root root 4096 Dec  2 23:08 csr
drwx------ 2 root root 4096 Dec  2 23:08 keys
drwx------ 4 root root 4096 Dec  2 23:08 live
drwxr-xr-x 2 root root 4096 Dec  2 23:08 renewal
drwxr-xr-x 5 root root 4096 Dec  2 23:08 renewal-hooks
-rwxr-xr-x 1 root root 1923 Dec  2 19:42 testcertnames

root@localhost:/etc/letsencrypt# ls -l live/*
live/example.com:
total 4
lrwxrwxrwx 1 root root  37 Dec  2 19:42 cert.pem -> ../../archive/example.com/cert1.pem
lrwxrwxrwx 1 root root  38 Dec  2 19:42 chain.pem -> ../../archive/example.com/chain1.pem
lrwxrwxrwx 1 root root  42 Dec  2 19:42 fullchain.pem -> ../../archive/example.com/fullchain1.pem
lrwxrwxrwx 1 root root  40 Dec  2 19:42 privkey.pem -> ../../archive/example.com /privkey1.pem
-rw-r--r-- 1 root root 692 Dec  2 19:42 README

live/foobar.com:
total 4
lrwxrwxrwx 1 root root  39 Dec  2 19:42 cert.pem -> ../../archive/foobar.com/cert1.pem
lrwxrwxrwx 1 root root  40 Dec  2 19:42 chain.pem -> ../../archive/foobar.com/chain1.pem
lrwxrwxrwx 1 root root  44 Dec  2 19:42 fullchain.pem -> ../../archive/foobar.com/fullchain1.pem
lrwxrwxrwx 1 root root  42 Dec  2 19:42 privkey.pem -> ../../archive/foobar.com/privkey1.pem
-rw-r--r-- 1 root root 692 Dec  2 19:42 README

Certificate renewal is done by cron and any firewall or web server handling is done by small shell scripts in the pre and post directories under renewal-hooks. Services that need to bounce when certificates change are reloaded by scripts in the deploy directory.

Certbot Variables

The certbot_certs variable contains a list of records describing the certificates that we should request. Each record includes a list of domains and aliases and an optional email. By default, this includes records for each website plus any extras.

certbot_certs: '{{ certbot_vhost_certs + certbot_extra_certs }}'

An individual record looks like:

 - domains: [ maindom, aliasdom1, aliasdom2 ]
   email: 'addr' (optional, default admin_email)

The certbot_vhost_certs is built from the name and aliases fields in each web_vhost entry. The certbot_mail_certs variable has one record for mail_server_hostname. To have separate certs for your mail server and webservers, just set the extra certs variable as follows.

certbot_extra_certs: '{{ certbot_mail_certs }}'

We run certbot standalone and use renewal hooks to temporarily take control of the web ports. The certbot_service_stop variable lists any web server to stop while renewing. If the machine normally blocks the http/https ports, setting certbot_open_firewall to true adds a temporary iptables rule to allow them.

The certbot_service_reload list contains services that need to have 'reload' called on them if the renewal process changes any certificates.

# Set to yes only if web ports are normally blocked
# certbot_open_firewall: no

# stop/start these during the challenge response period
certbot_service_stop:
  - apache2

# also reload these when we get a new cert.
certbot_service_reload:
  - postfix
  - dovecot

Disk Encryption

ROLE cryptdir

We create one directory for all data encrypted at rest. The encryption technology can be LUKS, EncFS, or none. With none, this role does no encryption.

We keep the mail spool, mail user database, DKIM keys, and sieve files in this directory. The mail queue is in there when using LUKS (EncFS probably won't work with the sockets in the postfix chroot). Web data is not there because that is public information and we want the web server to come up at boot.

At boot, everything starts normally except mail, which waits until we log in and unlock the encrypted directory with the mailboot script. Any incoming mail should remain queued by the sending system until the mail daemon is up again. The backup server can use the same technique to encrypt the bacula pools. See backup_server.yml in the group settings for more details.

LUKS encrypts a separate disk partition at the block level, then mounts it as a cleartext block device through the mapper. The cleartext device contains a plain Ext4 filesystem and is mounted normally. Some LUKS notes.

With EncFS, a separate directory contains individually encrypted files and is mounted loopback as cleartext through FUSE. EncFS can be used anywhere, but it is rather outdated, as is eCryptfs. Useful if you are not in a position to allocate a separate partition.

EncFS works at the file level, so it is not entirely transparent. When using it for the mail spool, Dovecot had problems with "Error: Couldn't create mailbox list lock file_create_locked() failed". Changing the lock_method to "dotlock" instead of the default (fcntl) fixed the issue, but block-level encryption avoids this sort of thing.

Disk Encryption Variables

You must set the password, usually through the vault_cryptdir_pw value in the hosts.example. Set cryptdir_type to 'encfs', 'luks', or 'none' to change the technology. The default is EncFS and does not need any other settings.

# EncFS encryption, no other settings needed.
# Data in /vault, ciphertext version in /vault_crypt

With LUKS, you must set cryptdir_luks_dev to the partition for your encrypted data. LUKS encryption completely overwrites that partition, so make sure that you don't have anything important there!

# LUKS encryption, must specify partition
# Data in /vault
# Cleartext block device /dev/mapper/vault-blk
# Ciphertext block device /dev/replace-with-real-device

cryptdir_type: luks
cryptdir_luks_dev: "/dev/replace-with-real-device"

You can turn off encryption if you have reasons.

# None, /vault unencrypted
cryptdir_type: none

The name of the central directory is given by cryptdir_root. This defaults to /vault. If you want to use this for a different machine, you can customize the mounting script that it generates with the boot script and boot services variables. The settings below are what the mailhost role uses.

# mailboot script generated by the cryptdir role
cryptdir_boot_script: "{{mail_cfg_dir}}/mailboot"
cryptdir_boot_services:
  - postfix
  - dovecot
  - redis-server
  - rspamd

Why These Packages?

Any cert can enable encrypted communication, even a self-signed one, but only a cert from a trusted certificate authority (CA) makes a statement about your identity. LetsEncrypt is the baseline that states that you control a particular domain. Other paid options can be used if you need a stronger declaration of identity. For simple HTTPS and IMAPS, LetsEncrypt is fine.

A virtual private server (VPS) is convenient but never as private as your own machine. The cloud company has full access to the hardware and disk resources, and with physical access, a whole spectrum of attacks become possible. Even then, as Ken Thompson pointed out in Reflections On Trusting Trust, unless you wrote the code yourself, on hardware that you built yourself, there is always room for mischief.

We encrypt the mail spool and other private things, which protects against a rogue employee in the data center, but probably not a nation state.

The encryption method depends on your situation. If you can create partitions, use LUKS, otherwise EncFS. If you have physical control of your server, you might choose 'none'. Another possibility is per-directory ext4 encryption with fscrypt. I don't have much experience with that, but it might do a better job than EncFS without needing a separate partition.