Published: Jun 08, 2024

Self Hosted Mailserver

Introduction

The following post will outline the requirements and steps to setup a fully self hosted mailserver that should pass all requirements to send mail to Gmail or Outlook users.

While the underlying protocol SMTP is very simple a lot more is required for you mail to not be rejected by various spam filters. The worst case would be setting up an open relay which is quickly going to be discovered and abused by bots. Not long after that your IP address and domain name will be listed on various “spam lists”. Once on those lists it can be very difficult to get off them again and might lead to a domain/server becoming unusable for mail.

To avoid ending up on those lists setting up SPF, DKIM, DMARC, rDNS and DNS before starting to send out mail is essential. Also you should avoid sending many mails in a short timeframe. The GMail mailserver for example employs various rate limits that slowly increase as your server becomes more “trustworthy” while Outlook will put you on their “Anti Spam List” immediately and you have to manually request to be delisted at https://sender.office.com/ .

While all of this sounds (and honestly is) quite intimidating it is also somewhat rewarding to setup your own mailserver and have full control over it. For me the main reasons to go though this were just curiosity and a desire for a more decentralized internet.

Anyways, time to get started!

Port 25

Port 25 is used for unencrypted SMTP communication and has high potential for abuse and spam and is often blocked by default by ISPs and server hosters. If this is your fist time setting up a mailserver this might cause unexpected errors so make sure to check if you are able to use the port as there is no way to setup a proper mailserver without it.

Preparation

DNS

Head over to your domain registrar and login to the webinterface to manage your DNS records. First you want to setup an A Record for your mailserver such as mail.egonr.dev that points to your mailserver. Don’t worry, your mails can still be received and sent from the “bare” domain (for example …@egonr.dev).

rDNS

rDNS enables finding a domain name from an IP address, therefore the name “reverse DNS”. Usually this can be configured in the webinterface of your server hoster. Enter your mailserver domain such as mail.egonr.dev .

SPF uses FCrDNS (full-circle reverse DNS) as a weak form of authentication that there is a relationship between the owner of a domain name and the owner of an IP address. Sometimes this is also called “double-reverse DNS”.

SPF

For SPF add a TXT record with the host set to @ and the following content:

v=spf1 a:egonr.dev -all

This record indicates SPF version 1, an allowed server “egonr.dev” and to mark all other mails claiming to be from egonr.dev but originating from a different IP as spam or rejected. This is a very restrictive configuration, if you need more flexibility there are many options to customize SPF records.

DKIM

DKIM uses asymmetric encryption to sign emails so the recipient knows that the message has not been faked or altered in transit. The public key is published using a TXT record and can be looked up by the receiving SMTP server.

Start by generating a keypair using openssl:

openssl genrsa -out private.key 2048
openssl rsa -in private.key -pubout -out public.key

As always, remember to never share your private key and only allow minimal access (like 600 or 640).

To configure the TXT records for DKIM you need to choose a selector, a popular choice is the date of creation of the key. For example set the host to 20210101._domainkey where 20210101 represents the selector:

v=DKIM1;t=y;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA5XLfZHsP7RdENdgNAT2q8gNE8mrSB/9bf0yuHBZmy6QyUfnjKaDHxtyL4DxQaw/2TnztH7dHf0QDQ9AUEhokFPpymzSkQZjAwn571PHufnY6nopIyfL1edTolFlrdV+nAue/46tDM2lvC1HyIRQfPpmccjYVVqI/sU3QZCD60B3r7uSaSjRxC+NJAEc7bt9ckXcnbE8Vmk5bUzUIrOd4hv1P9pygBnoIEAK/0/oPZq7p+hNgzrE0MxjdoHYkor4gBjN2e+Dy7ednJRbCK3ljDFLiKxgZeeWb/EHgsNpzeEIOgHVFSkcp3pTg/d+ZBkNw27CeSLqlnIJfYfRv8uaAnwIDAQAB

Note the t=y - this indicates that DKIM is setup in test mode. Most providers will respect this setting and not penalize your mails if DKIM fails. Remember to either remove t=y or set t=s in production for proper DKIM enforcement.

Some domain providers have rather short limits for the value of a TXT record such as 255 characters. In this case you have to split your DKIM record into multiple TXT records.

DMARC

For DMARC add a TXT record with the host set to _dmarc:

v=DMARC1; p=reject; adkim=s; aspf=s; rua=mailto:contact@egonr.dev

This record indicated DMARC version 1 with a policy to reject all mails failing SPF or DKIM checks which are performed in strict (s) mode. Reports of failed mails and spoofing attempts are sent to the address specified after ‘rua’.

TLS

For TLS I am using the free service provided by LetsEncrypt. To request the necessary certificates just run: certbot certonly —standalone -d mail.egonr.dev

This will place the certificates in /etc/letsencrypt/live/mail.egonr.dev. Most cerbot installations come with auto renew enabled by default via /etc/cron.d/certbot .

Verification

Before continuing make sure to double check your settings using some tools. At the time of writing I found mxtoolbox to be very useful.

Software

Now it’s time to setup the actual mailserver software. I wanted to start with a minimal setup that runs in a docker container which I can extend as needed. If you want a more feature rich solution mailcow or mail-in-a-box seem to be good alternatives. In this post I will be using docker-mailserver which is rather easy to configure and run.

Many of these mailserver packages share the same core software such as:

  • Postfix as MTA (Mail Transfer Agent)
  • Dovecot as IMAP/POP3 Server
  • OpenDKIM
  • SpamAssassin
  • ClamAV

Docker Compose

First get the mailserver.env from the docker-mailserver repository. For my usecase the defaults were mostly fine. The settings I changed are:

  • LOG_LEVEL=info -> LOG_LEVEL=debug
  • ENABLE_AMAVIS=1 -> ENABLE_AMAVIS=0
  • SSL_TYPE=letsencrypt

Now create docker compose file:

services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    container_name: mailserver
    # Provide the FQDN of your mail server here (Your DNS MX record should point to this value)
    hostname: mail.egonr.dev
    env_file: mailserver.env
    # More information about the mail-server ports:
    # https://docker-mailserver.github.io/docker-mailserver/latest/config/security/understanding-the-ports/
    ports:
      - "127.0.0.1:25:25"    # SMTP  (explicit TLS => STARTTLS, Authentication is DISABLED => use port 465/587 instead)
      - "127.0.0.1:143:143"  # IMAP4 (explicit TLS => STARTTLS)
      - "127.0.0.1:465:465"  # ESMTP (implicit TLS)
      - "127.0.0.1:587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "127.0.0.1:993:993"  # IMAP4 (implicit TLS)
    volumes:
      - ./docker-data/dms/mail-data/:/var/mail/
      - ./docker-data/dms/mail-state/:/var/mail-state/
      - ./docker-data/dms/mail-logs/:/var/log/mail/
      - ./docker-data/dms/config/:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
    restart: always
    stop_grace_period: 1m
    # Uncomment if using `ENABLE_FAIL2BAN=1`:
    # cap_add:
    #   - NET_ADMIN
    healthcheck:
      test: "ss --listening --tcp | grep -P 'LISTEN.+:smtp' || exit 1"
      timeout: 3s
      retries: 0

And start your mailserver: docker compose up

First you want to setup a user, for example the “root” or “postmaster” of the domain. To do that docker-mailserver provides a handy “setup” script. All you need to do is call “setup email add” from inside the container.

I also recommend having a look at the documentation and get familiar with the configuration options docker-mailserver

Now you should be able to connect to your mailserver locally and see if there are any obvious errors. If everything seems to be running correctly you can remove 127.0.0.1 from the ports and expose your mailserver to the internet. You will likely notice various bots probing your server after just a few seconds or minutes.

Bonus - Webmail

I like to be able to access my mails from any device that has a browser, if you want terminal access I can recommend mutt or aerc. Anyways, for the webmail solution I settled on roundcube since it was easy to setup and didn’t use a lot of resources. The compose file looks like:

services:
  roundcubemail:
    image: roundcube/roundcubemail:latest
    hostname: roundcube.egonr.dev
    container_name: roundcubemail
#    restart: unless-stopped
    volumes:
      - ./www:/var/www/html
      - ./db/sqlite:/var/roundcube/db
    ports:
      - "1234:80"
    environment:
      - ROUNDCUBEMAIL_DB_TYPE=sqlite
      - ROUNDCUBEMAIL_SKIN=elastic
      - ROUNDCUBEMAIL_DEFAULT_HOST=tls://mail.egonr.dev
      - ROUNDCUBEMAIL_SMTP_SERVER=tls://mail.egonr.dev

Another consideration was SoGo, but it ended up having more features than I needed and wanted, also there don’t seem to be actively maintained docker images except the one included in mailcow.

Final Test

To test your configuration I found mail-tester to be excellent. But be aware that there is a daily limit of a few mails, so only send a mail once you are somewhat sure your configuration is correct. Before that I would recommend just sending mail to one of your existing addresses.

Also you might have to wait a few hours or a day until the DNS records are actually active.

If you did everything correctly and your server is not already on some spam lists you should achieve a 10/10 score. If find yourself listed on a spam list you should try to contact them to get delisted. If it’s the first time it shouldn’t be too difficult - sometimes it’s as simple as entering your IP address and solving a captcha.