$_ BSDHowTo.ch
How To... Why not..? Scripts Patches RSS logo

How to configure a small mail server

Last update: 2020-09-27


Yes, another post about setting up a mail server. I know, there are plenty similar posts already out there… This one is about setting up a mail server on an Internet facing host. It will accept and send mails for a domain, store the accepted mails locally and deliver them using POP3. This is a rather lengthy post because there are quite some pieces to put together.

Building blocks

I use the following software to build my mail server out in the wild Internet:

OpenSMTPD will handle incoming and outgoing mail using SMTP. rspamd will support it in fighting incoming spam and malware (using ClamAV), and signing outgoing mail using DKIM. Dovecot stores received mail for users and deliver it using POP3. Finally acme-client(8) is used to manage the certificate from Let's Encrypt.

Basic assumptions

The configuration I describe in this post is based on some assumptions about the server and its environment:

This setup is suited for a small domain providing mail services to few users. User management is done manually and mail is stored locally. If this doesn't fit your needs you might consider using one of the many other mail server guides out there.


First you should install the required software packages:

$ doas pkg_add -i dovecot opensmtpd-extras rspamd opensmtpd-filter-rspamd

Next you create the system user which will be used for handling mails:

$ doas useradd -c "Virtual Mail" -d /home/vmail -g =uid -s $(which nologin) vmail
$ doas mkdir /home/vmail
$ doas chown vmail:vmail /home/vmail
$ doas chmod 0750 /home/vmail

And you create the passwd(5) which will hold the information about the mail users on your system:

$ smtpctl encrypt 1amApASSw0rd | set "s/^/muser:/;s/$/::::::/" > passwd
$ doas mv passwd /etc/mail/passwd
$ doas chown _dovecot:_smtpd /etc/mail/passwd
$ doas chmod 0440 /etc/mail/passwd

If you want to use DKIM to sign your outgoing mail it is time to create the key for it:

$ doas mkdir /etc/mail/dkim
$ doas openssl genrsa -out /etc/mail/dkim/example.net.key 1024
$ doas openssl rsa -in /etc/mail/dkim/example.net.key -pubout \
> -out /etc/mail/dkim/example.net.pub
$ doas chgrp _rspamd /etc/mail/dkim/example.net.key
$ doas chmod 0640 /etc/mail/dkim/example.net.key

There are some entries required in the DNS zone of your domain. The following command will generate a text file ready to import into the zone file:

$ echo '        MX  10 mail.example.net.' > rrs.txt
$ echo '        TXT "v=spf1 mx -all" >> rrs.txt
$ echo '_dmarc  TXT "v=DMARC1;p=none;pct=100;rua=mailto:postmaster@example.net"' >> rrs.txt
$ echo 'mail    A' >> rrs.txt
$ echo '        AAAA    2001:db8::c000:020a' >> rrs.txt
$ echo 'pop3    CNAME   mail' >> rrs.txt
$ echo '$ORIGIN _domainkey.example.net.' >> rrs.txt
$ pubkey=$(sed /^-/d /etc/mail/dkim/example.net.pub | tr -d '\n')
$ echo "default TXT \"v=DKIM1;k=rsa;p=${pubkey}\"" >> rrs.txt

Certificates from Let's Encrypt

Of course you can use any certificate provider who supports the ACME protocol. I use Let's Encrypt because they provide certificates for free, which is a huge win if you run a small site like this one.

You will use httpd(8) to answer the challenges. Create a /etc/httpd.conf similar to this one:

server "mail.example.net" {
    listen on egress port http
    alias "pop3.example.net"
    root "/"
    location "/.well-known/acme-challenge/*" {
        request strip 2
        root "/acme"

types {
    include "/usr/share/misc/mime.types"

Test your configuration, enable and start httpd(8):

$ doas httpd -n
$ doas rcctl enable httpd
$ doas rcctl start httpd

Now acmme-client(8) needs to know what to do and with whom. Take /etc/examples/acme-client.conf, adapt it to your needs and save the result as /etc/acme-client.conf:

authority letsencrypt {
    api url "https://acme-v02.api.letsencrypt.org/directory"
    account key "/etc/acme/letsencrypt-privkey.pem"

authority letsencrypt-staging {
    api url "https://acme-staging-v02.api.letsencrypt.org/directory"
    account key "/etc/acme/letsencrypt-staging-privkey.pem"

domain mail.example.net {
    alternative names {
    domain key "/etc/ssl/private/mail.example.net.key"
    domain certificate "/etc/ssl/mail.example.net.crt"
    domain full chain certificate "/etc/ssl/mail.example.net.fullchain.pem"
    sign with letsencrypt

This config will issue a valid certificate right away. If you feel like testing in the first place, you should change the line sign with to letsencrypt-staging until you feel comfortable with the process.

Before you can get your certificate you must make sure pf(4) lets the requests to httpd(8) actually pass through. Add rule similar to the following to your pf.conf(5):

pass in log on egress proto tcp from any to egress port http

After adding this rule to /etc/pf.conf check the file and load it into pf(4) with the following commands:

$ doas pfctl -nf /etc/pf.conf
$ doas pfctl -f /etc/pf.conf

Now you can get your certificate using the following command:

$ doas acme-client mail.example.net

Certificates have an expiry date, like groceries. You may want to make sure your certificate gets renewed automatically before it expires. The file /etc/daily.local can take care of this for you:


/usr/sbin/acme-client mail.example.net
[[ $? -eq 0 ]] && rcctl restart smtpd dovecot

Anti-malware shield ClamAV

ClamAV will be used by rspamd to scan attachments for malware. To do so you need some configuration for ClamAV first in order to run it as a daemon and to keep the malware database up to date. Or if you run a malware scan server in your environment you can connect rspamd to it.

First you configure the daemon freshclam to make sure the malware database of ClamAV stays up to date. The file /etc/freshclam.conf contains the following settings:

LogTime yes
LogSyslog yes
LogFacility LOG_DAEMON
DatabaseMirror db.ch.clamav.net
DatabaseMirror database.clamav.net
NotifyClamd /etc/clamd.conf

Enable and start freshclam now so it has time to update the signature database for ClamAV:

$ doas rcctl enable freshclam
$ doas rcctl start freshclam

Next you configure clamd. In /etc/clamd.conf the following lines are set:

LogTime yes
LogSyslog yes
LogFacility LOG_DAEMON
TemporaryDirectory /tmp
LocalSocket /var/clamav/clamd.sock
TCPSocket 3310
User _clamav
DetectPUA yes
AlertEncrypted yes
AlertEncryptedArchive yes
AlertEncryptedDoc yes
AlertOLE2Macros yes
AlertPhishingSSLMismatch yes
AlertPhishingCloak yes
MaxRecursion 12

You may want to study the man page of clamd.conf and consider each of the options named Alert*. Some of these may block attachments you actually don't want to get blocked on your mail server. As soon as you are happy with your configuration it is time to enable and start clamd:

$ doas rcctl enable clamd
$ doas rcctl start clamd

You may get a timeout warning after the start command. Using pgrep(1) you can check if clamd is actually running or not. In most cases it will be running and you can ignore the timeout message.

Santas little helper rspamd

rspamd will cover all the extra needs we have today when running a mail server out in the wild. It provides a highly customizable and trainable spam filter, malware filter, greylisting and DKIM signing.

While most modules work out of the box the DKIM signing needs configuration in order to find the key it should use to sign mails. Create the /etc/rspamd/local.d/dkim_signing.conf containing this:

allow_username_mismatch = true;

domain {
    example.net {
        path = "/etc/mail/dkim/example.net.key";
        selector = "default";

In case you use ClamAV for malware scanning this module of rspamd needs some configuration too. Create the file /etc/rspamd/local.d/antivirus.conf with the following content:

clamav {
    action = "reject";
    message = '${SCANNER}: virus found: "${VIRUS}"';
    scan_mime_parts = true;
    scan_image_mime = false;
    symbol = "CLAM_VIRUS";
    type = "clamav";
    prefix = "rs_cl_";
    servers = "/var/clamav/clamd.sock";
    whitelist = "${DBDIR}/wl_antivirus.map.local";

These settings are very strict by rejecting every mail that scans positive for malware. Depending on your needs you may want to reconsider this and add a high score to the mail instead. This gives you the chance to put into quarantine instead of blocking it completely.

If rspamd keeps misclassifying mails from particular domains you may want to improve the score of those mails by whitelisting these. Create the file /etc/rspamd/local.d/multimap.conf:

    type = "from";
    filter = "email:domain";
    map = "${DBDIR}/wl_sender_domain.map.local";
    score = -5.0;

It is time to enable and start rspamd and its memory storage Redis:

$ doas rcctl enable redis rspamd
$ doas rcctl start redis rspamd

Dovecot as POP3 server

First, follow the pkg-readme of Dovecot and create a own login class for it in /etc/login.conf:


I recommend to you to put the actual configuration of Dovecot into /etc/dovecot/local.conf and leave all the other config files alone (with one exception further down). This way updates won't destroy your configuration. For a POP3-only configuration the file should look similar to this one:

listen =, 2001:db8::c000:020a
local_greeting = "%s.flueckiger.lan ready"
mail_home = /home/vmail/%d
mail_location = mbox:~/%n
protocols = pop3
ssl = yes
ssl_cert = </etc/ssl/mail.bsdhowto.ch.fullchain.pem
ssl_key = </etc/ssl/private/mail.bsdhowto.ch.key
ssl_dh = </etc/ssl/dh4096.pem
ssl_min_protocol = TLSv1.2
ssl_cipher_list = ALL:!DH:!kRSA:!SRP:!kDHd:!DSS:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK:!RC4:!ADH:!LOW@STRENGTH
ssl_prefer_server_ciphers = yes

passdb {
    driver = passwd-file
    args = scheme=blf-crypt /etc/mail/passwd

service auth {
    unix_listener auth-userdb {
        mode = 0600
        user = vmail

service pop3-login {
    inet_listener pop3 {
        address =, ::1

service stats {
    unix_listener stats-writer {
        user = vmail
        group = _dovecot

userdb {
    driver = static
    args = uid=vmail gid=vmail home=/home/vmail/%d

The configuration makes POP3 available on the public IP addresses using TCP port 995 which requires an encrypted connection right from the start. This makes it hard for users to send clear text passwords over an clear text connection. For debugging purposes there is a clear text connection available on localhost TCP port 110.

Due to a bug in the config parser of Dovecot you must comment out two lines in /etc/dovecot/conf.d/10-ssl.conf or the loading of the certificate and key files will fail:

$ doas sed -i "/^ssl_[cert|key]/s/^/#/" /etc/dovecot/conf.d/10-ssl.conf

Check your configuration so far by testing the login for mail users with the following commands:

$ doas doveadm user muser@example.net
$ doas doveadm auth login muser@example.net

The first command should give you information about the account muser while the second one should check if the password you've set for muser is correct.

OpenSMTPD as mail transport agent

Your server probably got another hostname, so make sure OpenSMTPD always identifies with the right name:

# echo "mail.example.net" > /etc/mail/mailname

Next, you want to make sure that OpenSMTPD knows about the valid recipient addresses on the system and which of the default addresses get redirected to whom. You define such a table(5) in /etc/mail/virtuals:

abuse@example.net:      muser@example.net
hostmaster@example.net: muser@example.net
postmaster@example.net: muser@example.net
muser@example.net:      vmail

Each valid mailbox on the left side either gets redirected to another valid mailbox defined in this file or to the system user that handles mails for us.

There is another table called trusted. You can put IP addresses in it of hosts that you trust although both DNS checks fail for these. For the below example configuration to work you need to create the file at least:

$ doas touch /etc/mail/trusted

If you do want to redirect local mails into one of the mailboxes of the domain you should add a line similar to the following one to /etc/mail/aliases:

user:   muser@example.net

Up next is the actual configuration of OpenSMTPD. I suggest you start out with a fresh /etc/mail/smtpd.conf - either put away the original or clear it first.


table aliases file:/etc/mail/aliases
table passwd passwd:/etc/mail/passwd
table trusted file:/etc/mail/trusted
table virtuals file:/etc/mail/virtuals

smtp ciphers "TLSv1.3:TLSv1.2:!NULL"
smtp max-message-size "5M"

pki $name cert "/etc/ssl/mail.example.net.fullchain.pem"
pki $name key "/etc/ssl/private/mail.example.net.key"

filter trusted phase connect match src <trusted> bypass
filter no_rdns phase connect match !rdns disconnect \
    "550 rDNS is required around here"
filter no_fcrdns phase connect match !fcrdns disconnect \
    "550 FCrDNS is required around here"
filter rspamd proc-exec "filter-rspamd"
filter checks chain { trusted, no_rdns, no_fcrdns, rspamd }

listen on lo0
listen on $name tls pki $name filter checks
listen on $name smtps pki $name filter checks
listen on $name port submission tls-require pki $name \
    auth <passwd> filter rspamd

action "local" mbox alias <aliases>
action "deliver" mda "/usr/local/libexec/dovecot/dovecot-lda -f %{sender} -d %{dest}" \
    virtual <virtuals> user vmail
action "outbound" relay

match from local for local action "local"
match from any for domain "example.net" action "deliver"
match from local for any action "outbound"
match auth from any for any action "outbound"

Check your configuration and restart smtpd(8):

$ doas smtpd -n
$ doas rcctl restart smtpd

Firewall rules to go live

So far pf(4) is blocking access to the ports of the daemons you have configured and started. If you are confident that your setup is OK it is time to go live with the services. Add the following rules to /etc/pf.conf:

mail="{ smtp smtps submission pop3s }"

table <bruteforce> persist file "/etc/pf.bruteforce"

block drop in log quick on egress from <bruteforce> to any

pass in log on egress proto tcp from any to egress port $mail \
    (max-src-conn 5, max-src-conn-rate 1/1, overload <bruteforce> flush)

These rules allow traffic to pass to the TCP ports you have configured for OpenSMTPD and Dovecot. Packets from misbehaving clients get dropped silently. Misbehaving is defined as opening more than five connections from the same source IP or opening connections faster that one per second.

After adding these rules to /etc/pf.conf check the file and load it into pf(4) with the following commands:

$ doas pfctl -nf /etc/pf.conf
$ doas pfctl -f /etc/pf.conf

Teaching rspamd some lessons

No matter how good a spam detection system is, you will have both false positives (messages that are actually not spam) and false negatives (spam messages that are not detected as such). Lucky for you rspamd comes with a web interface to monitor and tune it. By default this web interface is accessible without any kind of authentication. Therefore it is only listening on localhost. The easiest way to access it is using port forwarding of ssh(1). Adding a line like this to ~/.ssh/config should do the trick:

Host mail.example.net
    LocalForward 11334

Whenever you are logged in to mail.example.net using ssh(1) you can access the rspamd web interface in your browser with this link: http://localhost:11334/