Setting up a mail server on your own is hard and you may consider going for some of the easier alternatives. Why you should consider not to do it yourself is argued elsewhere. In this post, I will try to give you an overview of what it takes to do it yourself and help you build a mental model of the solution based on Postfix and Dovecot. I will also cover other security and reputation related concerns which are very important.
SMTP
Emails are moving across the internet (mostly) using SMTP (Simple Mail Transfer Protocol). SMTP is the protocol for sending and receiving emails. It is text based protocol between SMTP Client and SMTP Server. During one SMTP session, client sends pieces of information (To, Subject, Attachment, Body etc.) to server and server responds with some status messages until the client sends a special character (single dot on a line) saying that it is done and at that moment session gets terminated. In order to send and receive emails we need to act both as a SMTP Client and as SMTP Server. Fortunately, the single piece of software supports this: postfix.
Postfix and Dovecot
Postfix is one of the most popular mail (SMTP) servers for Linux. It allows email submission, email sending and email receiving.
- Email submission is a process when user of mail server wants to send an email outside. In this flow, user creates an email using the email client (also called MUA (Mail User Agent)) of choice (Mozilla Thunderbird, Opera Mail, Mutt etc.). When user is ready and presses Send button, SMTP session (MUA being the SMPT client and Postfix being the SMTP server) is opened against Posfix submission port (587) and email is accepted by Postfix.
- Email sending is a process of sending submitted emails. Once email is accepted, Postfix tries to forward it to destination. For this reason, it takes destination address(es) and performs DNS lookup on destination domain and reads MX record (the one specifying the location of destination mail server). Then, Postfix opens SMPT session (now SMTP Client is Postfix and SMTP Server is the destination mail server (could be Postfix too)) to destination mail server delivery port (25) and delivers email.
- Email receiving is a process of accepting emails from remote mail servers and storing them to local user’s mailbox. In this scenario, somebody wants to deliver email to one of the users of our mail server. For them to find your mail server, you need to set MX record for your domain properly. Postfix is listening on port 25 and accepts SMTP sessions originating from across the globe. In this case, Postfix acts as a SMTP Server and SMTP Client is the originating mail server. Once email is accepted it is stored in local user’s mailbox (if Maildir format is used then email is stored as a file in user’s home directory).
Dovecot is a popular IMAP and POP3 server for Linux. It bridges the last part of email flow: it delivers emails from user’s mailboxes (on mail server) to email client. Protocol for this is not SMTP but IMAP or POP3. Dovecot also can implement authentication module, based on SASL, which can be used by Dovecot to authorize access to mailboxes as well as by Postfix to authorize submission of emails and allow only authenticated users to send email (thus preventing Postfix instance to become Open Relay).
When setting up users it is possible to use Linux users (/etc/passwd) or to set up a virtual users. For very small setup the former one should be quite OK, but if you need to have large number of email accounts, then you are probably better going with the later approach. Since my use case is quite simple, I went for the first option.
Dovecot can provide SASL authentication module which integrates with Linux users (through PAM (pluggable authentication module)). It can also offer this as a service (through unix socket) to Postfix to help it with authorizing email submissions.
Let’s build a high level mental model of the main data flows:
There are many tutorials on how to configure each piece and this is not trivial, but having this high level picture is certainly useful as you plumb the pipes at the lower level.
Mail spoofing protection
Having set up Postfix and Dovecot and properly configuring email client makes it possible to send and receive emails. However, there are more things to do in order to increase your chances to really deliver emails all the way to the end user’s Inbox. Emailing world is, unfortunately, full of spam and as prevention there are many spam filter which could sweep your message away before it ends in end user’s Inbox. Furthermore, somebody else could try to forge your originating address and pretend that it is you. In order to fight spam and bad guys there are some additional measures you should take.
SPF
Sender Policy Framework (SPF) specifies which servers could send email for specific domain. It should be published as DNS record of TXT type. Here is example of this value as it is set up for my doamin:
“v=spf1 a include:servers.mcsv.net -all”
It say that mails sent from @nikola-petkovic.com address(es) will come from server indicated by “A” DNS record (the main record which is used for domain name to IP mapping) or by MailChimp servers. So basically I allow my own mail server and MailChimp servers to send emails from this domain. If somebody else tries to do it, then their SPF check will fail and they are less likely to have their email delivered.
Setting up above DNS resource record helps others understand if somebody is spoofing one of our email addresses. However, to detect spoofed incoming emails, we need to implement SPF check in Postfix as a part of accepting emails from foreign mail server and decide what to do in that case.
DKIM
DomainKeys Identified Mail (DKIM) is another mechanism serving similar purpose as SPF, but it is more concerned with the integrity of emails. It relies on Public Key Infrastructure (PKI) to sign outgoing messages with a private key. Public key should be published in a DNS resource record of TXT type. For example:
Key: mail._domainkey.nikola-petkovic.com
Value: “v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNAD…”
The mail server receiving message will read this record and validate whether email was properly signed with matching private key. This ensures integrity of the message, so it cannot be forged along the way.
In order to validate DKIM signature on incoming emails it is needed to implement DKIM check in Postfix as a part of accepting emails. It is also possible to decide what to do in case it is not properly signed.
DKIM signing and check is not as straightforward as SPF check. OpenDKIM is open source software which you can install and plug it into Postfix. It can be configured to intercept outgoing emails and sign them (encrypt the hash of email content with private key and add it in email header). It can also be configured to intercept incoming emails and validate their DKIM signature (decrypt DKIM signature with other’s side public key and compare it with the hash of the message content).
DMARC
Domain-based Message Authentication, Reporting and Conformance (DMARC) is third mechanism which builds on top of previous two (SPF and DKIM). It gives a hint to receiving mail server what to do in case email doesn’t pass SPF and/or DKIM check. It also instructs the receiver to send a periodic report on messages which are sent to receiver from your domain. It gives you, as a domain owner, a feedback information on what is happening with the emails you are sending.
Again, it is set up as DNS resource record of TXT type which could look like this:
Key: _dmarc.nikola-petkovic.com
Value: “v=DMARC1; p=quarantine; rua=mailto:postmaster2@nikola-petkovic.com; ruf=mailto:postmaster2@nikola-petkovic.com; adkim=r; aspf=s”
This basically instructs receiver to quarantine messages which don’t pass SPF check (aspf=s, meaning strict) and to be more relaxed on DKIM check (adkim=r, meaning relaxed). Also, periodic, consolidated report should be sent back to mail server admin. This report can give you insights, based on which you can decide to change your DMARC rules to improve delivery of genuine emails and reduce delivery of emails with forged From address from bad guys.
Spam filtering
To protect yourself from spammers, it is useful to install some anti-spam software. SpamAssasin is open source software which you can install. You can configure Postfix to use SpamAssasin to intercept incoming (or outgoing) emails and compute spam score on them. Then, based on computed spam score, you can decide whether to completely drop the message or perhaps only add this score to email header. Email client can then use this score when deciding whether to put email in Inbox or Spam/Junk folder. The higher the score, the bigger the chance email is spam. In case there are many users set up on your mail server, it is a wise idea to also check spam score of outgoing emails as they could put your whole server on some blacklist.
Antivirus
Email is quite popular medium for transferring viruses. In order to protect your users, it is wise idea to add antivirus software in your email interceptors list. ClamAV seem to be a decent option here.
Blacklists
There are many blacklists out there and you don’t want your mail server to end up on any of those. They can decrease your reputation and increase the chance that your emails don’t get delivered. To decrease the chance of ending up on one of those, you should install SPF and DKIM and somehow regulate that spam is not sent out of your server (if you have many users, you can, for example, evaluate spam score on outgoing messages and warn users who send spam). In case you send Newsletters ensure there is unsubscribe mechanism. You should prefer unsubscribe to having user classify your emails as a spam, because if many do it, then you could easily end on one of the blacklists. If this is only your own server things are easy but if you are setting it up for many people or offer email account to your users then chance increases that one bad apple will spoil your reputation.
There are many websites (example) where you can check if your domain is blacklisted. Blacklists are managed by some entity, they have a website and you can inform yourself on how to get your domain out of it and how not to end up on it again.
Whitelists
As opposed to blacklists, you want to be on whitelists. Some anti-spam software uses them and being listed there could improve your score. www.dnswl.org seems to be the most important one at the moment and it is recommended to register your domain on their website. Of course you will have to prove that you are really the owner of the registered domain.
Testing email
For further ideas on how to improve your score you can also use service like www.mail-tester.com. They offer you to send email to one of their specific email addresses and then you can see the spam score of that email message as well as get advice on how to improve it.
Conclusion
I hope you have now high level overview of what it takes to run mail server on your own. Don’t expect to finish all setups in one day, there are many moving pieces here. And the devil is in detail, so make sure that you understand and properly configure each component. Furthermore, as any other software, mail servers require ongoing monitoring and maintenance. However, if you take a proper care when setting it up, it should just run and require only a light maintenance. So take your time and do it properly.
