Email is often the first service to fail when DNS is off by one character. It can also break fast when TLS points to the wrong file. This mail server setup tutorial shows a production-ready Postfix + Dovecot setup on an Ubuntu 24.04 VPS. It also covers the DNS records and verification steps you need for reliable delivery in 2026.
By the end, you’ll have SMTP submission on 587, IMAPS on 993, Let’s Encrypt certificates, and SPF/DKIM/DMARC. You’ll also have a short ops checklist to keep the stack stable.
If you’d rather avoid ongoing sysadmin work, a managed VPS hosting plan from HostMyCode (Affordable & Reliable Hosting) keeps patching, monitoring, and incident response off your calendar.
What you’ll build (and what you should decide first)
Before you install anything, define what “one mail server” means for your use case. For most VPS-based setups, you’ll want:
- Postfix for SMTP receive/send.
- Dovecot for IMAP mailbox access (and SASL auth for Postfix).
- Let’s Encrypt certificates for TLS.
- SPF, DKIM, DMARC so major providers trust your mail.
- rDNS (PTR) for your server IP. Missing or mismatched PTR still hurts deliverability.
This tutorial assumes:
- Ubuntu Server 24.04 LTS
- One hostname for the mail server, for example
mail.example.com - One domain you’ll send as, for example
example.com - You control the domain’s DNS zone
Mail server setup tutorial prerequisites: hostname, rDNS, and firewall
Mail delivery depends on basic identity checks. Get these right first. The rest of the setup stays straightforward.
1) Set the server hostname and FQDN
Pick a stable FQDN for the VPS. You’ll match it to DNS in the next step.
sudo hostnamectl set-hostname mail.example.com
hostname -fhostname -f should return mail.example.com.
2) Create DNS records (A/AAAA and MX)
In your DNS provider, add:
- A:
mail.example.com→ your VPS IPv4 - AAAA (optional but recommended):
mail.example.com→ your VPS IPv6 - MX:
example.com→mail.example.compriority 10
If you prefer to keep DNS and hosting under one roof, you can register/transfer domains through HostMyCode Domains and manage changes in one place.
3) Set reverse DNS (PTR) for the sending IP
Set your IP’s PTR record to mail.example.com. Then confirm forward and reverse lookups agree.
HostMyCode has a dedicated guide for this step: Reverse DNS setup guide.
Quick checks:
dig +short mail.example.com A
dig +short -x YOUR.SERVER.IP4) Open only the ports you need
If you use UFW, allow SSH, SMTP, submission, and IMAPS.
You can add port 465 later if you truly need implicit TLS.
sudo ufw allow 22/tcp
sudo ufw allow 25/tcp
sudo ufw allow 587/tcp
sudo ufw allow 993/tcp
sudo ufw enable
sudo ufw statusIf your VPS firewall rules are still wide open, lock them down first using UFW firewall setup tutorial, then return here.
Install Postfix and Dovecot (IMAP + SASL auth)
Ubuntu’s mail packages are well-tested and work cleanly with systemd. Install Postfix, Dovecot, and a few utilities for testing.
sudo apt update
sudo apt install -y postfix dovecot-imapd dovecot-pop3d dovecot-lmtpd \
dovecot-sieve dovecot-managesieved \
libsasl2-modules \
mailutils opensslDuring the Postfix installer prompts:
- Select Internet Site
- System mail name: example.com (your primary sending domain)
Confirm services are running
sudo systemctl status postfix --no-pager
sudo systemctl status dovecot --no-pagerConfigure Postfix for submission (587), TLS, and sane defaults
Postfix works out of the box. However, the default config is too loose for modern client submission.
In this section, you’ll set identity, enable SASL auth, and tighten relay rules.
1) Edit /etc/postfix/main.cf
Open the file:
sudo nano /etc/postfix/main.cfAdd or adjust these settings (replace domains/hostnames):
# Identity
myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
# Listen on all interfaces
inet_interfaces = all
inet_protocols = all
# Accept mail for these destinations
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
# Networks allowed to relay (keep tight)
mynetworks = 127.0.0.0/8 [::1]/128
# Mailbox format (Maildir works well with Dovecot)
home_mailbox = Maildir/
# TLS (we'll point to Let's Encrypt paths later)
smtpd_tls_security_level = may
smtp_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
# Restrict relaying
smtpd_recipient_restrictions =
permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination
# Enable SASL via Dovecot
auth_mechanisms = plain login
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
# Hardening
smtpd_helo_required = yes
disable_vrfy_command = yes
smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
# Message size (example: 25MB)
message_size_limit = 262144002) Enable the submission service in /etc/postfix/master.cf
Open:
sudo nano /etc/postfix/master.cfUncomment (or add) the submission service.
This configuration forces authenticated submission on 587:
submission inet n - y - - smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_client_restrictions=permit_sasl_authenticated,reject
-o smtpd_recipient_restrictions=permit_sasl_authenticated,reject_unauth_destination3) Reload Postfix
sudo systemctl restart postfix
sudo postconf -npostconf -n prints only non-default settings. Save the output somewhere. It speeds up troubleshooting later.
Configure Dovecot for IMAP, Maildir, and Postfix auth
Dovecot does two jobs here. It serves IMAP over TLS. It also exposes an auth socket for Postfix submission.
1) Set Maildir
Edit:
sudo nano /etc/dovecot/conf.d/10-mail.confEnsure:
mail_location = maildir:~/Maildir2) Enable plaintext auth only over TLS
Edit:
sudo nano /etc/dovecot/conf.d/10-auth.confSet:
disable_plaintext_auth = yes
auth_mechanisms = plain login3) Create the Postfix auth socket
Edit:
sudo nano /etc/dovecot/conf.d/10-master.confFind the service auth block and add (or adjust) this stanza:
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0660
user = postfix
group = postfix
}
}4) Enable IMAP and (optional) POP3
Edit:
sudo nano /etc/dovecot/dovecot.confEnsure:
protocols = imapPOP3 is still supported, but it increases exposure. If you don’t have a specific need, leave it disabled.
5) Restart Dovecot
sudo systemctl restart dovecotAdd TLS with Let’s Encrypt (and wire it into Postfix + Dovecot)
Modern clients expect TLS everywhere. Let’s Encrypt keeps certificates current without manual renewals.
1) Install Certbot
sudo apt install -y certbotIf this VPS also runs a site on 80/443, use the appropriate Certbot plugin (nginx/apache) and request the certificate there.
For a mail-only server, standalone mode works as long as port 80 is reachable.
2) Request a certificate
Make sure mail.example.com resolves to this server before running:
sudo certbot certonly --standalone -d mail.example.comCertificates land under:
/etc/letsencrypt/live/mail.example.com/fullchain.pem/etc/letsencrypt/live/mail.example.com/privkey.pem
3) Point Postfix to the cert
Edit /etc/postfix/main.cf and set:
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pemReload:
sudo systemctl reload postfix4) Point Dovecot to the cert
Edit:
sudo nano /etc/dovecot/conf.d/10-ssl.confSet:
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pemRestart:
sudo systemctl restart dovecot5) Verify TLS from your workstation
openssl s_client -connect mail.example.com:587 -starttls smtp -servername mail.example.com
openssl s_client -connect mail.example.com:993 -servername mail.example.comLook for a clean certificate chain and a hostname match.
Create mailboxes and test local delivery
This setup uses Linux system users with Maildir stored in each user’s home directory. It’s easy to reason about and works well for small teams.
If you host many domains and users, you’ll usually move to virtual users and database-backed maps.
1) Create a user
sudo adduser aliceDovecot will create the Maildir structure on first delivery. Send a local test message:
echo "hello" | mail -s "test" aliceConfirm the mailbox exists:
sudo ls -la /home/alice/Maildir2) Test IMAP login
Use a mail client (Thunderbird, Outlook, Apple Mail) or a CLI tool. IMAP settings:
- Server:
mail.example.com - Port:
993(IMAPS) - Username:
alice(oralice@example.comif you map it that way) - Authentication: password
SMTP submission settings:
- Server:
mail.example.com - Port:
587 - Encryption: STARTTLS
- Authentication: required
Publish SPF, DKIM, and DMARC (deliverability basics that actually move the needle)
If you skip authentication, expect major receivers (Gmail, Outlook, Yahoo) to treat your server as suspicious.
In 2026, even low-volume domains need clean SPF/DKIM/DMARC.
SPF: declare who can send for your domain
Add a TXT record for example.com:
example.com. TXT "v=spf1 mx -all"This reads as “only the MX hosts may send for this domain; everything else fails.” If you use a third-party sender, add it explicitly.
Avoid weakening SPF with ~all unless you have a clear reason.
DKIM: sign outgoing mail with your domain key
Install OpenDKIM and tools:
sudo apt install -y opendkim opendkim-toolsCreate a key directory:
sudo mkdir -p /etc/opendkim/keys/example.com
sudo chown -R opendkim:opendkim /etc/opendkim
sudo chmod go-rwx /etc/opendkim/keysGenerate a 2048-bit key (selector name: mail):
sudo opendkim-genkey -b 2048 -d example.com -D /etc/opendkim/keys/example.com -s mail
sudo chown opendkim:opendkim /etc/opendkim/keys/example.com/mail.privateConfigure OpenDKIM:
sudo nano /etc/opendkim.confEnsure these lines exist (or match):
Syslog yes
UMask 002
Mode sv
Canonicalization relaxed/simple
Socket local:/run/opendkim/opendkim.sock
UserID opendkim:opendkim
KeyTable /etc/opendkim/key.table
SigningTable /etc/opendkim/signing.table
ExternalIgnoreList /etc/opendkim/trusted.hosts
InternalHosts /etc/opendkim/trusted.hostsCreate the mapping files:
sudo nano /etc/opendkim/key.tablemail._domainkey.example.com example.com:mail:/etc/opendkim/keys/example.com/mail.privatesudo nano /etc/opendkim/signing.table*@example.com mail._domainkey.example.comsudo nano /etc/opendkim/trusted.hosts127.0.0.1
localhost
mail.example.comMake Postfix talk to OpenDKIM (milter):
sudo nano /etc/postfix/main.cfAdd:
milter_default_action = accept
milter_protocol = 6
smtpd_milters = unix:/run/opendkim/opendkim.sock
non_smtpd_milters = unix:/run/opendkim/opendkim.sockEnsure OpenDKIM creates the socket where Postfix can reach it. On Ubuntu, confirm the service uses /run/opendkim/opendkim.sock:
sudo systemctl restart opendkim
sudo systemctl restart postfixPublish the DKIM DNS record. Print it:
sudo cat /etc/opendkim/keys/example.com/mail.txtCreate the TXT record shown (host: mail._domainkey).
DMARC: tell receivers what to do when SPF/DKIM fail
Start in monitoring mode. Switch to enforcement once you trust your sending sources.
Add TXT at _dmarc.example.com:
_dmarc.example.com. TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com; adkim=s; aspf=s"After reports look clean, move to p=quarantine, then p=reject. HostMyCode’s step-by-step guide is here: SPF/DKIM/DMARC setup guide.
Verify DNS and authentication end-to-end
Don’t rely on assumptions. Check the records directly.
Then validate with real deliveries.
1) Check MX, SPF, DKIM, DMARC records
dig +short example.com MX
dig +short example.com TXT
dig +short mail._domainkey.example.com TXT
dig +short _dmarc.example.com TXT2) Send a message and inspect headers
Send an email from your new server to a mailbox you control (Gmail/Outlook). Open “Show original” and confirm:
- SPF: PASS
- DKIM: PASS
- DMARC: PASS (or aligned)
If messages still land in spam, follow HostMyCode’s troubleshooting flow: email deliverability troubleshooting.
Lock down SMTP abuse (rate-limits, bans, and safe defaults)
If your server becomes a spam relay, deliverability collapses fast.
Strong auth plus basic abuse controls beats hours of “tuning” later.
1) Confirm you are not an open relay
From another machine, attempt a relay test (replace domains). A correctly configured server should reject unauthenticated relaying.
swaks --to victim@external-domain.com --from test@example.com --server mail.example.comIf you don’t have swaks, install it on your workstation. What matters is the outcome.
The relay attempt should be denied.
2) Add Fail2Ban for Postfix and Dovecot
Install:
sudo apt install -y fail2banEnable common jails with a small override file:
sudo nano /etc/fail2ban/jail.d/mail.conf[postfix]
enabled = true
[dovecot]
enabled = trueRestart and check:
sudo systemctl restart fail2ban
sudo fail2ban-client status
sudo fail2ban-client status postfix
sudo fail2ban-client status dovecotFor a broader baseline (including SSH), see: Fail2Ban setup tutorial.
3) Keep logs readable
Mail troubleshooting is mostly log reading. These are the usual suspects:
/var/log/mail.log/var/log/mail.errjournalctl -u postfixjournalctl -u dovecot
Quick tail while testing:
sudo tail -f /var/log/mail.logOperational checklist: backups, updates, and queue health
A mail server needs routine care. Without it, you often learn about failures when password resets or invoices stop arriving.
1) Back up what matters
- Mailboxes:
/home/*/Maildir(or your virtual mailbox path) - Configs:
/etc/postfix,/etc/dovecot,/etc/opendkim - Let’s Encrypt:
/etc/letsencrypt
If you already run snapshots plus offsite copies, include these paths.
For a clean approach on Ubuntu, follow VPS backup setup guide and add the mail directories to your backup set.
2) Watch the mail queue
A growing queue usually points to DNS issues, poor IP reputation, or remote throttling.
sudo postqueue -p
sudo mailqTo inspect a specific queued message:
sudo postcat -q QUEUE_IDIf you need a structured flow, HostMyCode has a practical guide: Email queue troubleshooting.
3) Patch safely and predictably
Mail servers are high-value targets. Schedule updates, plan reboots, and keep a simple maintenance log.
This pairs well with a general routine: Linux Server Maintenance Tutorial (2026).
Common pitfalls (and fast diagnostics)
- PTR missing or mismatched: mail gets spammed. Fix rDNS, confirm forward/reverse alignment.
- Using port 25 from a residential ISP: outbound SMTP may be blocked. Use a VPS IP with clean reputation.
- Broken TLS chain: clients fail to connect. Re-run
openssl s_clienttests and check cert paths. - Open relay: you’ll be blacklisted quickly. Re-check
smtpd_recipient_restrictionsandmynetworks. - SPF too permissive: spoofing risk. Prefer explicit senders and
-allonce stable.
Summary: a production-ready mail stack you can operate
You now have a working Postfix + Dovecot mail server with encrypted client connections, authenticated submission, and the DNS authentication records that drive deliverability in 2026.
Reliability comes from operations: backups, log review, and staying ahead of queue growth.
If you want a VPS sized for email and general hosting workloads, start with a HostMyCode VPS. If you’d rather offload patching, monitoring, and urgent fixes, choose managed VPS hosting and spend your time building, not chasing mail logs.
Running your own mail server works well if you control DNS, take backups seriously, and keep security tight. HostMyCode can run it on a reliable VPS, or you can offload routine maintenance to managed VPS hosting so your mail stack stays patched, monitored, and recoverable.
FAQ
Do I need both SPF and DKIM?
Yes. SPF validates the sending server. DKIM validates the message signature. DMARC ties them together and tells receivers what to do on failure.
Should I enable port 465 (SMTPS) in 2026?
Most clients work well with port 587 + STARTTLS. Enable 465 only if you have legacy clients that require implicit TLS, and still require authentication.
Can I host email for multiple domains on the same VPS?
You can, but plan it first. You’ll add extra DKIM selectors, SPF policies (or include mechanisms), and possibly virtual mailbox mapping.
For multi-tenant needs, a control panel or managed service often reduces mistakes.
My emails are authenticated but still land in spam. What next?
Check rDNS alignment, IP reputation, and content patterns. Then read HostMyCode’s deliverability troubleshooting tutorial and review your logs for deferrals and throttling.
How do I monitor whether Postfix is quietly failing?
Watch mailq size and set alerts on queue growth, disk usage, and failed services. A lightweight reporting approach is Logwatch daily reports for mail and auth logs.