Linux VPS mail server setup (Postfix + Dovecot) on Ubuntu 24.04 in 2026: SMTP, IMAP, SPF/DKIM/DMARC, and TLS

Email is still one of the quickest ways to damage trust in your domain. Most failures come down to basics: DNS that doesn’t line up, TLS that’s miswired, or missing authentication.

This Linux VPS mail server setup walks through a Postfix + Dovecot build on Ubuntu 24.04 LTS. It also covers the deliverability requirements that matter in 2026: SPF, DKIM, DMARC, rDNS, and clean TLS.

The walkthrough assumes one domain (for example, example.com) on one VPS. You’ll end with SMTP submission on 587/465 and IMAP on 993. You’ll also have working mailbox logins and a checklist to verify each layer before you move on.

What you’ll build (and what you should not)

SMTP server: Postfix for inbound/outbound mail.
IMAP server: Dovecot for mailbox access.
Security: TLS certificates, SASL auth, sane cipher defaults.
Deliverability: SPF, DKIM (OpenDKIM), DMARC, rDNS alignment.

You should not use this as a bulk marketing sender. In 2026, major mailbox providers throttle and scrutinize new/self-hosted IPs that behave like ESPs.

This setup works best for transactional mail, small-team mailboxes, and service accounts.

Prerequisites and sizing checklist

Get the fundamentals right before installing anything:

Ubuntu 24.04 LTS server (fresh is best).
A static public IPv4 address (IPv6 is a plus but not required for this guide).
A domain you control: example.com.
Hostname/FQDN for the mail server: mail.example.com.
Reverse DNS (PTR) available from your hosting provider.

For a small mail server, 1 vCPU / 2 GB RAM / 40 GB SSD is a sensible floor. If you expect many concurrent IMAP connections or larger mailboxes, move up to 2 vCPU / 4 GB RAM.

A HostMyCode VPS fits well for predictable email workloads. If you’d rather not handle OS patching and core-service health yourself, use managed VPS hosting.

Step 1 — Set the server hostname and verify DNS basics

Set the system hostname to match your mail FQDN. On Ubuntu 24.04:

sudo hostnamectl set-hostname mail.example.com
hostnamectl

Make sure your A record resolves:

dig +short A mail.example.com

You should see your VPS IP.

Pitfall: If mail.example.com points to a CDN/WAF IP, mail delivery will fail. SMTP must reach your server’s real address.

Step 2 — Set rDNS (PTR) correctly (deliverability critical)

rDNS is still a gatekeeper in 2026. Your IP’s PTR record should resolve to the same hostname you use for mail.

PTR: <your-ip> → mail.example.com
Forward: mail.example.com → <your-ip>

Verify it from the VPS:

dig +short -x <your-ip>

If you can’t set PTR yourself, open a ticket with your provider.

Without PTR alignment, some receivers will junk or throttle your mail immediately.

Step 3 — Install Postfix, Dovecot, and OpenDKIM

Update packages, then install the core components:

sudo apt update
sudo apt -y upgrade
sudo apt -y install postfix dovecot-imapd dovecot-pop3d dovecot-lmtpd opendkim opendkim-tools certbot

During the Postfix setup prompts:

Select Internet Site
System mail name: example.com

Capture versions now. This saves time later during debugging.

postconf mail_version
dovecot --version
opendkim -V

Step 4 — Configure firewall ports (and keep them tight)

On Ubuntu, UFW is usually the quickest path:

sudo ufw allow OpenSSH
sudo ufw allow 25/tcp   # SMTP (inbound)
sudo ufw allow 587/tcp  # SMTP submission (STARTTLS)
sudo ufw allow 465/tcp  # SMTPS (optional, but common for clients)
sudo ufw allow 993/tcp  # IMAPS
sudo ufw enable
sudo ufw status verbose

If you want a broader hardening baseline, apply it after mail is working and tested.

HostMyCode’s checklist is a solid follow-up: Linux VPS hardening checklist in 2026.

Step 5 — Configure Postfix for submission, TLS, and Dovecot SASL

Edit /etc/postfix/main.cf. The settings below are a clean starting point for a single-domain server:

sudo nano /etc/postfix/main.cf

# Identity
myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
inet_interfaces = all
inet_protocols = all

# Mailbox delivery (Maildir)
home_mailbox = Maildir/

# TLS (we'll add cert paths after issuing)
smtpd_tls_security_level = may
smtp_tls_security_level = may
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes

# Submission via Dovecot SASL
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous

# Basic anti-open-relay rules
smtpd_recipient_restrictions =
  permit_mynetworks,
  permit_sasl_authenticated,
  reject_unauth_destination

# Limits (reasonable defaults)
message_size_limit = 26214400
mailbox_size_limit = 0

Next, enable the submission services in /etc/postfix/master.cf. Uncomment and adjust as shown:

sudo nano /etc/postfix/master.cf

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

Verification: restart Postfix. Then confirm it’s listening on the expected ports.

sudo systemctl restart postfix
sudo ss -lntp | egrep ':(25|587|465)\b'

Step 6 — Configure Dovecot for Maildir and SMTP auth

Dovecot handles IMAP and provides SMTP authentication to Postfix.

Start by setting Maildir and enabling the protocols you’ll use.

Edit /etc/dovecot/conf.d/10-mail.conf:

sudo nano /etc/dovecot/conf.d/10-mail.conf

mail_location = maildir:~/Maildir

Edit /etc/dovecot/dovecot.conf:

sudo nano /etc/dovecot/dovecot.conf

protocols = imap lmtp

Now configure the auth socket Postfix will use. In /etc/dovecot/conf.d/10-master.conf, find service auth.

Add or confirm this block:

sudo nano /etc/dovecot/conf.d/10-master.conf

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

Verification: restart Dovecot and confirm it’s listening on 993.

sudo systemctl restart dovecot
sudo ss -lntp | egrep ':(993)\b'

Step 7 — Issue a TLS certificate with Let’s Encrypt (Certbot)

For mail, you can use HTTP-01 with a web server. Or run Certbot in standalone mode if port 80 is free.

Standalone is often simplest on a new mail-only VPS.

sudo systemctl stop postfix dovecot
sudo certbot certonly --standalone -d mail.example.com

After a successful run, you’ll have:

/etc/letsencrypt/live/mail.example.com/fullchain.pem
/etc/letsencrypt/live/mail.example.com/privkey.pem

Point Postfix at the certificate (/etc/postfix/main.cf):

sudo postconf -e 'smtpd_tls_cert_file=/etc/letsencrypt/live/mail.example.com/fullchain.pem'
sudo postconf -e 'smtpd_tls_key_file=/etc/letsencrypt/live/mail.example.com/privkey.pem'

Point Dovecot at the same certificate (/etc/dovecot/conf.d/10-ssl.conf):

sudo nano /etc/dovecot/conf.d/10-ssl.conf

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem

Bring the services back up:

sudo systemctl start postfix dovecot

Verification: confirm the certificate presented on 993 and 587.

openssl s_client -connect mail.example.com:993 -servername mail.example.com </dev/null | openssl x509 -noout -subject -issuer -dates
openssl s_client -starttls smtp -connect mail.example.com:587 -servername mail.example.com </dev/null | openssl x509 -noout -subject -issuer -dates

Pitfall: Renewal failures usually happen because port 80 is unreachable or DNS changed.

A quick monthly check beats discovering an expired cert in a support ticket.

Step 8 — Create a mailbox user and send a local test

This guide uses system users with Maildir.

Create a dedicated user for your first mailbox:

sudo adduser alice

Create the Maildir folders:

sudo -u alice maildirmake.dovecot /home/alice/Maildir
sudo -u alice maildirmake.dovecot /home/alice/Maildir/.Sent
sudo -u alice maildirmake.dovecot /home/alice/Maildir/.Trash

Send a local test message:

echo "test from $(hostname -f)" | mail -s "local test" alice

Then check logs for delivery:

sudo journalctl -u postfix -n 80 --no-pager

Step 9 — Configure MX, SPF, DKIM, and DMARC DNS records

This is where most “why is everything going to spam?” problems start.

Set these records before you point users at the server.

MX record

Type: MX
Name: @ (or example.com)
Value: mail.example.com
Priority: 10

SPF record (TXT)

For a single mail server IP:

example.com.  TXT  "v=spf1 a:mail.example.com -all"

If you also send through a third-party provider, add their include.

Keep SPF under the 10-DNS-lookup limit.

DKIM with OpenDKIM

Create a key for your domain:

sudo mkdir -p /etc/opendkim/keys/example.com
sudo opendkim-genkey -b 2048 -d example.com -D /etc/opendkim/keys/example.com -s default
sudo chown -R opendkim:opendkim /etc/opendkim/keys/example.com
sudo chmod 0700 /etc/opendkim/keys/example.com

Configure OpenDKIM:

sudo nano /etc/opendkim.conf

Syslog                  yes
UMask                   002
Canonicalization        relaxed/simple
Mode                    sv
SubDomains              no
AutoRestart             yes
AutoRestartRate         10/1h
OversignHeaders         From
Socket                  local:/run/opendkim/opendkim.sock
KeyTable                /etc/opendkim/key.table
SigningTable            /etc/opendkim/signing.table
TrustedHosts            /etc/opendkim/trusted.hosts

Create the tables:

sudo nano /etc/opendkim/key.table

default._domainkey.example.com example.com:default:/etc/opendkim/keys/example.com/default.private

sudo nano /etc/opendkim/signing.table

*@example.com default._domainkey.example.com

sudo nano /etc/opendkim/trusted.hosts

127.0.0.1
localhost
mail.example.com
example.com

Create the socket directory (if needed). Then restart OpenDKIM:

sudo mkdir -p /run/opendkim
sudo chown opendkim:opendkim /run/opendkim
sudo systemctl enable --now opendkim
sudo systemctl restart opendkim

Attach DKIM to Postfix in /etc/postfix/main.cf:

sudo postconf -e 'milter_default_action=accept'
sudo postconf -e 'milter_protocol=6'
sudo postconf -e 'smtpd_milters=unix:/run/opendkim/opendkim.sock'
sudo postconf -e 'non_smtpd_milters=unix:/run/opendkim/opendkim.sock'
sudo systemctl restart postfix

Publish the DKIM DNS TXT record. Print it from the generated file:

sudo cat /etc/opendkim/keys/example.com/default.txt

It will look like:

default._domainkey IN TXT ( "v=DKIM1; k=rsa; p=..." )

Add that as a TXT record for default._domainkey.

DMARC record (TXT)

Start with monitoring so you can review reports before enforcement:

_dmarc.example.com. TXT "v=DMARC1; p=none; rua=mailto:dmarc@example.com; ruf=mailto:dmarc@example.com; fo=1"

Once you’ve confirmed alignment, move to p=quarantine, then possibly p=reject.

Go slower if you have other senders for the same domain.

Step 10 — Verify DNS and authentication end-to-end

Start with command-line checks. They’re faster than troubleshooting through a GUI client.

dig +short MX example.com

SPF:

dig +short TXT example.com | tr -d '"'

DKIM record exists:

dig +short TXT default._domainkey.example.com

DMARC:

dig +short TXT _dmarc.example.com | tr -d '"'

Next, send a real message to a mailbox you control (Gmail/Outlook). Use authenticated submission on 587.

That tests the same path your clients will use.

sudo apt -y install swaks
swaks --to you@gmail.com --from alice@example.com --server mail.example.com --port 587 --auth LOGIN --auth-user alice --auth-password 'YOUR_PASSWORD' --tls

In the received message headers, confirm:

spf=pass
dkim=pass
dmarc=pass (or at least aligned once you enforce)

Step 11 — Mail client settings (IMAP + SMTP submission)

Give users a single block they can copy into their mail client.

This prevents most setup mistakes.

IMAP server: mail.example.com
IMAP port: 993 (SSL/TLS)
SMTP server: mail.example.com
SMTP port: 587 (STARTTLS) or 465 (SSL/TLS)
Username: system username (e.g., alice)
Password: system password
Authentication: required

Pitfall: Even if your VPS can send outbound on port 25, many providers rate-limit new IPs.

Users should send via 587. Port 25 is mainly for server-to-server delivery.

Step 12 — Common troubleshooting (fast diagnostics)

Most issues fall into three buckets: DNS, TLS, or auth/relay rules.

Don’t guess. Check logs and confirm the ports.

Mail not arriving (MX/DNS issues)

Check MX points to mail.example.com.
Check mail.example.com A record matches the VPS IP.
Check firewall allows 25 inbound.

sudo ss -lntp | grep ':25 '
sudo journalctl -u postfix -n 120 --no-pager

Can receive mail but can’t send (submission/auth issues)

First confirm Postfix can see the Dovecot auth socket:

sudo ls -l /var/spool/postfix/private/auth
sudo journalctl -u dovecot -n 120 --no-pager

If you see “no SASL authentication mechanisms,” the socket path or permissions are almost always the culprit.

Goes to spam (authentication or reputation)

Ensure PTR is set and matches mail.example.com.
Ensure SPF and DKIM pass in headers.
Don’t send large bursts from a fresh IP.
Use consistent From domains; avoid From: mismatch.

Also keep logs from eating your disk. Mail servers get noisy under attack.

If you haven’t already, set up rotation: Linux VPS log rotation setup.

Step 13 — Backups: what to copy and how to prove you can restore

At minimum, back up:

/home/*/Maildir/ (mailboxes)
/etc/postfix/, /etc/dovecot/, /etc/opendkim/
/etc/letsencrypt/ (or be ready to re-issue certs)

File-level backups with versioning work well here.

The key is testing restores, not just taking backups. If you want a production-ready pattern with verification steps, follow: Linux VPS backup strategy with restic + S3.

Quick restore drill: restore one mailbox into a temporary path. Fix ownership. Then confirm Dovecot can read it and messages appear.

Step 14 — Operations: updates, monitoring signals, and safe changes

Mail is easiest to operate when monitoring is boring and specific.

Watch the signals that indicate real breakage.

SMTP port check (25 and 587 reachable).
IMAP port check (993 reachable).
Disk usage alerts (mailboxes grow quietly).
Queue depth: mailq should not grow without reason.

mailq | head
postqueue -p | head
sudo journalctl -u postfix -S "1 hour ago" --no-pager | tail -n 50

If you want a broader monitoring baseline for VPS services, use: Linux VPS performance monitoring in 2026.

Summary: a stable mail server you can support

This Linux VPS mail server setup gives you a supportable baseline: Postfix for SMTP, Dovecot for IMAP, Let’s Encrypt TLS, and the DNS authentication stack (SPF/DKIM/DMARC) that receivers expect in 2026.

Treat PTR and DNS as first-class requirements. Verify each layer before you change the next one.

If you’re building this for client mailboxes or you want predictable performance under real traffic, start with a HostMyCode VPS.

If you’d rather not babysit patches and core service health, managed VPS hosting is a cleaner long-term choice.

Self-hosted mail is manageable when your VPS behaves predictably. HostMyCode’s VPS plans give you stable resources and full control for Postfix/Dovecot. If you want someone else handling baseline hardening, updates, and troubleshooting, choose managed VPS hosting so you can stay focused on deliverability and user support.

FAQ

Do I need both ports 587 and 465?

No, but supporting both reduces client friction. Use 587 (STARTTLS) as the default. Keep 465 for older clients that insist on implicit TLS.

Can I host multiple domains on one VPS with this setup?

Yes, but plan it deliberately. You’ll add virtual domains in Postfix, separate DKIM selectors/keys per domain, and ensure SPF/DMARC alignment for each sender.

Why does Gmail accept my mail but Outlook blocks it?

It’s usually reputation and rDNS alignment. Check PTR → hostname → A record consistency, then confirm SPF/DKIM/DMARC pass. Also avoid sending bursts from a new IP.

What’s the safest way to migrate mailboxes to this server?

Use IMAP sync tools (like imapsync) for mailbox contents, then cut over DNS MX after you’ve verified new deliveries. For the “don’t break production” mindset, adapt the sequencing from this guide: VPS migration checklist.

How do I prevent disk-full outages from mail logs and spam bursts?

Rotate logs, monitor disk usage, and set sensible Postfix limits. Start with: logrotate setup on a Linux VPS, then add alerts for filesystem usage and mail queue growth.