Back to tutorials
Tutorial

SMTP Setup Tutorial (2026): Host Transactional Email on a VPS with Postfix + SPF/DKIM/DMARC

SMTP setup tutorial (2026) to send reliable email from a VPS using Postfix, rDNS, SPF, DKIM, DMARC, TLS, and queue checks.

By Anurag Singh
Updated on Jun 04, 2026
Category: Tutorial
Share article
SMTP Setup Tutorial (2026): Host Transactional Email on a VPS with Postfix + SPF/DKIM/DMARC

Email deliverability fails quietly. Your app “sends” password resets, invoices, and sign-up codes. But they land in spam—or never arrive at all. This SMTP setup tutorial shows a production-ready approach on an Ubuntu VPS: Postfix, correct DNS (rDNS, SPF, DKIM, DMARC), TLS, and the queue/log checks you’ll use during real incidents.

This guide is for transactional sending (your website/app sending mail). If you need full mailbox hosting (IMAP/POP plus webmail), you’ll want a different, heavier stack.

What you’ll build (and what you need before starting)

You’ll end up with:

  • A VPS running Postfix as an authenticated SMTP submission server (587) and a basic SMTP listener (25) for outbound delivery
  • Valid forward DNS (A/AAAA), reverse DNS (PTR), SPF, DKIM, and DMARC
  • TLS for submission and sane security defaults
  • Simple tests: connectivity, TLS, authentication, queue status, and log checks

Prereqs:

  • Ubuntu Server 24.04 LTS or 26.04 LTS on a VPS
  • A domain you control (example: example.com)
  • One hostname for your mail server (example: mail.example.com)
  • Ability to set DNS records for that domain
  • Ability to request/set rDNS (PTR) for your VPS IP

If you’re starting fresh, use a clean VPS. Make sure you control firewall rules and installed packages.

A HostMyCode VPS fits well for SMTP work. You get root access, dedicated resources, and stable networking.

Step 1 — Pick a hostname and set forward DNS

Start with the server identity. Choose a fully qualified domain name (FQDN) and use it everywhere:

  • Mail hostname: mail.example.com
  • Server hostname (same): mail.example.com

Create these DNS records in your zone:

  • A record: mail.example.com → your VPS IPv4
  • AAAA record (optional but recommended): mail.example.com → your VPS IPv6

On the VPS, set the system hostname:

sudo hostnamectl set-hostname mail.example.com
hostnamectl

Confirm it resolves correctly from outside. Run this from your laptop or another server:

dig +short mail.example.com A
dig +short mail.example.com AAAA

Step 2 — Set reverse DNS (PTR) to match your mail hostname

For email, rDNS isn’t optional. A missing or mismatched PTR can get you throttled—or ignored—by major receivers.

Set your VPS IP’s PTR record to:

  • your.server.ip.address → mail.example.com

Then verify:

dig +short -x YOUR_IPV4
# should return: mail.example.com.

If you haven’t configured PTR before, follow HostMyCode’s guide: Reverse DNS setup guide for VPS & dedicated servers.

Step 3 — Install Postfix and helpers (Ubuntu)

Update packages. Then install Postfix plus the pieces you’ll use for SASL auth and DKIM signing:

sudo apt update
sudo apt -y install postfix mailutils libsasl2-modules sasl2-bin opendkim opendkim-tools

During the Postfix prompt:

  • Select Internet Site
  • System mail name: example.com (your domain)

Check versions and service status:

postconf mail_version
systemctl status postfix --no-pager

Step 4 — Configure Postfix basics (identity, networks, TLS skeleton)

Edit /etc/postfix/main.cf. Keep it readable and explicit.

Add or adjust these values (replace example.com and mail.example.com):

sudo nano /etc/postfix/main.cf
# Identity
myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
mydestination = localhost
inet_interfaces = all
inet_protocols = all

# Basic SMTP banner hygiene
smtpd_banner = $myhostname ESMTP

# Don’t be an open relay
mynetworks = 127.0.0.0/8 [::1]/128
relay_domains =

# Size limits (tune to your use case)
message_size_limit = 26214400
mailbox_size_limit = 0

# Logging
maillog_file = /var/log/mail.log

# TLS (certs added in next step)
smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtp_tls_security_level = may
smtp_tls_loglevel = 1

Why these settings matter:

  • mydestination = localhost keeps Postfix focused on outbound/relay use. You’re not hosting inbound mailboxes for the domain in this tutorial.
  • mynetworks restricts relaying to localhost only; clients must authenticate on 587.

Restart Postfix to apply changes:

sudo systemctl restart postfix

Step 5 — Add TLS certificates for SMTP submission (Let’s Encrypt)

For submission on port 587, TLS isn’t negotiable. If you already have a certificate for mail.example.com, use it.

Otherwise, issue one with Let’s Encrypt.

Install Certbot:

sudo apt -y install certbot

Stop anything else binding to 80 (if needed). Then request a cert using the standalone method:

sudo certbot certonly --standalone -d mail.example.com

Your cert paths will look like:

  • /etc/letsencrypt/live/mail.example.com/fullchain.pem
  • /etc/letsencrypt/live/mail.example.com/privkey.pem

Now set these in /etc/postfix/main.cf:

smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file  = /etc/letsencrypt/live/mail.example.com/privkey.pem

Restart Postfix:

sudo systemctl restart postfix

If you want a deeper walk-through for certificate issuance and validation, see: SSL certificate setup guide for Ubuntu VPS. The same certificate concepts apply to SMTP services.

Step 6 — Enable authenticated SMTP on port 587 (Submission)

Next you’ll allow authenticated clients (your app, WordPress, a backend worker) to submit email on port 587.

Edit /etc/postfix/master.cf. Enable the submission service by uncommenting it.

Set options like this:

sudo nano /etc/postfix/master.cf
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_type=dovecot
  -o smtpd_sasl_path=private/auth
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

Those lines reference Dovecot for SASL auth. But Dovecot isn’t installed.

You have two practical options:

  1. Use Cyrus SASL with saslauthd (lighter, fine for a dedicated SMTP host)
  2. Use Dovecot auth (a better fit if you plan to add IMAP/webmail later)

This tutorial uses Cyrus SASL (no IMAP). Replace the Dovecot lines with Cyrus settings.

Back in /etc/postfix/master.cf, change the submission block to:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_type=cyrus
  -o smtpd_sasl_path=smtpd
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

Now add SASL options for Postfix in /etc/postfix/sasl/smtpd.conf:

sudo mkdir -p /etc/postfix/sasl
sudo nano /etc/postfix/sasl/smtpd.conf
pwcheck_method: saslauthd
mech_list: plain login

Enable and configure saslauthd to use system users. This setup is simple and effective:

sudo sed -i 's/^START=.*/START=yes/' /etc/default/saslauthd
sudo sed -i 's/^MECHANISMS=.*/MECHANISMS="pam"/' /etc/default/saslauthd
sudo systemctl enable --now saslauthd

Add the Postfix user to the SASL group. This lets Postfix talk to saslauthd:

sudo adduser postfix sasl
sudo systemctl restart saslauthd

Finally, ensure Postfix uses SASL authentication in /etc/postfix/main.cf:

smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
broken_sasl_auth_clients = yes

Restart Postfix:

sudo systemctl restart postfix

Step 7 — Create a dedicated SMTP user (don’t reuse root)

Create a system user dedicated to SMTP authentication. This becomes the username/password you plug into your application’s SMTP settings.

sudo adduser --disabled-login --gecos "SMTP User" smtpuser
sudo passwd smtpuser

Tip: Store the password in a manager. Rotate it if it ever leaks.

If multiple apps send mail, create one user per app. That makes revocation clean and low-risk.

Step 8 — Firewall: open only what you actually need

For transactional sending, keep it simple:

  • Allow 587/tcp for authenticated submission
  • Allow 25/tcp for outbound SMTP delivery (and inbound connections from other mail servers)
  • Optionally allow 22/tcp for SSH administration

If you use UFW:

sudo ufw allow 22/tcp
sudo ufw allow 25/tcp
sudo ufw allow 587/tcp
sudo ufw enable
sudo ufw status verbose

Need a safer baseline first? Use this: VPS hardening tutorial for hosting.

Do it before you expose SMTP services.

Step 9 — Add SPF, DKIM, and DMARC (deliverability basics you can verify)

DNS auth turns “mail was accepted somewhere” into “mail consistently lands.”

Set SPF, DKIM, and DMARC. Then verify each record from the command line.

SPF record

Create a TXT record for the root domain:

Host/Name: @
Type: TXT
Value: v=spf1 a mx ip4:YOUR_IPV4 -all

If you also send from another provider (for example, a newsletter service), include it explicitly. Avoid permissive +all.

DKIM signing with OpenDKIM

Generate a DKIM keypair. First, pick a selector name (example: mta1):

sudo mkdir -p /etc/opendkim/keys/example.com
sudo opendkim-genkey -D /etc/opendkim/keys/example.com -d example.com -s mta1
sudo chown -R opendkim:opendkim /etc/opendkim/keys/example.com
sudo chmod 600 /etc/opendkim/keys/example.com/mta1.private

Configure OpenDKIM:

sudo nano /etc/opendkim.conf

Ensure these lines exist (adjust if your distro already has similar defaults):

Syslog                  yes
UMask                   002
Canonicalization        relaxed/simple
Mode                    sv
SubDomains              no
AutoRestart             yes
AutoRestartRate         10/1h
Background              yes
DNSTimeout              5
SignatureAlgorithm      rsa-sha256

KeyTable                /etc/opendkim/key.table
SigningTable            /etc/opendkim/signing.table
ExternalIgnoreList      /etc/opendkim/trusted.hosts
InternalHosts           /etc/opendkim/trusted.hosts
Socket                  local:/run/opendkim/opendkim.sock

Create the tables:

sudo nano /etc/opendkim/key.table
mta1._domainkey.example.com example.com:mta1:/etc/opendkim/keys/example.com/mta1.private
sudo nano /etc/opendkim/signing.table
*@example.com mta1._domainkey.example.com
sudo nano /etc/opendkim/trusted.hosts
127.0.0.1
localhost
mail.example.com

Make sure the socket directory exists and permissions are correct:

sudo mkdir -p /run/opendkim
sudo chown opendkim:opendkim /run/opendkim

Start and enable OpenDKIM:

sudo systemctl enable --now opendkim
systemctl status opendkim --no-pager

Connect Postfix to OpenDKIM. Add this to /etc/postfix/main.cf:

milter_default_action = accept
milter_protocol = 6
smtpd_milters = unix:/run/opendkim/opendkim.sock
non_smtpd_milters = unix:/run/opendkim/opendkim.sock

Restart Postfix:

sudo systemctl restart postfix

Now publish your DKIM public key in DNS. Display the TXT record value:

sudo cat /etc/opendkim/keys/example.com/mta1.txt

Create a TXT record for mta1._domainkey.example.com with the content from that file.

DMARC policy

Start in monitoring mode. Tighten the policy after you confirm alignment and identify every legitimate sender.

Host/Name: _dmarc
Type: TXT
Value: v=DMARC1; p=none; rua=mailto:dmarc-reports@example.com; ruf=mailto:dmarc-fail@example.com; fo=1

You can tighten to p=quarantine and later p=reject once you’re confident you’re only sending from authorized sources.

If you want a dedicated, step-by-step DNS-auth walkthrough (with validation tips), use: SPF/DKIM/DMARC setup guide for VPS email deliverability.

Step 10 — Test SMTP submission, auth, and TLS

Run three checks. Confirm the ports are open. Confirm TLS negotiates cleanly. Then send a real authenticated message.

1) Confirm ports are listening

sudo ss -lntp | egrep ':(25|587)\s'

2) Test TLS on 587

From another machine:

openssl s_client -starttls smtp -connect mail.example.com:587 -servername mail.example.com

You’re looking for a valid certificate chain and no obvious TLS errors.

3) Send a test mail via authenticated SMTP

Install swaks (a practical SMTP testing tool):

sudo apt -y install swaks

Run a real submission test:

swaks --to you@gmail.com \
  --from test@example.com \
  --server mail.example.com \
  --port 587 \
  --auth LOGIN \
  --auth-user smtpuser \
  --auth-password 'YOUR_PASSWORD' \
  --tls

If it fails, go straight to the logs:

sudo tail -n 200 /var/log/mail.log

Step 11 — Common troubleshooting checks (fast, repeatable)

Most mail issues become obvious once you check the right place.

Start with the queue and the logs. Then validate DNS.

Queue checks

If mail is stuck, inspect the queue:

mailq
postqueue -p

Force a queue retry after fixing DNS or connectivity:

sudo postqueue -f

Delete a specific queued message (use carefully):

sudo postsuper -d QUEUE_ID

DNS sanity: SPF/DKIM/DMARC visible?

dig +short TXT example.com
dig +short TXT mta1._domainkey.example.com
dig +short TXT _dmarc.example.com

Port 25 blocked or throttled?

Some networks block outbound 25. When that happens, you’ll usually see timeouts and repeated deferrals in /var/log/mail.log.

Quick test from the VPS:

nc -vz gmail-smtp-in.l.google.com 25

If outbound 25 is blocked, you have two options:

  • Ask your provider to allow it (preferred for running your own SMTP)
  • Relay through a smart host on port 587 (different architecture; not covered here)

“Relay access denied” errors

This usually means your client is trying to send on port 25 without auth.

It can also mean your submission service isn’t enforcing permit_sasl_authenticated.

Use port 587 with auth. Then re-check the submission block in /etc/postfix/master.cf.

Step 12 — Production hardening checklist for SMTP on a VPS

Once basic sending works, treat the server like a production system—because it is.

  • Disable password SSH logins and use keys only.
  • Keep Postfix updated: sudo apt update && sudo apt -y upgrade (or unattended-upgrades).
  • Limit who can authenticate: one user per app; rotate passwords on change.
  • Set rate controls if you run public-facing endpoints that can trigger email floods.
  • Monitor logs: authentication failures, queue growth, deferred messages.
  • Start DMARC at p=none, then move to quarantine/reject after validating.

If your website runs on the same box, harden the web layer too.

Bot traffic hitting login and form endpoints often triggers email bursts (password resets, OTPs). Nginx rate limiting is a solid companion here: Nginx rate limiting tutorial.

Step 13 — Configure WordPress (or your app) to use your SMTP server

For WordPress, use a reputable SMTP plugin and set:

  • SMTP host: mail.example.com
  • Encryption: TLS
  • Port: 587
  • Authentication: On
  • Username: smtpuser
  • Password: (from your password manager)
  • From address: something like no-reply@example.com or support@example.com

Test with a password reset and a form submission, not just the plugin’s “send test email” button.

Real templates and headers expose the issues you’ll see in production.

If you’re running WordPress on a VPS and want a safer workflow for changes before pushing to production, this staging guide pairs well with email testing: WordPress staging site tutorial.

Summary: a reliable SMTP server is mostly DNS + discipline

This build stays simple. It stays strict about the details that decide deliverability.

That means: a consistent hostname, matching rDNS, authenticated submission on 587, TLS, and correctly published SPF/DKIM/DMARC.

Once those are correct, operations are routine. Watch the queue, read the logs, and keep the box patched.

If you’d rather not handle OS updates and mail-stack maintenance yourself, managed VPS hosting from HostMyCode can cover routine admin work while you keep control of your app and sending.

If you’re setting up transactional email for WordPress, WooCommerce, or an app backend, run it on a VPS you control so your DNS, rDNS, and security settings stay consistent. Start with a HostMyCode VPS, and move to managed VPS hosting if you want help with patching, monitoring, and ongoing server maintenance.

FAQ

Do I need port 25 open if I’m only sending mail from my website?

Yes, if your Postfix server will deliver directly to other mail servers. Your app submits via 587, but your MTA still needs port 25 to reach recipient MX servers.

Why does rDNS (PTR) matter for SMTP deliverability?

Receivers use PTR as a trust signal. A missing or mismatched PTR can cause spam scoring, throttling, or outright rejection. Set PTR to your mail hostname and keep it stable.

Should I set DMARC to “reject” immediately?

No. Start with p=none to collect reports and confirm you’re not forgetting any legitimate senders. Tighten to quarantine/reject once alignment is consistently clean.

Can I run SMTP on the same VPS as my website?

You can, and it’s common for small stacks. Keep an eye on resource spikes and security. If your site gets abused (form spam, brute force), your mail reputation can suffer.

What log file should I check first when mail doesn’t arrive?

On Ubuntu, start with /var/log/mail.log. Look for “deferred”, DNS failures, authentication errors, and remote server responses.