Mail Transfer Agent Setup Guide Tutorial (2026): Build a VPS SMTP Relay with Postfix, TLS, SPF/DKIM and Safe Rate Limits

Your site can be fast and rock-solid, yet your emails still bounce with “policy” errors or land in spam. That usually means the mail stack was rushed. Common causes include a sloppy hostname, missing DNS records, weak TLS, or no throttling.

This mail transfer agent setup guide tutorial shows how to build a small-business SMTP relay on a Linux VPS with Postfix. It also covers deliverability basics and safety rails that reduce blacklist risk.

This walkthrough is for admins who run sites on a VPS or dedicated server and need reliable outbound email. Typical use cases include contact forms, invoices, app notifications, and low-to-medium transactional volume.

You’ll set up an SMTP submission service. Then you’ll point your apps, WordPress, or other servers at it for sending.

What you’ll build (and what you won’t)

Outbound relay with Postfix (SMTP submission on ports 587/465).
TLS with Let’s Encrypt and modern protocol settings.
Authenticated sending (SASL) so strangers can’t use your server.
Deliverability foundation: rDNS, SPF, DKIM, DMARC, sensible HELO.
Rate limits to reduce risk during compromised-account events.

You will not build a full inbound mail hosting platform with IMAP/POP/webmail. If you need mailboxes, keep an inbound provider and use this VPS strictly for outbound sending.

Prerequisites checklist (do this before you touch Postfix)

Start with a clean VPS. For predictable results, use Ubuntu 24.04 LTS or Debian 12.

You’ll need:

A VPS with a dedicated IPv4 (recommended) and working outbound SMTP (some networks block port 25 by default).
A domain you control (for example, example.com).
DNS access for A, PTR/rDNS, SPF, DKIM, and DMARC records.
Root SSH access.

If you’re setting this up for customer sites or a repeatable reseller workflow, start with a network you can trust. Use an IP with a clean reputation.

A HostMyCode VPS works well for outbound relay use. You control DNS, firewall policy, and logs without shared-hosting limitations.

Step 1 — Set the correct hostname (avoid “Bad HELO” and policy bounces)

Receivers compare three items constantly: your HELO/EHLO name, reverse DNS, and forward DNS. If they don’t line up, you’ll see bounces or higher spam scoring.

Pick a mail hostname like mail.example.com.
Create an A record: mail.example.com → your VPS IP.
Set your server hostname:

sudo hostnamectl set-hostname mail.example.com
hostnamectl

Ensure /etc/hosts has a sensible mapping (adjust IP if needed):

127.0.0.1 localhost
127.0.1.1 mail.example.com mail

Next, set reverse DNS (PTR) for your VPS IP to mail.example.com. Most providers expose this in a control panel.

If you want a quick refresher on rDNS and how to verify it, use this reverse DNS setup tutorial.

Step 2 — Install Postfix + SASL on Ubuntu/Debian

Update packages. Then install Postfix and the SASL components you’ll use for authenticated submission.

sudo apt update
sudo apt -y install postfix libsasl2-modules sasl2-bin mailutils

During the Postfix installer prompt:

Select: Internet Site
System mail name: example.com (or your preferred domain)

Confirm Postfix is running:

systemctl status postfix --no-pager

Step 3 — Lock down the firewall for mail submission

For an outbound relay, you normally only need inbound access for:

SSH (restricted to your admin IP if possible)
SMTP submission (587) and optionally SMTPS (465)
Port 25 inbound is optional (many relays don’t need public inbound 25)

If you use UFW:

sudo ufw allow OpenSSH
sudo ufw allow 587/tcp
sudo ufw allow 465/tcp
sudo ufw enable
sudo ufw status verbose

If you prefer nftables/iptables-style rules and want a production-grade baseline, follow our iptables/nftables firewall tutorial. Include ports 587/465 for submission.

Step 4 — Configure Postfix as a submission-only relay (the safe baseline)

Edit /etc/postfix/main.cf. Make a backup first so you can roll back cleanly.

sudo cp -a /etc/postfix/main.cf /etc/postfix/main.cf.bak.$(date +%F)
sudo nano /etc/postfix/main.cf

Add or adjust these settings (replace example.com and hostname values):

# Identity
myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
smtpd_banner = $myhostname ESMTP

# Listen on all interfaces (submission service will be restricted separately)
inet_interfaces = all
inet_protocols = ipv4

# Don't be an open relay
mynetworks = 127.0.0.0/8 [::1]/128
smtpd_relay_restrictions = permit_sasl_authenticated, reject_unauth_destination

# Basic recipient checks (lightweight but useful)
smtpd_recipient_restrictions =
  permit_sasl_authenticated,
  reject_unauth_destination,
  reject_unknown_recipient_domain,
  reject_non_fqdn_recipient

# TLS defaults (cert paths added later)
smtpd_tls_security_level = may
smtp_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes

# Modern TLS (Postfix uses system OpenSSL; keep it conservative)
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# Mail size (set to what your business needs)
message_size_limit = 26214400

Why “may” for TLS? For server-to-server delivery, you can’t require TLS everywhere. Some destinations still won’t negotiate it.

For client submission (587/465), you should require encryption. You’ll enforce that in master.cf.

Step 5 — Enable authenticated SMTP submission (587) and SMTPS (465)

Edit /etc/postfix/master.cf:

sudo cp -a /etc/postfix/master.cf /etc/postfix/master.cf.bak.$(date +%F)
sudo nano /etc/postfix/master.cf

Find the submission service. Make sure it’s enabled and configured like this:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

If you want port 465 (SMTPS), enable smtps as well:

smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject

Step 6 — Configure SASL authentication (no plaintext passwords)

On Ubuntu/Debian with Postfix, you can do SASL via Dovecot or Cyrus SASL. For a submission-only relay, Cyrus SASL with local auth is simple and contained.

Create /etc/postfix/sasl/smtpd.conf:

sudo mkdir -p /etc/postfix/sasl
sudo nano /etc/postfix/sasl/smtpd.conf

pwcheck_method: saslauthd
mech_list: plain login

Enable and configure saslauthd to use the local PAM database:

sudo sed -i 's/^START=.*/START=yes/' /etc/default/saslauthd
sudo sed -i 's/^MECHANISMS=.*/MECHANISMS="pam"/' /etc/default/saslauthd
sudo systemctl enable --now saslauthd

Add Postfix user to the sasl group:

sudo adduser postfix sasl

Now tell Postfix to use SASL in /etc/postfix/main.cf:

sudo postconf -e 'smtpd_sasl_auth_enable = yes'
sudo postconf -e 'smtpd_sasl_security_options = noanonymous'
sudo postconf -e 'broken_sasl_auth_clients = yes'

Create a dedicated system user for SMTP auth. Don’t reuse your admin login.

sudo adduser smtpuser

Restart services:

sudo systemctl restart saslauthd
sudo systemctl restart postfix

Step 7 — Add TLS certificates (Let’s Encrypt) and validate from the outside

For SMTP TLS, receivers and clients expect a certificate that matches mail.example.com.

Install Certbot:

sudo apt -y install certbot

If this VPS doesn’t run a website, use the standalone method. It temporarily binds to port 80.

sudo ufw allow 80/tcp
sudo certbot certonly --standalone -d mail.example.com

Point Postfix at the cert and key:

sudo postconf -e 'smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem'
sudo postconf -e 'smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem'
sudo systemctl reload postfix

Test STARTTLS on 587 from your local machine:

openssl s_client -starttls smtp -connect mail.example.com:587 -servername mail.example.com

If you need a deeper reference for common certificate problems (DNS, permissions, renewal timers), keep this SSL certificate setup guide nearby.

Step 8 — Publish SPF, DKIM, and DMARC (minimum viable deliverability)

Receivers want proof that your domain authorizes this server to send. You don’t need perfection on day one. You do need the basics in place.

SPF record

Add a TXT record for your root domain (example.com):

v=spf1 ip4:YOUR_VPS_IP -all

If you also send through a third-party provider (Google Workspace, Microsoft, etc.), include them in the same record.

Keep SPF to one accurate statement, not a collection of guesses.

DKIM signing (OpenDKIM)

Install OpenDKIM and tools:

sudo apt -y install opendkim opendkim-tools

Create directories:

sudo mkdir -p /etc/opendkim/keys/example.com
sudo chown -R opendkim:opendkim /etc/opendkim
sudo chmod go-rwx /etc/opendkim/keys

Generate a key (use a selector like smtp):

sudo -u opendkim opendkim-genkey -b 2048 -d example.com -D /etc/opendkim/keys/example.com -s smtp
sudo -u opendkim mv /etc/opendkim/keys/example.com/smtp.private /etc/opendkim/keys/example.com/smtp.key

Configure OpenDKIM:

sudo nano /etc/opendkim.conf

Syslog                  yes
UMask                   002
Mode                    sv
Canonicalization        relaxed/simple
KeyTable                /etc/opendkim/key.table
SigningTable            /etc/opendkim/signing.table
ExternalIgnoreList      /etc/opendkim/trusted.hosts
InternalHosts           /etc/opendkim/trusted.hosts
Socket                  inet:12301@localhost

Create the tables:

sudo nano /etc/opendkim/trusted.hosts

127.0.0.1
localhost
mail.example.com

sudo nano /etc/opendkim/key.table

smtp._domainkey.example.com example.com:smtp:/etc/opendkim/keys/example.com/smtp.key

sudo nano /etc/opendkim/signing.table

*@example.com smtp._domainkey.example.com

Enable and start OpenDKIM:

sudo systemctl enable --now opendkim

Now connect Postfix to the milter. Add to /etc/postfix/main.cf:

sudo postconf -e 'milter_default_action = accept'
sudo postconf -e 'milter_protocol = 6'
sudo postconf -e 'smtpd_milters = inet:localhost:12301'
sudo postconf -e 'non_smtpd_milters = inet:localhost:12301'
sudo systemctl restart postfix

Publish the DKIM DNS record. Print the public key:

sudo cat /etc/opendkim/keys/example.com/smtp.txt

Create a TXT record for smtp._domainkey.example.com using the value shown (it starts with v=DKIM1;).

DMARC record

Add this TXT record at _dmarc.example.com:

v=DMARC1; p=none; rua=mailto:dmarc-reports@example.com; adkim=s; aspf=s

Once you’ve confirmed alignment and you’re comfortable with the results, move to p=quarantine or p=reject.

If you want a cPanel-centric walkthrough for SPF/DKIM/DMARC concepts and validation steps, read our SPF/DKIM/DMARC setup guide. The DNS logic is the same even without cPanel.

Step 9 — Add practical rate limits (your seatbelt)

Rate limiting isn’t a “nice to have.” One compromised WordPress plugin can dump thousands of messages through your relay in minutes.

Add conservative defaults to /etc/postfix/main.cf:

# Slow down abusive bursts
anvil_rate_time_unit = 60s
smtpd_client_connection_rate_limit = 30
smtpd_client_message_rate_limit = 60
smtpd_client_recipient_rate_limit = 200

# Avoid unlimited parallelism
smtpd_client_connection_count_limit = 20

Reload Postfix:

sudo systemctl reload postfix

Adjust these numbers based on real traffic. For many small businesses, 60 messages per minute per authenticated client is already plenty.

Step 10 — Test sending end-to-end (and read the logs)

Run an authenticated SMTP test first. Do this before you wire up apps and forms.

Install swaks:

sudo apt -y install swaks

Run a test (replace addresses and credentials):

swaks --to you@gmail.com \
  --from test@example.com \
  --server mail.example.com \
  --port 587 \
  --auth LOGIN \
  --auth-user smtpuser \
  --auth-password 'YOUR_PASSWORD' \
  --tls

Watch the logs while you test:

sudo tail -f /var/log/mail.log

If you see bounces that mention HELO/EHLO or hostname, don’t “tune” filters yet. Fix the identity layer first.

Start with hostname, PTR, and A record alignment. Then retest.

For the most common “Bad HELO” fixes, follow this HELO/EHLO hostname fix tutorial.

Step 11 — Keep your relay healthy: queues, retries, and quick diagnostics

Even a clean setup will queue mail during provider outages, DNS issues, or temporary blocks.

These commands cover most first-pass triage:

Show queue: mailq or postqueue -p
Flush queue: postqueue -f
Show Postfix config diffs: postconf -n
Check DNS from the server: dig TXT example.com, dig TXT smtp._domainkey.example.com

If the queue keeps growing and you need a repeatable process for deferrals and stuck mail, see our VPS email queue troubleshooting tutorial.

Step 12 — Operational hardening: updates, backups, and access control

A mail relay is a high-value target. Treat it like production infrastructure, not a one-time setup.

Patch cadence

Apply security updates weekly. Patch quickly after any OpenSSL/Postfix advisory.

Unattended-upgrades can handle the basics. You should still review what changed.

Back up the only things that matter

/etc/postfix/
/etc/opendkim/ (especially private keys)
Let’s Encrypt: /etc/letsencrypt/

Store encrypted copies offsite. If you want a practical “restore drill” routine, use this backup verification tutorial to set up repeatable testing.

SSH access discipline

Use SSH keys and limit who can log in. If you manage multiple servers, a jump host setup is often worth it.

HostMyCode customers often pair an outbound mail relay with managed VPS hosting when they want help with patching, monitoring, and incident response.

Common pitfalls (read this before you go live)

No rDNS: many receivers penalize or reject outright. Set PTR to your mail hostname.
SPF “+all”: this is basically permission for anyone to spoof you. Use -all or ~all.
DKIM key copied wrong: extra quotes, line breaks, or missing semicolons break verification.
Relay open to the world: if unauthenticated users can relay, you’ll be blacklisted quickly.
No rate limits: compromise turns into a reputation incident instead of a contained problem.

Summary: a production-ready relay you can operate

You now have a Postfix SMTP relay with authenticated submission, TLS, DKIM signing, and DNS alignment (SPF/DMARC/rDNS).

From here, the work is operational. Watch logs, keep the server patched, and keep rate limits in place as basic damage control.

If you want a stable platform for email and web workloads—without sharing resources with noisy neighbors—run this on a HostMyCode VPS.

If you’d rather not own the maintenance burden, managed VPS hosting can handle routine upkeep while you keep control of your sending domain.

If you’re putting email into production, start with infrastructure you can actually control: a predictable IP, clean DNS, and root access for auditing and troubleshooting. A HostMyCode VPS is a solid base for an outbound relay, and managed VPS hosting is available if you want help with hardening, updates, and monitoring.

FAQ

Do I need to open inbound port 25 for an SMTP relay?

Not always. If your only goal is authenticated submission (587/465) for your apps and sites, you can keep 25 closed. Outbound delivery still uses port 25 from your server to recipient servers.

How do I know if DKIM is actually signing?

Send a test email to a mailbox that shows “original message” headers. You should see a DKIM-Signature header and an Authentication-Results line showing dkim=pass.

What’s the minimum DNS set for decent deliverability?

PTR/rDNS aligned to your hostname, an A record for that hostname, SPF for your sending IP, DKIM for the domain, and a basic DMARC policy (p=none to start).

Can I use this relay with WordPress?

Yes. Use an SMTP plugin that supports STARTTLS on port 587, authenticate with your relay user, and set the “From” domain to one you’ve configured with SPF/DKIM/DMARC.

What should I do if my IP is already on a blacklist?

First stop the source of spam (compromised app, stolen credentials, open relay). Then fix DNS alignment and rate limits. Only after that should you request delisting, otherwise you’ll be listed again.