SMTP TLS Troubleshooting Tutorial (2026): Fix STARTTLS, Certificate, and Port 587 Issues on a Hosting VPS

Your mail server can look “fine” and still fail where it matters: the TLS handshake. The symptoms are familiar. Outlook won’t send, Gmail bounces with “TLS required,” or submission on port 587 times out with no obvious clue. This SMTP TLS troubleshooting tutorial gives you a repeatable, command-first workflow to find the break and fix it on a hosting VPS.

The examples assume Postfix on Ubuntu 24.04 LTS or Debian 12/13. The same tests apply to most Linux mail stacks.

If the VPS also runs websites, schedule changes during a quiet window. TLS changes and service reloads can interrupt active SMTP sessions.

What you’re troubleshooting (and why it breaks on VPS mail)

SMTP encryption usually shows up in one of two patterns:

STARTTLS on port 587 (submission): the client connects in plain text, then upgrades to TLS.
Implicit TLS on port 465 (smtps): TLS starts immediately.

In 2026, most client and deliverability issues still trace back to submission (587). Common causes include an incomplete certificate chain, a hostname mismatch, strict TLS policies, client-side SNI quirks, or a firewall rule that quietly blocks the port.

You can isolate the cause quickly with openssl s_client and a few targeted Postfix checks.

Prerequisites and safe defaults

Root or sudo access to the VPS
A real hostname for mail, typically mail.example.com
Correct DNS: A/AAAA for mail.example.com, and ideally PTR (rDNS) for the VPS IP
Ports open as needed: 587 for submission, 465 optional, 25 for server-to-server (often restricted by providers)

If you haven’t validated PTR yet, do it now. You’ll retest anyway, and rDNS issues create noisy false negatives.

Use this guide: Reverse DNS setup tutorial.

Hosting note: if you want a clean baseline VPS for mail + web, start fresh. Harden it before you pile on services.

HostMyCode’s HostMyCode VPS plans are a good fit when you need predictable networking and full control over SMTP.

Step 1: Confirm ports are actually reachable (from outside)

Run this from a different network (a laptop on a mobile hotspot works). Check reachability before you touch TLS:

nc -vz mail.example.com 587
nc -vz mail.example.com 465
nc -vz mail.example.com 25

If you get timeouts, stop there. You’re not debugging TLS yet.

You’re debugging basic connectivity.

Usual culprits:

Provider blocks inbound/outbound SMTP ports
UFW/iptables/nftables blocks
Postfix listening on the wrong interface

On the server, confirm Postfix is listening:

sudo ss -lntp | egrep ':(25|465|587)\b'

Then check firewall rules (Ubuntu UFW example):

sudo ufw status verbose

If you want a safe firewall baseline that doesn’t accidentally break mail, follow: UFW firewall configuration tutorial.

Step 2: Run a real STARTTLS handshake test (the fastest truth)

This is the quickest way to see what the server presents to clients. It exposes protocol support, the certificate chain, and verification status.

openssl s_client -starttls smtp -connect mail.example.com:587 -servername mail.example.com -showcerts

What a healthy result looks like:

CONNECTED appears quickly (no long stall)
Verify return code: 0 (ok)
The certificate CN or SAN includes mail.example.com

Common failures and what they usually point to:

verify error:num=20:unable to get local issuer certificate → missing intermediate(s); the server isn’t sending a complete chain
verify error:num=62:Hostname mismatch → the cert doesn’t match the hostname you connected to
Handshake works, but clients still fail → policy mismatch (TLS versions/ciphers), SNI behavior, or the client is using 465 vs 587 incorrectly

Step 3: Identify your mail TLS certificate paths (Postfix + Dovecot)

On a typical Postfix + Dovecot host, you usually have two places to validate TLS:

Postfix for SMTP submission and server-to-server SMTP
Dovecot for IMAP/POP3 (not the focus here, but keep certs consistent)

Start with Postfix. Pull the active TLS-related settings:

sudo postconf -n | egrep '^(smtpd_tls_|smtp_tls_|smtpd_tls_security_level|smtpd_tls_cert_file|smtpd_tls_key_file|smtpd_tls_CAfile|smtpd_tls_CApath)'

On most VPS deployments, you want:

smtpd_tls_cert_file pointing to a full chain file
smtpd_tls_key_file pointing to the private key
smtpd_tls_security_level = may for inbound SMTP (port 25)
smtpd_tls_security_level = encrypt for submission when you require TLS (usually via submission overrides)

For Let’s Encrypt, these paths are typically correct:

/etc/letsencrypt/live/mail.example.com/fullchain.pem
/etc/letsencrypt/live/mail.example.com/privkey.pem

If you need to install/renew Let’s Encrypt safely on a host running web + mail, use: SSL setup guide tutorial.

Issue the certificate for mail.example.com.

Step 4: Fix the #1 issue: incomplete certificate chain

Mail clients are often stricter than browsers. If openssl s_client reports issuer errors, the server is probably sending only the leaf certificate.

Clients expect the full chain.

With Let’s Encrypt, Postfix should almost always use fullchain.pem (not cert.pem). Set the paths and reload:

sudo postconf -e 'smtpd_tls_cert_file=/etc/letsencrypt/live/mail.example.com/fullchain.pem'
sudo postconf -e 'smtpd_tls_key_file=/etc/letsencrypt/live/mail.example.com/privkey.pem'
sudo systemctl reload postfix

Retest and focus on subject/issuer plus the verify code:

openssl s_client -starttls smtp -connect mail.example.com:587 -servername mail.example.com | tail -n +1 | egrep 'Verify return code|subject=|issuer='

If verification still fails, check file access. Keep private keys root-only.

Postfix can read them as needed without you loosening permissions across the system.

Step 5: Fix hostname mismatch (cert SAN vs MX vs submission hostname)

Submission should use a stable hostname. The certificate must match that exact name.

In practice, that’s usually mail.example.com with an A/AAAA record pointing to the VPS.

Your MX can point to mail.example.com too, but it doesn’t have to.

Confirm Postfix is advertising the name you expect:

sudo postconf myhostname
sudo postconf smtpd_banner

If needed, set myhostname to your mail hostname and reload:

sudo postconf -e 'myhostname=mail.example.com'
sudo systemctl reload postfix

Then verify the certificate includes the hostname (check SANs):

openssl x509 -in /etc/letsencrypt/live/mail.example.com/fullchain.pem -noout -text | sed -n '/Subject:/,/X509v3 Subject Alternative Name:/p'
openssl x509 -in /etc/letsencrypt/live/mail.example.com/fullchain.pem -noout -text | sed -n '/X509v3 Subject Alternative Name/,+2p'

If mail.example.com isn’t in the SAN list, reissue the certificate for the correct name.

Step 6: Ensure submission (587) enforces auth + TLS (and doesn’t behave like port 25)

Many “TLS problems” are really service-definition problems. Port 587 is for authenticated users.

It should not behave like inbound SMTP from the open internet.

In Postfix, submission settings live in /etc/postfix/master.cf. You want something along these lines:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

After edits, validate and reload:

sudo postfix check
sudo systemctl reload postfix

Retest STARTTLS. Then confirm the server advertises the right capabilities on submission:

openssl s_client -starttls smtp -connect mail.example.com:587 -servername mail.example.com -crlf

Type:

EHLO test.example

You should see 250-STARTTLS. After negotiating TLS, you should see AUTH methods.

Step 7: Fix TLS version / cipher mismatch without breaking older clients

In 2026, TLSv1.2 and TLSv1.3 are the practical baseline for submission. If you lock down too aggressively, older devices fail.

If you leave legacy protocols enabled, you invite policy and compliance problems.

Check what you enforce today:

sudo postconf -n | egrep 'tls_(protocols|mandatory_protocols|high_cipherlist|medium_cipherlist|ciphers)'

A reasonable default for many VPS setups is to disable TLSv1.0/1.1 and SSL. Keep TLSv1.2/1.3 available:

sudo postconf -e 'smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3,!TLSv1,!TLSv1.1'
sudo postconf -e 'smtpd_tls_protocols=!SSLv2,!SSLv3,!TLSv1,!TLSv1.1'
sudo postconf -e 'smtpd_tls_ciphers=medium'
sudo systemctl reload postfix

If a specific client still fails, test what the server accepts from a workstation:

openssl s_client -starttls smtp -connect mail.example.com:587 -tls1_2
openssl s_client -starttls smtp -connect mail.example.com:587 -tls1_3

If TLSv1.3 fails but TLSv1.2 succeeds, your OpenSSL/Postfix build (or a policy setting) may be limiting TLSv1.3.

Keep TLSv1.2 enabled. It remains secure and widely compatible when configured correctly.

Step 8: Verify the correct certificate is used (SNI and multi-IP pitfalls)

On a VPS hosting multiple domains, it’s easy to present the wrong certificate. SMTP can use SNI, but not every client sends it.

The safest pattern is one stable submission hostname with one certificate.

Compare the presented certificate with and without SNI:

openssl s_client -starttls smtp -connect mail.example.com:587 -showcerts
openssl s_client -starttls smtp -connect mail.example.com:587 -servername mail.example.com -showcerts

If the certificate changes depending on -servername, some clients will see the “wrong” cert.

In that situation, stick to a single submission hostname. Make sure Postfix always serves that certificate.

Step 9: Check logs that actually matter (and turn up the right verbosity)

On Ubuntu/Debian, keep the mail log open while you run handshake tests:

sudo tail -f /var/log/mail.log

These lines often tell you exactly what broke:

SSL_accept error (server-side TLS negotiation failure)
lost connection after STARTTLS (client drops during the handshake)
warning: TLS library problem (often protocol/cipher policy issues)

If you’re dealing with slow delivery, deferrals, or queue growth, use a queue-focused workflow instead: Mail queue troubleshooting tutorial.

Step 10: Confirm authentication path isn’t confusing clients (Submission vs relay)

Mail apps often label failures as “TLS errors” when the handshake is fine and authentication fails afterward.

To separate those cases, test with swaks from a controlled environment:

sudo apt-get update
sudo apt-get install -y swaks

Then run an authenticated submission test (replace credentials):

swaks --to you@example.com --from you@example.com \
  --server mail.example.com --port 587 \
  --auth LOGIN --auth-user 'you@example.com' --auth-password 'YOURPASSWORD' \
  --tls --tls-verify

If --tls-verify fails, go back to certificate/hostname/chain.

If TLS verification passes but AUTH fails, inspect your SASL integration (Dovecot, Cyrus, etc.). Also confirm you didn’t disable the auth mechanisms your client expects.

Quick checklist: common fixes you can apply in 15 minutes

Submission uses fullchain.pem, not cert.pem
Certificate SAN includes mail.example.com
myhostname matches the mail hostname
Port 587 has smtpd_tls_security_level=encrypt and SASL enabled
Allow TLSv1.2 and TLSv1.3; disable TLSv1.0/1.1 and SSLv2/3
UFW/provider allows inbound 587 from your user base

Hardening extras that improve real-world outcomes (without turning this into a science project)

Once TLS is stable, tighten the rest of the setup. That reduces the chance you’ll revisit this under pressure.

SPF/DKIM/DMARC aligned with your sending domain
rDNS matches your mail hostname
Rate limits for auth attempts and abusive clients (especially on submission)

For authentication records, follow: Email authentication setup tutorial.

If you need temporary access to admin panels and services while you work, an SSH tunnel is safer than opening new public ports: SSH port forwarding tutorial.

When to choose managed help (and when not to)

If email is revenue-critical (business operations, a reseller platform, or a busy app), small TLS mistakes can create silent failures for hours.

Managed help makes sense when you want someone to watch logs, test from multiple networks, and validate DNS and certificates end to end.

For hands-off operations, consider managed VPS hosting.

If you’re comfortable running Postfix but want predictable compute and networking, a standard HostMyCode VPS is a solid base.

If you want SMTP submission that stays stable through renewals, migrations, and policy changes, run it on a VPS you control. HostMyCode offers VPS hosting for mail + web stacks, and managed VPS hosting if you’d rather hand off the day-to-day TLS and mail troubleshooting.

FAQ

Should I use port 465 or 587 for sending mail?

Use port 587 for authenticated submission unless you have a client that only supports 465. Get 587 stable first. It’s the default for most modern mail apps.

Why does `openssl s_client` say “verify error” but email still sends sometimes?

Some clients accept an untrusted chain (or rely on cached trust decisions). Others fail immediately.

Fix the chain so verification returns code 0. The “random” client behavior usually disappears.

What’s the quickest way to confirm I fixed the right thing?

Run openssl s_client -starttls smtp against port 587 and confirm Verify return code: 0 (ok). Then send an authenticated test with swaks --tls --tls-verify.

Do I need rDNS (PTR) for TLS to work?

No. TLS works without PTR. Deliverability and trust signals often improve when PTR aligns with your HELO and hostname.

Can a website SSL certificate be reused for SMTP TLS?

Yes—if it covers the correct hostname (usually mail.example.com) and you install the full chain in Postfix.

Don’t reuse a cert that only covers www unless clients connect using that name (they shouldn’t).

Summary: your repeatable SMTP TLS fix workflow

Prove port reachability first. Then validate STARTTLS on 587 with openssl s_client.

Fix chain and hostname issues before you change anything else.

After that, tune TLS versions. Confirm port 587 is configured as submission (auth + TLS), not a clone of port 25.

If you’re building a dependable mail + hosting stack in 2026, do it on infrastructure you can test and control. HostMyCode’s Affordable & Reliable Hosting VPS plans give you the access you need for proper TLS diagnostics, with the option to move to managed VPS hosting when you want the operational overhead off your plate.