DEV Community

Cover image for How to Self-Host Your Own Email Server (And Stop Depending on Third Parties)
Alan West
Alan West

Posted on

How to Self-Host Your Own Email Server (And Stop Depending on Third Parties)

#selfhosted #email #devops #linux

So you've been burned by an email hosting provider. Maybe they changed their pricing, maybe their support went sideways, or maybe you just woke up one morning and realized that trusting a critical piece of your infrastructure to a company you can't control is a risk you're no longer comfortable with.

Whatever your reason, self-hosting email is one of those tasks that has a reputation for being nightmarish. And honestly? Parts of it are tricky. But in 2024, the tooling has gotten good enough that a developer with some Linux experience can get a reliable mail server running in an afternoon.

Let me walk you through how I did it, what broke, and how I fixed it.

Why Self-Hosting Email Is Hard (But Not Impossible)

The actual software setup isn't the hard part. The hard part is deliverability — making sure your emails actually land in inboxes instead of spam folders. The big providers (Gmail, Outlook, Yahoo) are aggressively skeptical of mail from unknown servers, and for good reason.

Here's what you're up against:

  • Your IP address needs a clean reputation
  • You need proper DNS records (SPF, DKIM, DMARC)
  • Reverse DNS (PTR record) must match your mail server hostname
  • Your VPS provider needs to allow outbound port 25
  • You need TLS configured correctly

Miss any one of these, and your emails vanish into the void. No bounce, no error — just silence.

Step 1: Choose Your Stack

I went with Mailcow — it's a dockerized mail server suite that bundles Postfix, Dovecot, Rspamd, SOGo, and a web UI. There are other solid options like Mail-in-a-Box or rolling your own with Postfix + Dovecot, but Mailcow hits the sweet spot between control and convenience.

# Clone and set up Mailcow on a fresh Ubuntu/Debian VPS
git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized

# Generate the config — it'll ask for your mail hostname
./generate_config.sh

# Fire it up
docker compose pull
docker compose up -d
Enter fullscreen mode Exit fullscreen mode

Before you even think about running this, make sure your VPS provider doesn't block port 25. Some do by default (looking at you, most cloud providers). You may need to submit a support ticket to get it unblocked. I'd recommend checking providers like Hetzner or OVH that are more mail-friendly.

Step 2: DNS — The Part Everyone Gets Wrong

This is where most self-hosted mail setups die. You need four DNS records configured correctly, and each one serves a different purpose.

; MX record — tells the world where to deliver mail for your domain
example.com.    IN  MX  10  mail.example.com.

; A record — points your mail hostname to your server IP
mail.example.com.  IN  A  203.0.113.42

; SPF record — declares which servers can send mail for your domain
example.com.    IN  TXT  "v=spf1 mx a -all"

; DMARC record — tells receivers what to do with mail that fails checks
_dmarc.example.com.  IN  TXT  "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com"
Enter fullscreen mode Exit fullscreen mode

And then there's DKIM, which is a cryptographic signature added to every outgoing email. Mailcow generates this for you automatically — you just need to copy the public key into a DNS TXT record.

The one people forget? The PTR record (reverse DNS). This has to be set on your VPS provider's control panel, not your domain registrar. It should resolve your server's IP back to mail.example.com.

# Verify your PTR record is set correctly
dig -x 203.0.113.42 +short
# Should return: mail.example.com.
Enter fullscreen mode Exit fullscreen mode

If this doesn't match, Gmail will silently trash your emails. I spent two hours debugging deliverability issues before realizing my PTR record was still pointing to the default VPS hostname.

Step 3: Debugging Deliverability

You've got everything running. You send a test email to your Gmail account. It doesn't arrive. Now what?

First, check your mail logs:

# If using Mailcow/Docker
docker compose logs --tail=100 postfix-mailcow

# Look for lines like these:
# status=deferred (host gmail-smtp-in.l.google.com said: 421 try again later)
# status=bounced (550 5.7.1 rejected by recipient domain)
Enter fullscreen mode Exit fullscreen mode

Common issues and fixes:

  • "421 try again later" — Your IP reputation is too new. Send a few emails to yourself first, mark them as "not spam," and wait a day or two. Warming up is real.
  • "550 rejected" — Check your SPF and DKIM. Use mail-tester.com to get a detailed score breakdown.
  • Emails land in spam — Usually a DMARC or DKIM alignment issue. Make sure your From: domain matches the domain in your DKIM signature.

Here's a quick script I use to validate my setup:

#!/bin/bash
DOMAIN="example.com"
MAIL_HOST="mail.${DOMAIN}"

echo "=== Checking MX ==="
dig MX $DOMAIN +short

echo "=== Checking SPF ==="
dig TXT $DOMAIN +short | grep spf

echo "=== Checking DKIM ==="
# Replace 'dkim' with your actual DKIM selector
dig TXT dkim._domainkey.${DOMAIN} +short

echo "=== Checking DMARC ==="
dig TXT _dmarc.${DOMAIN} +short

echo "=== Checking PTR ==="
SERVER_IP=$(dig A $MAIL_HOST +short)
dig -x $SERVER_IP +short

echo "=== Checking TLS on port 587 ==="
openssl s_client -starttls smtp -connect ${MAIL_HOST}:587 < /dev/null 2>&1 | grep "Verify return code"
Enter fullscreen mode Exit fullscreen mode

Step 4: Backups and Maintenance

Self-hosting means you're on the hook for backups. Don't learn this lesson the hard way.

  • Set up automated daily backups of your mail directory and database
  • Monitor disk space — mailboxes grow faster than you think
  • Keep your server updated — Postfix and Dovecot vulnerabilities are high-value targets
  • Use fail2ban or similar to block brute-force login attempts 
# Simple backup script for Mailcow
#!/bin/bash
BACKUP_DIR="/opt/mailcow-backups"
cd /opt/mailcow-dockerized

# Mailcow includes a backup helper
./helper-scripts/backup_and_restore.sh backup all --delete-days 7
Enter fullscreen mode Exit fullscreen mode

Prevention: When to Self-Host (And When Not To)

I'll be honest — self-hosting email isn't for everyone. Here's my mental checklist:

  • Do self-host if: You're running infrastructure for a small team, you value data ownership, or you're a homelab enthusiast who enjoys this stuff
  • Don't self-host if: You're sending high-volume transactional email (use a dedicated sending service), you can't commit to monitoring, or you don't have a static IP

The biggest ongoing cost isn't money — it's attention. A mail server that goes down at 2 AM means missed emails, and unlike a web app, people expect email to just work.

The Takeaway

Self-hosting email in 2024 is a solved problem from a technical standpoint. Tools like Mailcow have turned what used to be a week-long sysadmin project into a docker compose session. The real challenge is deliverability and ongoing maintenance.

But here's the thing — once you get it working, it's incredibly satisfying. You own your data, you control your infrastructure, and you'll never have to worry about a third-party provider's business decisions affecting your communication.

Just make sure your PTR record is set. Trust me on that one.

Top comments (0)