Built My Own SMTP Mail Server using iRedMail

I Built My Own SMTP Mail Server (and Learned Why Email Is Hard)
Email feels boring. It’s just… there. You sign up for a service, you get a verification link, life moves on. But at some point I realized something embarrassing for someone who likes networking and security:

I had never actually built an email server.

So I decided to fix that.

This post is about how I built a fully working SMTP mail server from scratch on a VPS, wired it to my own domain, fought DNS demons, broke TLS configs, landed in Gmail spam, and eventually saw that beautiful moment where an external reply landed back in my inbox.

This wasn’t about production readiness or scaling.
This was about learning how email actually works under the hood.

The Why: “I Should Probably Know This”
If you’re into backend, security, or infrastructure, email keeps popping up:

Password resets

Alerts

Phishing detection

SPF / DKIM / DMARC checks in security tools

Mail logs in incident response

And yet, most of us just treat it as a black box.

I wanted to understand

What actually happens when you hit “Send”

Why emails get marked as spam

Why everyone warns you: “Don’t run your own mail server”

So I did exactly that. 😅

What I Built (High-Level)
I set up a real SMTP mail server for my domain:

Domain: vibinsec.live

Mail host: mail.vibinsec.live

Stack: iRedMail on Ubuntu (Postfix + Dovecot + friends)

Webmail: Roundcube

Admin panel: iRedAdmin

TLS: Let’s Encrypt

Auth: SPF, DKIM, DMARC

Reverse DNS (PTR): Configured at the VPS provider

And most importantly:

✅ Sent mail to Gmail
✅ Received replies back
✅ End-to-end email flow actually worked

No mock servers. No localhost-only magic.

The Architecture (a.k.a. “So Many Moving Parts”)
Here’s what’s really running when you say “mail server”:

Postfix – handles SMTP (sending & receiving mail)

Dovecot – handles IMAP (reading mail)

Amavis / DKIM tools – signs outgoing mail

Nginx – serves Roundcube and admin panels

DNS – A, MX, TXT records doing half the trust work

TLS – not just for HTTPS, but SMTP and IMAP too

Nothing here is conceptually insane.
But the integration is where the pain lives.

The Juicy Part: The Struggles
This project humbled me. Repeatedly.

1. DNS Is Where Hope Goes to Die Setting up:

A record

MX record

SPF

DKIM

DMARC

…sounds straightforward until you realize:

One typo = silent failure

Propagation delays make debugging awful

Every provider has a slightly different UI

I spent a lot of time thinking something was broken when it was just DNS not updating yet.

Skill gained:
Real-world DNS troubleshooting and patience I didn’t know I needed.

1. Reverse DNS (PTR) Is Not Optional I learned this the hard way.

Everything looked right.
SPF? Good.
DKIM? Signed.
TLS? Enabled.

Still landing in spam.

Turns out:

If your server IP doesn’t reverse-resolve to your mail hostname, many providers don’t trust you.

Configuring PTR records meant going into the VPS provider’s dashboard, not my domain registrar. That mental context switch was easy to miss.

Lesson: Email trust is reputation-based and unforgiving.

1. TLS Everywhere (Not Just the Website) I thought:

“Cool, Let’s Encrypt for HTTPS. Done.”

Nope.

Mail servers need TLS for:

SMTP (Postfix)

IMAP (Dovecot)

Webmail (Nginx)

That meant manually pointing each service to the right cert paths and restarting everything without breaking it.

One wrong file path = service won’t start.

Skill gained:
Reading logs instead of guessing.
Seriously, mail logs are your lifeline.

1. DKIM: Cryptography Meets Copy-Paste Fear DKIM was one of those moments where you know it’s important, but it still feels sketchy:

Extract public key from the server

Add it as a TXT record

Hope nothing is truncated

Wait for DNS

Test again

When it finally validated, I actually felt relief.

Skill gained:
Understanding how cryptographic trust is applied at the protocol level, not just theory.

1. Spam Happens (Even When You Do Everything Right) My first Gmail test landed in spam.

Honestly? That was expected.

New domain.
New IP.
Zero reputation.

This taught me something important:
Correct configuration doesn’t guarantee trust. Time does.

What I Actually Learned (Hard + Soft Skills)
Hard Skills
SMTP, IMAP, and how they interact

Postfix and Dovecot basics

DNS beyond “add an A record”

SPF / DKIM / DMARC in practice

TLS for non-HTTP services

Reading and trusting system logs

Soft Skills
Debugging layered systems

Slowing down instead of randomly changing configs

Accepting that some problems just need time (DNS, reputation)

Not panicking when email disappears into the void

Proof It Worked (The Best Part)
The moment that mattered:

Sent an email from Roundcube

Saw it arrive externally

Got a reply

Watched it land back in my inbox

That’s when it clicked.

“Oh. This is the entire email system. I built this.”

Would I Run This in Production?
Absolutely not. 😄
Running mail servers long-term is a responsibility I respect a lot more now.

But as a learning project?

This was gold.

What I’d Improve Next
If I take this further:

Automated TLS renewals + service reloads

Better spam filtering visibility

Monitoring and alerting on mail queues

IPv6 tuning and reputation management

Final Thoughts
If you’re into backend, infra, or security and you’ve never built a mail server, I honestly recommend doing it once.

Not because it’s fun (it’s not).
Not because it’s easy (it isn’t).

But because email touches everything, and understanding it makes you a better engineer.

If you want the full step-by-step lab with screenshots, configs, and commands, the full README is here:
👉 GitHub: https://github.com/VibhavChennamadhava/Designing-an-SMTP-Mail-Server-Using-iRedMail

If you try this yourself:
Good luck.
Read the logs.
And don’t forget the PTR record. 😄