DEV Community

Cover image for How I accidentally learned email infrastructure while trying to avoid a SendGrid bill
TymoteuszGluch
TymoteuszGluch

Posted on

How I accidentally learned email infrastructure while trying to avoid a SendGrid bill


If you have ever worked on a side project or an open source project, you probably know that feeling. You spend hours trying to reduce costs, comparing providers, looking for ways to save at least one dolar more.
But not becouse you have to, becouse that thought of paying for something, you may probably host yourself for free feel offensive.

That exacly how i ended with my own self-hosted mail server.


Why am I paying for this when I can host it myself?

Before I started working on it, i thought it couldn't be that hard. I probably only need to buy a server to store data, add a simple Nginx1 config, buy a domain, and set up a few common DNS records2.

After all, I just wanted to stop paying for an email delivery service and host everything myself.

Then the world introduced me to SPF3, DKIM4, DMARC5, and many other strange acronyms that I had never heard of before.


The hidden subscription fee: your time and your RAM

Mission: Find place to storage mail server

First of all, I had to find a server as cheap as possible, and there were a few possibilities, like buying a physical server or using a VM provider6, which is probably a better option for scaling.

For example, Oracle has one of the most affordable VM-s7, but in my experience their technical support is terrible. If you're not going to have any problems, then it might be an option for you.

In my opinion, the best VM7 provider is Hetzner, which is probably the only option where you don't pay for transfer, and the bill is based on the time your server is running.

Mission: Find the most afordable, but trusted domain provider

Sounds simple, but then internet introduced me to strange words like SPF3, DKIM4 and DMARC5. I had no idea what those acronyms means, but I quickly found out that I needed all them. Without correct configuration, Mail providers will reject emails coming from a your brand new domain.

In my case, I chose Cloudflare. Their pricing is reasonable, DNS management is excellent, and they make it relatively easy to configure all the records needed for a mail server.

Mission: Start to Setup my first Mail Service

At some point, I had to stop reading articles and actually choose a mail server. One of the most popular open-source solutions is Mailcow Dockerized8, and for good reason. It comes with a web interface, account management, built-in Nginx, rate limiting, authentication, and most of the things you would expect from a modern mail service and it also has a simple setup script

Unfortunately, Mailcow is focused on managing email and don't have email templates. In my case, I still needed a separate backend service responsible for storing templates, generating emails, and communicating with the mail server.

Mission: Add some infrastructure

At this point, I needed more than just a mail server. I also wanted to run my own application responsible for managing email templates and communicating with Mailcow. There was only one problem. Mailcow was already using port 4439. As usual, the simple idea turned into a networking problem. To make everything work together, I had to override part of Mailcow's Nginx configuration, move its services behind localhost10, and place another Nginx instance in front of everything. The result was one public HTTPS entry point routing traffic to the right service.

I'm still working on this setup. Right now, it includes the basic infrastructure needed for my application, and I'm experimenting with mTLS11 and VPN-only12 access for administrative endpoints. Because apparently running a mail server wasn't enough infrastructure for one project.


The moment I realized SendGrid wasn't expensive

When i started this project, i felt all popular mail services are to expensive, then i go to work, and it turned out that the math wasn't on my side.

Only my Mailcow instance already consumes almost 4 GB of RAM while do nothing. No users, whos need to recive mails, only exitsting. Its work but its doesnt leave place to growth. I'd like to add proper monitoring with Grafana13, Prometheus14 and Loki15. Of course, I also need backups, but even now, while using a very cheap VPS, I'm already paying around $10/month.

Then there is IP reputation. Sometimes a cheap VPS comes with an IP address that has seen things. Things you would rather not know about and if you're unlucky, getting a different IP address may cost extra. Fortunately, I ended up with an IP address that already had a good reputation.


That day, I learned how spammers ruined SMTP for everyone else

After setting up my mail server, I thought, I was ready to send my first email. There was only one problem. I couldn't.

As it turns out, that a lot of VPS providers block ports 25 and 465 by default. At first, I thought it was some annoying limitation. Then I discovered it was the result of years of abuse, spam campaigns, and providers trying to protect their IP reputation.

how scammers use cheap VM-s7 to spam everyone

Unfortunately, not only developers are interesting in cheap VPS. Scammers often use stolen credit cards, fake identities and compromised accounts to buy large numbers of servers. They then use those servers to send millions of spam emails, phishing campaigns, and other messages nobody asked for.

From a VPS provider's perspective, a brand-new mail server is statistically more likely to be a spam machine than an honest developer trying to save a few dollars. Which, unfortunately, explains why my perfectly innocent mail server was treated like a criminal from day one.

3 potentialy solution

At this point, I had three options:

  • Use a trusted popular SMTP16 service for outgoing emails. - Congratulations, you are now paying for the thing, that you where trying to avoid paying for.

  • Wait - Many VPS providers become more trusting after a few weeks. This may take approximately 30 days.

  • Prove that you're not a impostor. - configure your DNS records, set up your mail server properly, and send reque for the ports to be unblocked. If everything looks legitimate, some providers will approve the request much sooner.

I went with option three. Mostly because after spending all that time setting up my own mail server, giving up felt emotionally unacceptable.


The day I realized my mail server needed adult supervision

Getting a mail server running is only half the battle. The other half is making sure it doesn't suddenly start sending emails on behalf of someone else.

The first thing I would recommend is monitoring. Grafana13, Prometheus14 and Loki15 can help you spot unusual activity, such as a sudden spike in outgoing emails at 3 AM. If that happens and it wasn't you, congratulations - you've found a problem before the rest of the Internet does.

The second thing is protecting your APIs. If someone wants to abuse your mail server, your application's email endpoints are probably the easiest target. A VPN12 can be a simple solution for internal tools, while mTLS provides stronger protection if you're willing to deal with certificates and the occasional headache they bring.

Mutual TLS11 can be surprisingly tricky with self-hosted solutions like Mailcow. Updates, custom Nginx configurations, reverse proxies17 and certificate management all add extra complexity, But that not only problem, becouse thats all can be owerriten if you pull latest Mailcow changes

Finally, keep an eye on your IP reputation18. A perfectly configured mail server with a bad reputation is still a perfectly configured mail server whose emails land in spam.

And, of course, backups. Because every self-hosting story eventually ends with someone explaining why they should have had backups.


Was it worth it?

Probably not.

Was it fun?

Absolutely.

Would I do it again?

Unfortunately, yes.


The code I didn't plan to write

One unexpected side effect of this project was that I ended up writing much more code than I originally planned. What started as "I don't want to pay for SendGrid" eventually turned into a couple of open-source repositories that I'm still actively working on:

  • Morphyxis-Infra - infrastructure and deployment configuration used to run the platform.
  • Morphyxis-Mail-Service - an experimental service for email template management, rendering and delivery.

Neither project is production-ready, and both are still evolving as I learn more about email infrastructure, DevOps and self-hosting. But if you're curious how all of these lessons, mistakes and experiments ended up in code, feel free to take a look.


Useful articles:

This post is more focused on my experience and doesn't go too deep into the technical details. However, if these topics are important to you, I highly recommend reading the articles below:


Footnotes:

Definitions of selected terms used in this article.


  1. Nginx - a web server and reverse proxy commonly used to host websites and route traffic to applications. 

  2. DNS records - settings that connect a domain name to services such as websites or email servers. For example, an A or CNAME record can point a domain to a server hosting a website. 

  3. SPF (Sender Policy Framework) - a DNS record that specifies which servers are allowed to send emails on behalf of a domain. 

  4. DKIM (DomainKeys Identified Mail) - a method of digitally signing emails to prove that they were sent by an authorized server and were not modified in transit. 

  5. DMARC (Domain based Message Authentication, Reporting, and Conformance) - a policy that tells email providers how to handle messages that fail SPF or DKIM checks. 

  6. VM provider - a company that rents virtual servers, allowing you to run applications without buying and maintaining physical hardware f.g. AWS, Hetzner or Oracle cloude 

  7. VM (Virtual Machine) - a virtual computer that runs inside a physical server and behaves like an independent server 

  8. Mailcow Dockerized - an open-source mail server suite distributed as Docker containers. It includes a web interface, account management, spam protection, webmail, and other components needed to run a modern mail service. 

  9. Port 443 (HTTPS) - the standard network port used for secure HTTPS connections. Most websites and web applications use this port. 

  10. localhost - a special hostname that refers to the current machine. Services bound to localhost are accessible only from the server itself. 

  11. mTLS (Mutual Transport Layer Security) - an extension of TLS where both the client and the server authenticate each other using certificates. 

  12. VPN (Virtual Private Network) - a private network connection that allows users to securely access internal services over the Internet. 

  13. Grafana - an open-source platform used to visualize metrics, logs, and monitoring data through dashboards. 

  14. Prometheus - an open-source monitoring system that collects and stores metrics from applications and servers. 

  15. Loki - an open-source log aggregation system designed to collect, store, and search application and server logs. 

  16. SMTP (Simple Mail Transfer Protocol) - the standard protocol used to send emails between mail servers and email clients. 

  17. Reverse proxy - a server that receives incoming requests and forwards them to other services running behind it. Commonly used with Nginx to expose multiple applications under a single public address. 

  18. IP reputation - a trust score associated with an IP address. Mail providers use it to decide whether emails should be delivered, flagged as spam, or rejected. 

Top comments (0)