DEV Community: Tomer Shvueli

Migrating Email Hosts and Registrars with Minimal Downtime

Tomer Shvueli — Thu, 12 Sep 2019 09:34:02 +0000

Email is the focal point of our professional online identities. Whether you like it or not, that is the most reliable and consistent way for people to get in touch with you, and it’s not going away anytime soon (sorry, Slack). That’s why it’s imperative for your email provider to be reliable, resilient, and especially in today’s world, trust-worthy.

Recently I realized that my email host is not great. I registered my domain and set up email accounts there long before I was thinking about privacy concerns or shopping for competitors. They were ok at what they did, but they specialized in assisted hosting services and email was clearly an afterthought, let alone any privacy or scalability concerns.

I knew I needed to migrate email providers. And I knew that this is going to be a tricky and intricate operation. What I was looking for was:

minimal downtime,
end up on a provider that I trust long term and,
migrate all of my current emails and folders.

It also happens that my domain was registered through the same email provider host and I wanted to transfer that over to my everyday registrar, Namecheap. Might as well knock everything out at once and be in good shape for the next while so that I can sleep well at night without worry.

Decisions, decisions, decisions

First thing’s first is you have to find the new registrar and email provider you want to migrate to. This involved some heavy research on my end, because you don’t want to have to do this migration more than once. Everyone has their own priorities in what they’re looking for; for me it was privacy, management, and support. That One Privacy Site has a great article on what to look for in email providers, along with a Simple Email Comparison chart. I spent quite a bit of time looking through these options, and ironically ended up picking a host that wasn’t even on the list, Migadu. As for a registrar, I default to Namecheap as I already have quite a few domains registered there and like their service.

Inform Your Users

I am a big believer in transparency, and especially with today’s internet climate, more transparency is better. I sent a letter to all of the active users on the domain stating that 1) I plan to migrate our email provider and why, 2) to which provider and why and any differences in services they should expect, 3) estimated timeline, and 4) I need their email passwords.

That last point is a point of contention. In order to back up and migrate my users’ email accounts to a new host, I need to ask them to give up their privacy temporarily. Since I am my users’ everyday ‘IT guy’, they trust me and didn’t worry about this point. As a good measure, I linked them to the page where they can change their passwords before they share it with me, so if their password is one they use a lot, they can change it to something they’re more comfortable sharing.

That being said, even with the proper transparency, be sure to listen to your users and address their concerns. If you did the proper research ahead of time, they should trust you with this transition, but it is up to you to keep them at ease and make sure that they are ok with your plan moving forward.

Start Back Up & Migration (Emails)

I wanted to not only migrate my current emails, but back them all up as well, in case something went horribly wrong. Here’s the thing, there were a lot of emails on our server. I wanted this back up and migration to run continuously on a stable connection. AWS to the rescue! I simply spun up a t2.micro EC2 instance running Ubuntu and attached a 100GB Block Store to it. This was going to be my ‘command center’ for all things IMAP (and SMTP) during this transition.

Behind every software success story is a powerful open source tool behind it. This story is no different.

As for the tools necessary, I searched far and wide for reliable, stable, and open source solutions capable of iterative processes. I came across imap-backup for backing up emails and imapsync for migrating. Once you have these 2 tools installed, follow their respective set up processes and plug in your user’s credentials.

imap-backup

For imap-backup, you want your config.json file to look something like this:

{
  "accounts": [
    {
      "username": "username@domain.com",
      "password": "",
      "local_path": "/ext_vol_data/username_domain",
      "folders": [

      ],
      "server": "imap.server.com"
    }
  ],
  "debug": true
}

Of course, be sure to fill in the values properly. You can leave the folders array empty to move over all of the folders. The local_path here is currently pointing to an EBS store that I mounted onto the EC2 instance. And I have debug set to true so that I can monitor the progress of the script through its output. Then, simply create 1 account object per user on your email domain.

Since this is simply backing up our accounts, you can start running this immediately! This script works iteratively so you only need to do a ‘full’ backup once, and every time after that it will only download the difference.

imapsync

Once imapsync was installed properly, I found this Github Gist to help sync multiple accounts at once. For multiple accounts, simply duplicate the line in the accounts.list file with the proper credentials. Make sure you also properly set the CONFIGURATION variables in the imapsync.sh file. This script is now ready to run, but it won’t work until you have both hosts set up and ready to receive requests. More on this to come.

Start Back Up & Migration (Hosting)

The domain I was migrating also happened to have 2 different sites hosted on it on 2 subdomains that I wanted to bring along with me. Be sure you’re prepared for this by picking a new host and making sure that your environments, databases, and source code are all backed up and can be spun up in a new hosting environment.

Take any action you need to try and spin it up before the actual migration. This is easier to do than an email migration since you can have the same site set up on multiple servers, but you can only have 1 email host at a time! Ideally, your sites should be migrated and simply waiting for the DNS change to take effect.

Configure New Email Host

Because I wasn’t only migrating email hosts, but also registrars, I couldn’t pull the trigger on the domain transfer until I was sure that things looked right on the new email host’s side. I got in touch with Migadu’s support and they worked with me to ‘enable’ my domain name even though it wasn’t transferred yet. They also gave me a 1 month free trial ‘Basic’ account so I can see the full extent of the service.

Using this, I was able to make sure that the new email host was up and running.

I created email accounts for all our users on Migadu with template passwords that I could provide them for their first login. I also set up things like alias email addresses (so one can use emails like name+service@domain.com) for everyone.

Regex for ‘+’ email aliases: ^username\+(.+)@domain.com

Since I know had the original email host up and the new email host up, I was able to use imapsync to start transferring all the data to the new email host. With the proper configuration, the previously mentioned imapsync.sh script made the transfer smooth and seamless, migrating over all emails within their respective folders, and incrementally.

Admin Stuff

If you also plan on transferring your domain to a new registrar, be sure that the ‘webmaster’ email for that domain is an email you have definite access to and that is not tied to the domain that you are migrating.

Pulling the Trigger

Ok, now everything is in place and backed up. What’s left is actually pulling the trigger and starting the migration process. Pick a Friday or Saturday night, a relatively quiet night for emails in order to minimize the ‘blast radius’ in case anything does go wrong. Again, for transparency purposes, be sure to inform your users when you plan on doing this, so that if they see anything fishy, they know that you’re be behind the scenes working on it.

Before officially starting the transfer process, be sure to run the imapsync and imap-backup commands one last time to ensure the mailboxes on each host and backups are as up to date as can be.

Now, begin by transferring the domain to your new registrar. You can do this by going to the ‘transfer domain’ page at your new registrar. You will need to ‘unlock’ your domain from your previous registrar and make a note of the domain code for the transfer.

Once that is triggered, be on the look out for any emails coming through to the webmaster email of the domain from your previous registrar. A lot of times they will confirm it with you before enabling the transfer to go through.

Now you should start configuring any MX and DNS settings for your domain on the new registrar. The MX records are the most important to not have any downtime for your email accounts. Once those are complete, be sure to set up any other DNS records you may need for the sites you had hosted on the domain and point the nameservers over to the new web host.

Time to make sure everything is working properly!

In your mail client of choice, update your email settings to point to the new email host with updated credentials. The first thing you should notice is that your Inbox and all other folders should be present just as you expected, thanks imapsync! Now compose an email from any other email address of yours to the email on the domain you just transferred and make sure it comes through – IMAP is working properly. And compose an email from your newly transferred domain to your other email account – SMTP is working properly.

Inform your users that the migration has taken place and provide them with the new host’s settings (IMAP and SMTP servers, ports, etc) and their new passwords. Of course, you’ll have to do this through means other than their email Be sure to point them towards the settings page they need in order to update their passwords to something personal and unique.

Once you’re sure your email is working properly, be sure to also test any hosted sites you migrated and all of their functionality as they now live in a new environment.

It’s All About the Customer

Follow up with your users to make sure that everything is working as expected for them. This is also a good opportunity to talk to them and see how they felt about the whole process: was it smooth? Painful? What could have been better? Hopefully you won’t need to do this again, but it’s good to get this type of feedback in general.

Clean Up

So now you should be sure that all of your emails and sites are migrated properly and working as expected.

What’s left here is cleanup, which is just as important as set up! Do not cross this off your list until you finish properly cleaning up any set up you may have done. I would still wait a solid few weeks or so before starting this to make sure that everything is in place before you can’t go back, but be sure to set yourself a reminder so you don’t forget.

Our main cleanup consists of running rm -rf on any and all email backups in the EC2 and Block Store. Be sure to do this for all the settings and config files as well, as your user’s (as well as your own) passwords are present there! You can now tear down these resources confidently knowing that your information lives only on your new host’s servers.

Aftermath

Phew! Give yourself a pat on the back for a job well done!

Be sure you have a real, paid account with your new host to stay on their good side and not let a trial expire without realizing.

That’s it! Congratulations on a completed migration performed well with minimal downtime!

Photo by Web Hosting on Unsplash

Share Your Stack – How to Architect the Perfect Travel Blog, WhereTheHellAreTomerAndMichelle.com

Tomer Shvueli — Tue, 18 Jun 2019 09:36:45 +0000

When Michelle and I started hatching together our plan for our trip, I knew I wanted to grow, both as a person, and as a developer. The very first step that I could do to start growing was to pick a new technology stack to host our travel blog. I had already worked with WordPress, published on Dev.to, and I knew I wanted to host something (i.e. not Medium).

Requirements

Besides the basic blog necessities, my requirements were:

Static – This would be a simple site and I wanted to keep it light weight – no need for server side lifting here.
Easily updatable – I wanted to be able to easily update it and remove any roadblocks from publishing new posts or updating the site. Once you go CI/CD, you never go back!
Posts in Markdown – Markdown is simple, and it’s now everywhere. This is also a good way to separate your content from your site if you ever want/need to migrate your posts to a diffrent stack.

For a long time I had been hearing about Jekyll and Github Pages hosting and the simplicity and integration between the 2 technologies. This seemed as good a time as any to pick it up and run with it!

I use the Lagrange Jekyll theme for our site as it seemed pretty simple and we liked the way it looked (how does anyone pick anything??). I went ahead and forked the project and then started putting our own touches on it.

Basic Setup

Jekyll makes it very easy to customize certain parts of the site in the settings.yml file. We made sure to update our social links, About page, contact, etc.

We also definitely wanted our own domain name and not a github.io subdomain. Who would’ve thought that WhereTheHellAreTomerAndMichelle.com was available?? With my favorite domain name registrar, NameCheap, and $13, we were the proud new owners! Github Pages makes it easy to host your site on a custom domain name as long as the repository is public. Using the Pages Settings for your repo, it will configure your custom domain name by creating a CNAME file in your repository. Github Pages will also automatically take care of an SSL certificate, how thoughtful! We were off to the races!

Travel-ize It

Since it is our travel blog after all, we knew we needed to put our itinerary and picture slideshow on it. For the interactive map/itinerary we chose TravellersPoint.com‘s embedded map. I have used them before for my Eat, Pray, Code travel and loved what they provided. To customize the embedded map even further, I upgraded to a ‘Budding Member’ and placed the embed code directly in the Jekyll page. Since this is an iframe, I also added some Javascript and proper styles to ensure that the map works well on mobile as well as desktop views.

Pictures were slightly trickier. The first part was figuring out where to store my photos. I wanted to be able to upload only select photos, not have to worry about storage limitations (at least for the forseeable future), and be able to share a public album. Eventually Google Photos won out because of their ability to share an album publicly. (I am still open to suggestions of other photo storing/sharing solutions out there!)

The next issue with Google Photos is that I wanted to create an automatically updating embedded slideshow on our homepage. The tricky part comes with Google Photos’ lack of an API. Luckily, I came across this page where, given a publicly accessible Google Photos Album, it will create an embedded slideshow widget. At the moment this does not automatically update, I simply have to update the slider code every so often when I publish new photos to the shared album.

Publishing Posts

One thing that I was really looking forward to with this new stack (and Jekyll in general) was publishing posts in Markdown. As a developer, I love the simplicity and ubiquitousness of it, while still giving the author control over any elements they may need. When I begin to create a new post, I simply pop open my Markdown editor of choice, MacDown, and start typing away! When I finish a post and run it by Michelle for final touches and editing, I can publish it using a simple git push command!

Comments

I’m sure we’ve all come across Disqus at one point or another in our WWW adventures, and that is what I originally set up for our site. But seeing as I am a self hosting connoisseur, when I heard of Commento.io, I immediately knew I wanted to try it out. Luckily, not long after that, the Cloudron team pushed out the Commento.io self hosted application on their platform. I set it up on my Cloudron instance, imported all the data from the Disqus instance, and replaced the comments code with the proper Commento embed script. Viola! You now have self hosted, secure, and privacy focused comments on our site.

Analytics

Google Analytics is probably the go-to for many sites and blogs, but I have long been using Matamo (formerly Piwik), also self hosted. Using the Matamo Analytics dashboard to set up a new site is a breeze. Then, simply replace the Google Analytics code provided by default by the Jekyll theme with Matamo’s code, and push it up!

“But what if you get hit by a bus?”

Michelle, out of her love and thoughtfullness, one day asked me, ‘how do I publish new posts if you die?’ She had a point. Not only is she not a developer and has never used Github before, but the site’s repository is also only under my username.

After some quick research, I came across Forestry.io – a Jekyll CMS. This seems to be a fork off of Jekyll Admin, but either way, Forestry is a fantastic tool to make everything easier for managing a Jekyll site in the way of hosting media, direct integration to Github and Github Pages, and of course, creating and editing posts. And yes, they do offer a generous free plan for a personal site! Getting this set up was a breeze, the only slightly tricky thing was getting the media folder set up properly, but even that was done without a hitch. And since they support multiple users per site, both Michelle and myself have access to update our Jekyll site – including writing and publishing posts – through their CMS platform.

Stack Birds Eye View

Domain: NameCheap
Hosting, with HTTPS: Github Pages
Framework: Jekyll
Jekyll Theme: Lagrange
Analytics: Matamo (self-hosted)
Comments: Commento (self-hosted)
Interactive map: TravellersPoint.com
Photos Hosting and Shared Album: Google Photos
CMS: Forestry.io

As most good things in life, this took time and is not done evolving. You can check out the Github repository to see for yourself how each of the items above grew and were implemented over time. Sure, I had a general idea of what I wanted on the blog to begin with, but a lot of these ideas took time to develop and implement. Be sure to check out the final product (for now)!

When Google Shuts Down a Service, a New Browser Window is Opened

Tomer Shvueli — Sat, 02 Mar 2019 12:39:37 +0000

How a Google Project Shutdown Opened the World of Self Hosting to my Eyes

I want to take you all the way back to 2013, before major privacy scandals peered their heads every other week, when everyone finally could answer the question ‘What does the Fox Say?’, and when Google shutting down a(nother) service of theirs still didn’t surprise us.

I was an avid user of Google Reader, the search giant’s RSS aggregator that allowed one to follow multiple news sources in a clean, streamlined, and unobtrusive manner. I forget how I first came across this service, or even RSS as a whole, all that I knew is that this was my main, and pretty much only way to consume news online. It made sense; there weren’t ads or paywalls on any of the news sources I followed, I could save stories for later viewing, and I could always go to the original source if I wanted.

Then came that fateful day in March 2013 when Google announced they were going to shut down Google Reader. Of course, by this time, it wasn’t much of a surprise that Google was sunsetting yet another one of their ‘Lab’ projects. And with RSS being a pretty niche medium within the ‘nerd-o-sphere’, this announcement went off without much of a mainstream fuss. All the while, I’m scrambling for a solution to migrate to after this unjustified killing.

Little did I know, not only my RSS aggregator, but my entire online lifestyle and primary hobby that keeps me up at night, was going to change.

There were a few different alternatives to Google Reader that started popping up after this news. A lot of them even would help you transfer over your Reader data, so you can pick up right where you left off. Call me a cynic, but it was hard for me to hand over my data, and more importantly trust, after the unfair shutting down of a service that I relied so heavily on. And then I came across the ultimate solution: Tiny Tiny RSS.

Tiny Tiny RSS (or TTRSS) is an open-source (meaning anyone can view what makes it run, add suggestions or improvements, or update the application for their own needs), self hosted (meaning it can run on your own computer, instead of only on a company’s computers) RSS aggregator. This meant a few things for me:

I can move off of Google Reader safely and migrate my data to this new service (a feature of TTRSS).
If there’s an issue or a new feature I want, I can make the updates on my own.
I won’t need to trust any other 3rd party to house my data or to stay online, it will all live on my computer!

Getting it set up and working properly was a relatively easy task to a software hacker like myself. And I was amazed by the results! TTRSS instantly became my most used tab, like Google Reader before it, except now I knew that no one could tell me that this was ever going to be shutdown – it all ran right on my computer!

After years of using this application, it really engrained in me the value of self hosting and owning one’s data. It’s truly a beautiful thing knowing that you are not at the mercy of any company’s will or bottom line.

And then I discovered the entire sub-culture of self hosting.

This opened my eyes to a whole slew of applications that I can self host and stop relying on others for. My most crucial find during this self hosting renaissance was Cloudron. Cloudron is a self hosted platform that allows one to install applications through an App Store, a la Apple’s App Store.

I currently am self hosting a Cloudron instance running a few applications including Wallabag, Monica HQ, and, of course, the trusty TTRSS (still my most used tab to this day!).

Privacy

A major aspect of self hosting one’s own services is the fact that your data is always in your hands (or computer). In today’s privacy-less world where companies are hungry for your data, self hosting provides a haven where you own your data.

I wouldn’t have known it at the time, but I am very thankful that Google shut down their Reader service. If it weren’t for that, I might never have come across this intriguing and captivating subculture of the internet.

If you are tech savvy and want to try to rid yourself of the handcuffs that data aggregators have on you – and also have some fun while at it – I highly encourage you to check out the world of self hosting. Some great resources to start are the Awesome Self Hosted Github page and Reddit’s r/selfhosted.

🍻 to a reliant-free and self hosted future!

Sunsetting a Live Service

Tomer Shvueli — Sat, 29 Sep 2018 19:30:25 +0000

“People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully.”

-Steve Jobs

As a developer who’s started more projects than I can count, it’s good to take a look at one’s portfolio from time to time and do a bit of a spring cleaning.

What

BaDumChh was a daily joke texting service that I launched a few years ago. You can read about its inception here. TL;DR I was really excited about this idea for a long time and even ended up doing a relaunch after it was running for a while already.

Why

This decision was not easy. BaDumChh was close to my heart and not only was I a user of it, but I originally built this service for myself. So why did I decide to sunset it? A few reasons:

Not Curated Content

I always strived for BaDumChh to be a self-sustaining service that could run on its own. And it was that. But without moderation of jokes, there were some days where the joke that was sent out was inappropriate, offensive, or just not funny. This wasn’t the type of quality that I wanted tied to my name.

Growth

Growth was non-existent. I had only a small handful of people on the service and that number hadn’t changed in years.

Business Model

While payments were implemented and could run without intervention from myself, there were no paying users. And since this was a texting service, I was actually paying for this out of my pocket.

Needless to say, when I found myself looking ahead to the future and trying to focus on quality projects that I’d be proud to put my name on, BaDumChh didn’t make the cut.

How

Because BaDumChh was a live service and had many moving pieces, I had to make sure to come up with a graceful exit plan.

Using a feature I had built into the service a while back, I was able to schedule a message to all users on a certain Friday.

Once this message was sent out, I pushed up changes to the site that I had made ahead of time that would alert incoming users of the discontinuation of the service. These changes included:

Invalidating the API
Invalidating webhooks
Invalidating cron scripts
Updating the home page to say that the service has been discontinued
Updating the Twilio endpoints to have a proper response

After these were pushed up, I still had to make some updates to the environment itself. That included:

Removing the cron jobs entirely
Deleting environment variables for live services
Deleting the test environment
Emptying the database with user’s information. I value my user’s privacy.
Deleting the subscription plans in Stripe

I also have a scheduled calendar notification in 1 month to release the Twilio phone numbers I used. I wanted to keep these around for a transitionary period in case someone were to reach these.

Lessons/Ode

Some things that I experienced first hand from running BaDumChh:

‘If you build it, they will come’ is not real.
If you want consistent and good quality, you cannot rely purely on an algorithm. You must either curate the content, or have some sort of democratic system (a la Reddit upvotes).
Know when to say ‘good bye’.

I don’t regret creating BaDumChh. I am a big believer in learning from one’s failures and moving forward. Otherwise, I wouldn’t be where I am today, and I’d like to think I’m in a pretty good place. I absolutely love the system I created for this service, even if it will no longer see the light of day. The only personally identifiable information I kept on my servers were user’s phone numbers. The whole system ran on its own without administration (though this was eventually part of its downfall). The login process was smooth as butter. And it even ended up putting a smile on some people’s faces. But alas, the time has come to focus.