My new side project

Steeve Lennmark — Sat, 31 Oct 2020 10:45:56 +0000

So, first time writing anything public, ever. Hi!

I picked up a side project and thought I'd write a bit about the choices of tech, hardware, services and so on. For the moment I'm gonna keep it a secret exactly what it is I am working on but I can still talk about the broader view.

First things first, my name is Steeve, I currently live in Romania but I'm originally from Sweden, where I spent the first 33 years of my life (currently clocking in at almost 40). I'm co-founder of a startup in Sweden called Matspar where we collect data from online grocery stores (only Sweden at this moment) and help users figure out which store would be cheapest.

Now on to this new project which currently receives most of my free time where I'm not coding on Matspar. In its most basic form it's a webpage where people (with account) can upload a certain type of file and have it parsed, and then later dig around in this data. This might not sound that hard but it's a lot of data and needs to be extremely snappy, which is why I chose to write it in Golang since it's a good combination of fun to write and fast to run.

This project initially started as a small cli application for personal use and someone snowballed into this huge thing that I'm now super stoked about. Also gives me a chance to learn about some pretty cool technologies I've yet to dig my teeth in to.

The tech I've chosen for this project is as follows:

Next.js - Looking into maybe switching to Remix since I am a bit unhappy with Next.js overall
GraphQL (Golang) for the public facing API
gRCP (Golang) for the internal API
PostgreSQL for storage
Redis for some storage but mostly for cache
HashiCorp Consul for service discovery
HashiCorp Nomad for docker orchestration and deployments
Traefik for dynamic

The "hardware":

1 x 2950X (16-Core ThreadRipper), 128G ram, 2 x 960 NVMe SSD

I did a lot of benchmarking before finally settling on this choice. What I pay for this dedicated server over at Hetzner I get basically nothing over at AWS (which would have been my preferred choice). Just for a reference, I can parse around 35K chunks of uploads per second on my i7 laptop. I maxed out at 20K on AWS and I get a whopping 180K/s on the ThreadRipper. I have in my backlog to benchmark an i9 too since it has higher single core clock speeds and if that pans out I might switch to 3 of those machines instead which would land me around $250/mo.

The cloud services:

GitHub for code hosting
GitHub actions for building and deployment
Vercel for hosting the site
CloudFlare in front of the API, mostly because I get free SSL without hassle
Digital Ocean Spaces for storing uploaded files and backups
Magic for authentication, though I am NOT happy with the latency from Europe to their AWS servers in the US so I might have to find an alternative or implement passwordless login myself.

I guess that's it for now and I am sure I forgot to mention a lot of tech being used but I hope to write something about this project bi-weekly here for whoever might be interested. And hopefully you guys can come back with some feedback on improvements and new stuff I've yet to disover!

Have a great one and stay safe!
Steeve

DEV Community: Steeve Lennmark

My new side project

The tech I've chosen for this project is as follows:

The "hardware":

The cloud services: