DEV Community: x-t

The Hitchhiker's Guide to over-engineering your site

x-t — Tue, 08 Mar 2022 16:18:50 +0000

Do you have a personal site? If you dabbled with computers in your lifetime, there's a really high chance that you do. Maybe it's not hosted, you've watched a video on HTML when you were 10 and decided to make something of it, maybe it's a simple Neocities page, a Blogspot site or even a Weebly cookie-cutter homepage. Whatever it is, it's a nice feeling to know that you have a corner of the internet that's yours.

While nowadays hosting infrastructure for your site is hard and expensive, as each user requires more and more bandwidth, for 99.99% of us, this doesn't matter. Most of our personal sites are relatively obscure, with free hosting tiers covering all traffic, to accidentally create and host another Wordle with millions of active daily users isn't on our schedule.

But infrastructure costs aside, there's another good cost that's associated with creating and maintaining a website - time for development. Sometimes we are limited in the amount of tools that we can use to make sites, for example, if all you have is Neocities, you're essentially left with plain old HTML, CSS and JS, all hand-written. While for small, simple sites this is more than enough, for larger projects, this is nowhere near ideal. Committing to hand-writing hundreds of lines of JS is going to end up in buggy and slow code, which is why nearly everyone, including small sites, have migrated to using libraries, as even the most basic ones (Vite, Alpine, TypeScript, ESLint) can increase productivity and decrease stupid bugs ten-fold. Or if you used GitHub Pages before the days of CI/CD, you had plain HTML or minimal Jekyll at your disposal.

We live in a very, very different world now. A world where computing resources have become as cheap as water, and thus, you can join the dark side and totally over-blow your blog into a brilliant masterpiece of complexity. This guide will be based off my real website, zxyz.gay (GitHub).

Always think big.

What do you want your homepage to have? A bio? Some contact info? That's clearly not good enough. Try several services, several pages. A simple blog? Make it two. One JS framework? Make it three.

Here's the layout of my site:

Frontpage
Guestbook
OS Reviews
URL Shortener
Blog

In short this makes 3 backends, 2 frontend frameworks (3 if you count Jekyll) and 3 languages. Does anyone need this much stuff? Probably not, but, if we can add it, then, why not?

If it wasn't enterprise, it is now.

The mindset of enterprise-level software is always the best to follow. For example, Sentry is a good service for crash reports, so - add it to every single file of your site. Even if there's barely anything in there that could fail, add it anyway, can't be too sure.

"Yeah, I can see this being Google-scale"

Nowadays, nearly all developer tools are open sourced and available for anyone with an internet connection, so, make use of them. Use Docker for everything, even your static site, because, why not? Hell, throw in some Kubernetes in there too.

Another trend you can take advantage of is making your site into a monorepo. You can use Nx or Turborepo, both free, turn your semi-simple site into something straight out of Facebook's arsenal. Don't forget to not include any docs, no one else should be able to host it, only you.

This is your ideal site configuration.

Remember - your computing resources don't matter.

As I said previously, computing resources are cheap. A lot of it is free now, too. This means you can do whatever crazy stuff you want and you can totally get away with it.

For hosting, any free static file service will do, many of them have quotas that are far beyond what you'll even care to reach, so, it doesn't matter that much.

For backend, many free sites like Deta, Vercel, Northflank, Koyeb, Glitch. Make as many services for your sites as you desire without even entering in your credit card.

For source code, GitHub hosts anything and everything, a whole lot of it without any limits whatsoever. I have 40+ repos, one of them being 5 Minecraft worlds, with all the world files and even the server JARs included. It's essentially cloud storage, for free. Blow up your site with files, what's the worst that could happen?

For building, CI/CD pipelines are really, really cheap and really, really fast. GitHub Actions has 2000 free minutes on private repositories and unlimited free minutes on public repositories. Create tens of pipelines, use several compilers, upload to a multitude of hosting services, it's all free.

For coding, GitPod has 50 free hours of private Visual Studio Code workspaces. This is not only convenient, as you get to install less stuff on your computer and the Google Cloud container will most certainly be faster than your machine, it will also allow you to code on your projects from your Chromebook or your iPad. But most importantly, you can have 10 services and run them all without blowing your fuse. Thus, you're free to over-engineer even more.

Real world example - stupid me.

Now, I didn't really want to over-engineer my site. It just sort of happened, I wanted to learn several technologies and adding pages to my site was fun, so, it just ended up like this.

Frontpage

The magnum opus, the self-made Carrd. Many years of iteration, ending up in a design that's questionable at best and a codebase with so many compromises you really couldn't count them all.

Technologies used:

Vite - build tool. Bundles all JS modules.
Vanilla HTML, CSS - Simple enough
Alpine - Simplest framework I found, used for the guestbook.

Hosting providers:

GitHub Pages - main host
Neocities - mirror

Guestbook

You can read about the infrastructure and its history in detail in my other blog post.

Technologies used:

Hosting providers:

URL Shortener

I don't really use it for anything, it's more of a joke, so I tried to slim it down to the bare essentials with a hosting provided that would be willing to just work.

Technologies used:

Express
Deta Base

Hosting providers:

Deta

OS Reviews

It's essentially a blog, but with its infrequent updates, and the fact that it was made before the blog, it ended up being a simpler Markdown-flavored site. With styling being primarily made with Tailwind, I got the final design in under a day, which is rather simple, yet pleasant.

One of the downsides of Jekyll is the fact that it takes 4-10 minutes to build the entire site, generating hundreds of images scaled differently for mobile and desktop devices and many Ruby dependencies that take... a really, really long time to install.

Since it's the same static site as the frontpage, I just build it and deploy it into a subdirectory of the site, which is then mirrored to Neocities.

Technologies used:

Hosting providers:

GitHub Pages
Neocities

Blog

The blog is pretty boring, the amount of boilerplate that Vercel and Sanity have done made it a really easy project to get started on. A headless CMS allows for quicker build times, skipping the image optimisation. With new features like on-demand incremental static generation in Next, it also means that updating or creating a post can be done in seconds rather than minutes, unlike the Jekyll setup.

But ultimately this does leave you with two billing accounts to worry about - the usage quotas for Vercel and Sanity. While their free tiers are generous, you might want to think twice about the amount of vendor lock-in you're signing up on if you plan your blog to have thousands of viewers. Which, for this tutorial anyway, is rather perfect.

Technologies used:

Next
Sanity
TailwindCSS

Hosting providers:

Vercel
Sanity

And now - you're a master-class over-engineer.

Remember, being a 10X engineer doesn't mean that your 10X will be towards good quality engineering, but rather totally incoherent, unreadable code that 20 people will later have to try and maintain.

Take these tips as do or do not do's, on one hand I can see that this is a total mess and I should probably spend some time trying to simplify it. But on another hand, it's rather fun to learn to use that many tools in the real world. I'm not trying to make a new unicorn, I just wanna learn.

So go make something, chances are - it won't be good, but, it doesn't matter. Start up Microsoft FrontPage and make your own next site. It's only one $300 monthly AWS bill away!

Hosting a guestbook, how hard can it be?

x-t — Fri, 11 Feb 2022 14:19:33 +0000

Ah, the good old internet. Geocities, GIFs, banners, view counters and one of my favorite things - guestbooks. Things that undeniably, died out. And no real wonder, the barrier to overcome for web development has gotten higher, the amount of people on the web increased by several orders of magnitude, many introduced in the late 2000s with touchscreen devices that couldn't be used for any kind of development. Social media took the place to provide a polished, easy experience for anyone's internet presence, without having to worry about code, hosting, scaling, design, accessibility or anything else.

The design of social media doesn't allow these old tropes to exist. Geocities was made for the older internet, failing to scale to the new one for billions more users. GIFs couldn't be plastered anywhere anymore, as they looked terrible, companies got good at design and not triggering epileptic seizures from anyone daring enough to double-click the Internet Explorer icon. View counters don't make sense anymore, while analytics for pages exist, it is unquestionably inflated by the fact that clicking a profile is easier than finding a page. And guestbooks, well, have been replaced with just posts. Twitter, Facebook, Reddit and everything else is structured into posts and replies. A guestbook in a Twitter profile would really just be a simple reply.

But the beautiful thing about being a web developer is you can make whatever you want. And while guestbooks nowadays are more popular in just coding demos, there are groups of people that specifically want to bring back the old web. And you could say, I'm in one of them. I made a guestbook afterall.

It's actually pretty simple

A guestbook is a really primitive form of user interactivity.

It's definitely simpler than a comment section, it's just some logic and a database. And for the old web, as well as my old website, it worked pretty well.

This is the first generation of the guestbook from 2018. The amazing front-end design aside, the structure was rather simple.

This is what nowadays we'd call server-side rendering. A single DigitalOcean VPS, running Debian with a MySQL process and a PHP-FPM server running on Nginx. The source code is as simple as the infrastructure. (Source code uses JSON files instead of a MySQL database, the actual code used on the server was in private git repositories and lost to time.)

comments);
foreach ($arr_cmts as $comment) {
    echo "[" . $comment->time . "] " . htmlspecialchars($comment->name) . ": " . htmlspecialchars($comment->data) . "\n";
}
?>

The fact that it was a full server meant that I had quite little to worry about the usage of the database and PHP. Things like compute and bandwidth costs wouldn't need to be accounted for, it's just a flat $5/mo for the virtual machine and some usage quota, that's it.

As there was barely any traffic, the server was also used for a multitude of other things, like a Quake 3 server, my friend's website, an analytics engine that tracked essentially nothing but search engine bots, it was quite a lot to learn.

When money strikes

But all good things must come to an end, and this DigitalOcean VPS was one of those. I had a $50 credit from the GitHub Student Developer Pack, which ended in less than a year. And I wasn't gonna start paying that much for a server that wasn't actually used a whole lot for anything else than learning, so for the lack of any users for the site, I nuked it. Moved the site to GitHub Pages. For more about the actual move, you can read the blog post from 2018.

Jamstack

Nowadays it's easier to get a back-end site host for free. Vercel has many Javascript server-side frameworks supported, all with the help of AWS Lambda. And if you're still thinking of learning LAMP, the shady market of things like Infinityfree will get you a full LAMP stack for free, with some limits.

However, a few years back, I decided to go with GitHub Pages, as the backend offerings I wasn't that interested in and I could get a free github.io subdomain, plus it would be hosted straight from a GitHub repo, without the need of a website like Neocities. This meant that I embarked on the Jamstack journey, even if I didn't know what it meant back then. And while Jamstack has many advantages, such as speed, any server content becomes exponentially more difficult. Thus the guestbook was abandoned for quite a couple of years before trying again.

A redesign that prompted new ideas

In 2021, I totally redesigned my site from the ground up using the styling of the very old internet, aided with Claris Home Page.

This meant that I could redo with the old internet tropes. Serif fonts, banners, GIFs, bad HTML and something that I knew I'd be able to do now - a guestbook.

Of course, I would not move the site out of GitHub Pages, which meant that I was stuck on Jamstack and couldn't just plop it back in with the same PHP code, so, ideas had to be made.

Welcome to Azure

The way the Jamstack infrastructure works is it consists of 3 layers:

The database, to hold all guestbook posts
The API layer, to allow safe insertions/retrievals of guestbook data
The frontend, to render all the guestbook data

The database was held up by Azure Cosmos DB, a free MongoDB/SQL-compatible database, I used MongoDB for it, which worked pretty well. However, it was really slow, unsure why.

The API layer was a Go serverless application written primarily by me, running on the Azure Functions serverless runtime on Windows.

For the front-end, a new HTML file was created, which would download the JSON from the API and display all posts, with some clever local caching strategy to cut down on the amount of API requests sent by every visitor. Also, since it's Jamstack, it used a client-side form, since there was no real backend server to just hand off the form to.

It was slow

A fetch would be fast if it took 7 seconds, more often than not it'd be a cold start which would take anywhere from 12 to 30 seconds to actually see the content. It was completely free, so I didn't complain that much.

Google Cloud - old friends return

I decided to take out the free trial of Google Cloud, and instead of doing something useful with its' offerings, I just spun up a Compute Engine VM with RHEL. While the guestbook Go code was made to work on serverless and MongoDB, I migrated it over to Postgres and ran it as a systemd socket. The frontend stayed on GitHub Pages, so while the backend structure almost looked similar to the one before, it wasn't the same. Still, this showed me how it'd be great to not have vendor lock-in for your projects.

The fact that it was written in Go and not in JavaScript with a weird Azure library meant that I was able to just run the same code without a rewrite. In fact, if I had hosted a MongoDB database instead of Postgres, it'd be a drop-in replacement.

But the credits, just like on DigitalOcean, would end, and I'd return back to the Azure solution.

Free tiers galore

The beautiful thing about startups is the venture capital investments. This means less money they need to stay afloat from the users, which ends in some truly spectacular free tiers. I ended up splitting the infrastructure in two, one way for fetching and another for making posts.

For the fetching, I first used Supabase, which is a Firebase alternative, but then I ended up with the problem that 7 days of inactivity would just freeze the database, so I migrated to Nhost, which provides a free Postgres database backed by Hasura.

For the posting, I needed some custom logic, such as an hCaptcha checker, so that was replaced with a client-side form and a Fly container.

This sped things up significantly. Not as fast as the GCP solution, but still significantly faster. At this point I also migrated from the Go project to a Typescript/Express stack inside a monorepo, which would be much easier to maintain than the aging Go code.

But, startups aren't perfect

Nhost, while working perfectly fine, had a major flaw when I was migrating the site from a really awful vanilla JS frontend to the Alpine framework. The uptime.

I'm unsure if they were just freezing the containers like a serverless cold start or the servers just keep going down, several times I wouldn't be able to fetch any messages. Sometimes it'd return with a "Service temporarily down", other times it'd just hold the HTTP connection and provide the guestbook JSON 5 minutes later. Considering I don't monitor the uptime, and it was happening constantly while I was just developing the site, I just took a random guess that when I'm not developing on it, the database would be down just as much, so I decided to migrate.

Good timing

Just as I was trying to find suitable alternatives to host the guestbook on, Fly announced that they'd be making 3 GB of volume data free, thus meaning that running a PostgreSQL database for free was now possible.

Both the database and the backend API now run on Fly, both cost $0, are always-on, have significantly better uptime than Nhost and are crazy fast. With a simple hack like:

allows the frontend to fetch the guestbook as it is loading the Javascript to render it at the same time. It's faster than anything before, and even when all of the above solutions cost $0, this one might be the cheapest as you get the most.

While it is lacking in resources (128MB RAM, 1GB disk), you get 100GB+ of free bandwidth, a full unrestricted VM (not just an user) with Postgres, so while it is a little more effort to administrate, it's actually more powerful than Nhost or even Supabase.

Conclusion

However simple a guestbook is in schema, in the modern age of the web it is actually quite some effort to host. And mind you, I am not even doing things properly. It's just two VMs in Europe - one with the Express server, another with Postgres. There's no high-availability, no edge, nothing fancy. These things cost money, and I want the guestbook to stay free. And through many design iterations, I found Fly to have the best solution for hosting a Jamstack guestbook in terms of free allowance, speed and control.

A simple Twitter profile looks like a way simpler investment now, doesn't it?