DEV Community: Jerry Ng

A Look Back on 7 Years of Automating Stuff

Jerry Ng — Mon, 04 Dec 2023 00:00:34 +0000

A little bit more than 7 years into my career, I thought it would be fun to pen down a summary post about my adventure in automating various tiny aspects of my life. Most if not all of the stuff here sprouted from my own problems and itches I needed to scratch.

This post will probably read more like a personal diary of the minor nuances that I encountered and the sweet minutes/hours that I managed to snatch back through automation.

On to the first one –

Six Percent: Automating ASNB Purchases

2018 – 2022

The startup window of the bot. This was the first thing that I've made that people actually use.

It all started with this Python script called "six-percent", a seemingly dumb bot to click buttons for some investment.

For context, ASNB a unit trust management company offers a fixed-price fund (i.e. it can never go up or down!) that promised a sweet 6% p.a. dividend back then. It’s virtually risk-free.

Well, there was a catch. The limited funds pool meant that it was selling like hotcakes — you had to keep retrying to snag those units. So, what did I do? I wrote this.

Cost

As this turns out to be a tiny Windows executable, it didn’t really cost me any money to host or anything.

Free lunch, literally

Beyond saving me countless mindless hours clicking like a headless chicken, I got a free thank-you meal out of it.

Fast forward to today, and I'm no longer actively using or maintaining the project. Over the years, there have been a few minor bug fixes here and there, but it's essentially retired. I can't guarantee that it still works, but man, it saved me so many hours.

What did I learn

Today, I’ve become quite comfortable with any form of browser automation. If I can interact with it on the web, I can automate it. The script opened doors for tackling repetitive/mundane tasks, aiding in integration testing, etc.

💡 Years later, I stumbled upon go-rod which is significantly better in terms of ease of use and developer experience.

That aside, I did also learn other cool tricks like how to solve a CAPTCHA using OCR and how to package a Python app using PyInstaller!

Todoleet: Daily Leetcode Questions in Todoist

2021 – Present

Back in early 2021, I started to solve the LeetCode Daily challenge as part of my morning routine. Quite frankly, it wasn’t exactly fun for me; it was necessary.

My personal Todolist. Nope, I'm not attempting this one.

How it works

Even though I've ditched my morning LeetCode routine, Todoleet is still up and running today.

Under the hood, it’s merely a simple JS script that talks to the undocumented LeetCode API and then creates a new to-do task using the Todoist API.

If you're looking for the nitty-gritty, I've spilled the beans on how I sync the Daily Leetcode Challenge to my Todoist.

Cost

$0. The free Cloudflare Worker tier has got me covered! This solution costs 1 request per day and I didn’t need to store anything.

Time saved

I did manage to save a few clicks (seconds) every day. They all add up, I guess?

Any learnings?

This was my introduction to Cloudflare Worker. It paved the road for all the other projects I've tinkered with in my free time!

Future plans

I did consider taking this to another level and listing it as a Todoist integration. But, to be honest, I didn't care enough to make it happen.

Burplist: Sipping on Craft Beer Savings

2021 – Present

Craft beers are delicious. There was just one hiccup — the price. Then I figured, wouldn’t it be great if I could have current and historical prices of all craft beers in Singapore, all in a place?

Now, instead of wrestling with 10+ websites for the best deals, a quick search from my database would do the trick. This saves me time and sanity.

I was looking for some Dark Ale as I was writing this

How it works

Burplist is essentially a web scraper built using Scrapy. It scours over 10 local online stores and e-commerce sites every morning (Singapore time), fetching craft beer prices and storing them in a Postgres database. As a result, I've amassed two years of historical craft beer price data in Singapore.

Cost

Other than the ~$10/year for the domain name, running Burplist has always been free. After Heroku phased out of its free tier plan:

The scraper Cron job was moved to Northflank
The Postgres database is hosted on Railway
The website was moved to Koyeb

Free beer for that Christmas

So, initially, I gave it a shot to make some money out of this. All I did was put it up for sale on Gumroad, but it didn't really catch on.

I also tried reaching out to a few companies through cold emails to see if they'd be interested in partnering up with some affiliate links, but only one replied — well, it didn’t work out either.

But hey, no big deal! It's all good because something awesome still came out of it! Thirsty, this local craft beer company, actually surprised me with this amazing package of craft beer for Christmas that year! I was over the moon.

I got a box of delicious craft beers for Christmas that year

Any lesson learned?

This was quite a big one. I’ve had so much fun and learned so much from making this project. Burplist is the fanciest web scraper that I’ve built thus far. I’ve written down some of the learnings in a blog post.

😂 Oh, and here's a quirky side effect: I’m now able to figure out whether a mega-sale/promotion is the real deal or just a clever markup with a discount disguise.

Looking ahead

Future plans? Maybe migrate from Postgres to SQLite. Then, update the daily job to push the SQLite file directly to GitHub and present the data through something like this nifty method. Expected result? One less webserver to babysit. If this ever happens, I’ll probably write about it somewhere.

Wraith: Automating Ghost Blog Backup

2022 – Present

A screenshot of my terminal emulator

At the start of my blogging journey, I didn’t really think too much about what would happen if this $6/month Droplet crashed. I mean, I had all my blog entries in Notion, so I figured "Meh, it wouldn’t hurt to copy-paste ~10 of them back if anything bad happens".

As time went on, the realization hit — losing all my data and whatnot now would really suck. So, the solution? Automate the backup. With the blog serving around 10k views monthly, any downtime or a prolonged 404 or 500 is something I'd rather avoid.

What’s underneath

It’s a pretty simple Bash script that does three things:

ghost backup
mysqldump
rclone to a remote drive (e.g. Dropbox, Google Drive, etc.)

This Bash script gets a weekly run in a Cron job, and it's as easy as that.

Cost

$0. I didn’t have to pay for Dropbox; the free tier fits my needs just fine.

Time saved

I mean, the alternative would be manually SSH-ing into my Droplet, running backup steps one by one — a process taking roughly about 5 minutes or less. So, that's the weekly saving of ~5 minutes.

What did I learn

Picked up a few best practices for writing Bash script along the way. Besides that, I did learn about new tools like expect, rclone, and pass (the Linux password manager).

Future plans

Not a whole lot to be honest. Perhaps a backup restore script could be handy. Thought about moving passwords from plain text to using pass. But, that means users dealing with GPG + Pass CLI setup — an extra hurdle.

Tournacat: Sync Esports Schedules to Google Calendar

Started 2023

I got tired of missing highly anticipated Dota 2 Esports matches, dealing with wonky time zones, and the mental gymnastics of remembering it all. Since Google Calendar is practically my second brain, I figured, why not sync upcoming matches straight into it?

Month view of Google Calendar

Turning it into a micro SaaS

I started by sharing the Dota 2 calendar with friends. Then, a lightbulb moment — if it's handy for them, maybe others would pay for it. And so, D2GCal was born. Pay, and get a public link to the Google Calendar. Simple.

But wait — people are picky about their calendars. Some find it “noisy”, and some want a customized experience. Enter Tournacat, supporting 10+ Esports titles (not just limited to Dota 2!), giving users the power to own and customize their calendars.

Cost

~$10/year for now for the domain name.

Firstly, the website (tournacat.com) operates as a static site built using Hugo. It's currently hosted using Cloudflare Pages which is free.

The Google Workspace add-on is built and runs on Google Apps Script (GAS) for free.

Lastly, Tournacat has an API server running on Cloudflare Worker. It fetches the upcoming Esports events from a data source. Running on Cloudflare Worker is currently still within the free tier limit but it won't be so for long.

The neat part? Tournacat doesn't store any user info. No names, no emails — nothing at all. This means that no database is needed.

Revenue

Lemon Squeezy payout page

Today, Tournacat has about 90 users. In terms of paid subscriptions, it's made a tiny profit of $40.23 in 11 months.

Honestly, I never expected to rake in big bucks anyway. The goal was to cover growing worker usage costs. Any surplus? Well, that's coffee money.

Lessons learned

The journey with Tournacat has been a blast. The realization that a handful of people want and use what you built yourself is an indescribable feeling.

This tiny venture has taught me many valuable lessons:

Developing on the GAS platform
Publishing of a Google Calendar add-on to Workspace
Working with payments and subscriptions
Crafting a micro SaaS

Besides, the journey also introduced me to the mundane world of social media marketing, which, isn't my cup of tea. On the flip side, I had an amazing time speaking to people about their feedback (feel free to ask me anything)!

Overall, I feel like the Esports landscape is shaped by a generation accustomed to abundant free entertainment and tools. Convincing people to spend money can be a tough sell.

Any future plans?

There are plenty of tasks/tickets on the project board, but I'm taking it slow. The plan? I’ll just be working on minor improvements here and there unless specific user requests pop up.

Being a solo developer has its perks — the turnaround time for new requests is pretty quick. I've been able to incorporate feedback almost immediately (as long as it's somewhat reasonable).

Overall, the product feels pretty complete as it is.

SGS Issuance Calendar: Automated T-Bill Tracking

Started 2023

Feeling the manual-checking fatigue for the Singapore MAS T-bill issuance calendar, I decided, "Why not automate this?" So, I crafted a schedule to run every month using Google Apps Script (GAS) and detailed the process in this blog post.

T-bill announcement and auction dates are on my calendar!

How it Works

Under the hood, it's a small GAS project written in TypeScript. The script gets triggered every month; pings the MAS API, and then creates important dates (e.g. announcement/auction date) as Google Calendar events. Simple as that!

Cost

$0. The best part about using GAS is that I don’t need to be bothered by the underlying infrastructure hassle. No fretting over scaling, upgrades, backups, availability — none of that. It just does its thing.

Some lessons learned

This project brought some learnings to the table. I am now able to build GAS projects in TypeScript and delved into the world of writing unit tests for GAS. This newfound skillset later found a home in Tournacat's add-on UI codebase, which is also written in GAS.

What's next?

I've added support for SGS and SSB calendars as well! For now, no concrete plans unless user feedback or GitHub issues come knocking. Two months in, and it's been serving me well.

Closing Thoughts

The journey has been nothing short of fun. My approach to automating/building usually unfolds like this:

Identify my own pain point or problem (note it down immediately!); however small it may be. Look for quick wins!
Prefer leveraging what's already out there. Check if there's something in the market for it, like Zapier or IFTTT
If not, then roll up my sleeves and code/build
See if I can generalize the solution or approach
Write down the process somewhere
Rinse and repeat

“Aren’t some of these side projects?"

Well, I guess. I just think the term feels so worn out at this point. It's like, they're more about automating some parts of my life. Sure, one of them is making coffee money monthly, a bit laughable. The whole idea of “hustling” just doesn't quite vibe with me. I've realized I just want to do things just for fun. No, really.

Thanks for sticking around! 🍻 Here's to more automation to come!

I Built a Google Calendar Add-on. Here's What I Learnt

Jerry Ng — Tue, 02 May 2023 00:00:21 +0000

I made Tournacat, an add-on that syncs upcoming Esports schedules of tournaments to Google Calendar. Two months later, Tournacat has garnered 4 paid users and is slowly growing.

In this post, I will share my experience building Tournacat using various technologies including Cloudflare Worker, Hugo, and Google Apps Script.

Along the way, I encountered some interesting challenges and learned valuable lessons. Specifically, I want to share my anecdotal experience with issues I encountered while developing the Tournacat add-on using Google Apps Script.

Why Did I Make Tournacat

A "Month" view of upcoming Esports schedules in Google Calendar.

Tournacat started as an idea from D2GCal, a link to a Google Calendar featuring upcoming Dota 2 tournaments available for purchase on Gumroad.

I built Tournacat out of the frustration of missing out on big Dota 2 matches. I was tired of constantly scouring the internet to find out when my favorite matches were happening.

As an avid Esports fan, I knew there had to be a better way (cliché I know). So, I started building Tournacat for my own use.

Today, Tournacat has grown to support major Esports titles including Counter-Strike, Dota 2, LoL, Valorant, Overwatch, and more.

Why Google Calendar

I use Google Calendar extensively in my daily life, from scheduling meetings to keeping track of personal events. So, having an Esports schedule alongside everything in one place is a huge plus.

With Tournacat, you can just install the add-on to your Google Calendar and let it work its magic. It's a seamless solution that integrates perfectly with Google Calendar.

A "Schedule" view of upcoming Esports schedules in Google Calendar.

Existing Solutions (Calendars)

I've been on the lookout for a solution like Tournacat for years. I've tried all sorts of public Esports calendars in the past, but they always fell short in one way or another.

Either the author of the calendar stopped updating it, or it was just an unreliable source that was prone to human error. It got to the point where I gave up hope of ever finding a calendar that actually worked.

Tech Stack

Tournacat is built using a variety of tools and technologies. The tech stack includes:

Frontend — Hugo
Backend — Cloudflare Worker
Add-on — Google Apps Script

Hugo

Hugo is a static site generator that I used to create the Tournacat website. I decided to use Hugo for a couple of reasons.

First, I'm not particularly skilled in front-end development, so I wanted to use a static site generator that would allow me to use pre-built themes without having to write much front-end code myself. Hugo fit the bill perfectly in this regard.

Second, Hugo is known for its speed and ease of use. I wanted to be able to iterate quickly and not have to spend a lot of time fussing with my site's performance or configuration. Hugo's documentation and tooling made it easy to get up and running quickly. With that, I was able to focus on building out the content for Tournacat’s website rather than worrying about technical details.

I then hosted the website on Cloudflare Pages. Using Cloudflare Pages allowed me to keep costs down, as I didn't need to pay for expensive hosting services.

Cloudflare Worker

For the backend, I decided to go with Cloudflare Worker. Cloudflare Worker is a serverless computing platform that allows you to run JavaScript code on the edge.

To make my life easier, I decided to build the backend server Tournacat on top of a framework — Worktop, a lightweight Cloudflare Worker web framework that simplifies the development of APIs using TypeScript.

[

GitHub - lukeed/worktop: The next generation web framework for Cloudflare Workers

The next generation web framework for Cloudflare Workers - GitHub - lukeed/worktop: The next generation web framework for Cloudflare Workers

GitHublukeed

](https://github.com/lukeed/worktop)

Given the opportunity to start over, I would consider using Hono instead, as it appears to be more actively maintained on GitHub.

My decision to choose Cloudflare Worker was based on my comfort level with it, having done several previous projects with it. The developer experience was nothing short of amazing, and deployment was easy. Furthermore, it was affordable with a generous free tier.

While the performance was impressive, it was not the main reason why I picked it.

Google Apps Script

Google Apps Script is the core technology behind the Tournacat add-on. It is a scripting language based on JavaScript that allows you to extend and automate Google Workspace products like Google Calendar, Sheets, and Drive.

While building Google Workspace add-ons is possible with other coding languages (runtimes), I decided to go for Google Apps Script anyway due to convenience. Frankly, I never thought that I’d be impressed at how simple it is to build a workspace add-on!

Do keep in mind that it does come with some limitations and quotas.

Navigating the Bumps

Creating a Google Workspace add-on is relatively straightforward. The add-on tutorial provided by Google Workspace is pretty easy to follow.

In short, It took me a day to figure out most things and about a week to get the add-on approved and published on Google Workspace Marketplace.

Advantages

One of the pros of using Google Apps Script is that it literally cost nothing to run.
Since it uses JavaScript, it's easy for most developers to pick up and start working with.

Challenges

Although Google claims that Apps Script is now supported by the V8 runtime, it doesn't actually support Promises or async-await (Issue Tracker). Consequently, you may need to resort to using workarounds like triggers (Stackoverflow references: 1, 2).
Pushing changes to the Workspace add-on is rather unorthodox. I noticed this when one of my users couldn’t get their time-based triggers to work properly after a new deployment. Turns out the “correct” way to update a workspace add-on is to edit a versioned deployment instead of creating a new one (Stackoverflow reference).

Edit a versioned deployment with "New version" to update a workspace add-on

It has to respect the limits set by Google. However, it wasn't documented if the limits were shared across all add-on users cumulatively (hint: it is not).
Lastly, I came across an authentication-related issue (Stackoverflow references: 1, 2) when attempting to use Google Apps Script as a webhook to allow my users to automatically activate their subscriptions after payment. Some answers suggested temporarily disabling the V8 runtime to fix this.

Despite these hiccups, Google Apps Script is a great platform to work with. It gets 90% of the job done very effectively and the last 10% is the challenges that I mentioned.

Some Lessons Learned

Use what makes you the most productive. Don’t worry about picking the perfect tech stack that handles scale.
Users couldn’t care less about your code. Nobody except you cares about the fancy frameworks, design patterns, or architecture used.
Keep your deployment simple. This allows for fast iterations. While I could have gone all-in with AWS and set up a sophisticated Terraform setup, I realized that I would have taken more time to ship if I were to go this route.

Closing Thoughts

I have always had the itch to build a micro SaaS. So, I decided to build one realizing that it wouldn’t make me rich or change the world. I simply enjoy building stuff that people want to use and money is the ultimate validation for that.

However, building a SaaS was not exactly what I thought it would be — keeping my head down and coding away. I needed to understand my users and their needs and learn how to market my product effectively.

I must admit, marketing is not something I truly enjoy. But, I realized that it's a necessary job that comes with building a product.

Overall, building Tournacat was a rewarding experience. Seeing Tournacat slowly grow and gain traction has been an exciting journey. I'm eager to continue on this journey and I look forward to what the future holds.

Python Decorators: Explained in What Why When

Jerry Ng — Sun, 12 Feb 2023 16:00:00 +0000

Decorators are incredibly powerful in Python. They are useful for creating modular and reusable code. Plus, they're pretty cool once you get the hang of them!

In this blog post, we'll take a look at what decorators are, why you should use them, and when you should and shouldn't use them.

As always, I’ll try to explain everything in layman's terms. My goal is to get you comfortable with decorators.

TL;DR

Think of Python decorators as little "wrappers" that you can place around a function (or another object) to modify its behavior in some way.
You would typically use decorators for logging, authentication, or to make a function run faster by caching its result.

What are decorators

Let’s start with a simple explanation of what decorators are in Python.

A decorator is really just a function that takes in a function as an argument. It then returns a modified version of that function.

Here’s a classic example:

def my_decorator(func):
    def wrapper():
        print("Before the function is called.")
        func()
        print("After the function is called.")
    return wrapper

In this example, the my_decorator function takes a function as an argument (which we'll call func) and defines a new inner function called wrapper.
The wrapper function prints a message before and after calling the original func function.
The my_decorator function then returns the wrapper function.

To use a decorator, we would apply it to a function using the @ symbol, e.g. @my_decorator. Here’s an example:

def my_decorator(func):
    ...

@my_decorator
def greet_ben():
    print("Hello, Ben!")

If we were to call greet_ben, the output would look like this:

Before the function is called.
Hello, Ben!
After the function is called.

See how we effectively modified greet_ben to print additional messages without changing the code of the original function itself?

Syntactic sugar

You see, the decorator syntax @my_decorator is merely syntactic sugar. The 2 following function definitions are semantically equivalent:

def my_decorator(func):
    ...

# Function definition 1
def greet_ben():
    ...
greet_ben = my_decorator(greet_ben)

# Function definition 2
@my_decorator
def greet_ben():
    ...

Interesting, right?

Python decorator template

Now, hold your thought for a second, that was not how we would typically write a decorator function in Python — there are better ways.

Instead, check out this template later to learn how to write a generic decorator function.

Why use decorators

Understanding the “why” is kind of like adding the icing on the cake — it just makes the whole thing a lot sweeter. So why use decorators at all?

Well, decorators provide a flexible and convenient way to add extra functionality to existing functions, making them more reusable and modular.

This is useful when you want to add a feature to a function that is used in multiple places in your code, but you don't want to modify the original function itself.

Avoid cluttering

For example, you might want to add logging or error handling to a function. But, you don't want to clutter the original function with that extra code.

By using a decorator, you can add extra functionality without changing the original function, making your code cleaner and more organized.

When to use decorators

Now that we know why we should use decorators, let's talk about when to use them.

Generally speaking, Python decorators are most useful when you want to add extra functionality to a function without modifying its code. This might include things like:

Logging
Error handling
Caching
Authentication (see example in Flask)
Timing a function execution (see an advanced example)

A caching example

Here's an example of how you might implement a simple cache for a Fibonacci function:

cache = {}

def fib(n):
    if n in cache:
        return cache[n]

    if n <= 1:
        return n

    result = fib(n - 1) + fib(n - 2)
    cache[n] = result
    return result

print(fib(100)) # 354224848179261915075

Notice that the caching code lives inside the fib function itself.

However, with decorators, we can simply define a caching decorator and use it to add caching functionality to our Fibonacci function:

cache = {}

def cache_results(func):
    def wrapper(n):
        if n in cache:
            return cache[n]

        result = func(n)
        cache[n] = result
        return result
    return wrapper

@cache_results
def fib(n):
    if n <= 1:
        return n
    return fib(n - 1) + fib(n - 2)

print(fib(100)) # 354224848179261915075

Here, we've implemented our own simple cache using a dictionary.
We've also defined our own decorator function cache_results that takes a function as an argument and returns a new function that wraps the original function.
This inner function checks the cache to see if a result has already been calculated, and if so, it returns the cached result.
Otherwise, it calculates the result and adds it to the cache before returning it.
This way, we can apply the cache_results decorator to any function we want to add caching for.

While both examples work, the latter allows us to easily add caching to other functions without having to write the caching code inside each function.

This makes the caching code in our example more reusable and modular!

A better example with built-in modules!

Instead of reinventing the wheel, we can use the lru_cache decorator from the functools module to add caching to your fib function:

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    if n <= 1:
        return n
    return fib(n - 1) + fib(n - 2)

print(fib(100)) # 354224848179261915075

Using lru_cache decorator is really easy – all you have to do is add the decorator to your function and it automatically handles the caching for you.
So, using lru_cache can be a great way to optimize your code without a lot of extra effort.

Again, this is just one example of how you can use decorators to make your code more efficient and elegant.

When NOT to use decorators

So, when should you avoid using decorators?

Well, decorators are great for keeping your code clean and easy to read, but they can make your code harder to understand if overused or used incorrectly.

In general, it's best to avoid using decorators if they make your code more complex than it needs to be, or if you're not sure how they work.

For instance, if you're working on a very small project and you don't need to keep track of how many times your functions are called, then you probably don't need to use a decorator.

In some cases, you may find that it might be more appropriate to modify the code of the original function directly, rather than using a decorator.

It's always a good idea to keep things simple and easy to understand!

Summary

In short, Python decorators are like little "wrappers" that you can place around a function (or another object) to modify its behavior in some way.

Decorators are useful for creating modular and reusable code in Python.

You can define a decorator once and then apply it to any number of functions. This allows you to easily add the same extra functionality to multiple functions without repeating yourself.

This article was originally published at jerrynsh.com

Say Goodbye to Heroku Free Tier: Here Are 4 Alternatives

Jerry Ng — Sun, 12 Feb 2023 14:12:12 +0000

28 November 2022 was a sad day for developers. If you haven't heard, Salesforce (Heroku’s parent organization) has phased out its free tier plan on this date.

For many years, Heroku has been the de facto standard platform as a service (PaaS). So many students and developers deployed their first web application on Heroku. Anecdotally, Heroku has been pivotal for my career.

TL;DR, if you’re looking for Heroku free tier alternatives, check out free-for.dev/#/?id=paas. I migrated my Cron jobs to Northflank and Heroku Dynos to Koyeb.

A lot has happened

For context, a lot has happened in the year 2022 for Heroku. The two most notable events are:

On April 2022, Heroku had a security breach (incident) where CI and Review App secrets were compromised. GitHub Actions integrations on Heroku were down for a couple of months. Comms from Heroku were objectively bad. I experienced this firsthand when I had to resort to using a GitHub CI Action for my app deployments.
On August 2022, Heroku announced the removal of their free product plans.

If you have been keeping an eye on the Internet, you would see people expressing concerns about how Heroku is losing its magic. Ever since the security breach incident, I see more talks about Heroku alternatives taking place.

Though things started to look bad, whatever Heroku had to offer was still great and I decided to stick to it. Until now, that is.

Heroku alternatives

For the past couple of weeks, I have been moving my demos and tiny projects out of Heroku.

If you’re still looking for Heroku free tier alternatives, check out free-for.dev (GitHub).

While there are a bunch of blog posts and recommendations scattered on the Internet now, free-for.dev provides the best coverage. The list comes with a brief description of what each Heroku alternative does.

What I was looking for

Do note that my use case is mostly for small-scale personal projects. So, whatever I have written may not be ideal for your use case.

Equivalent free tiers in terms of CPU and RAM. (spoiler: none of them were as good)
Support for Cron jobs out of the box. Ideally, no weird workaround is required
Custom domain
GitHub integration
Support for basic logging and monitoring (i.e. CPU, memory, disk, network metrics)

The bandwidth cost and supported region of each PaaS are not my primary concerns.

My picks

Heroku Postgres — since May 2022, I have migrated from Heroku Postgres to Railway. So far, I have no complaints about it. Another alternative that I was looking at was PlanetScale. However, I didn’t go for it because it isn’t Postgres-compatible.

Heroku Dynos apps — Burplist was migrated to Koyeb. Having tried out other notable Heroku alternatives like Fly.io, Northflank, and Railway, I can safely say that the migration from Heroku to Koyeb required the least amount of effort and kinks. It just works in my case.

Heroku Scheduler — for my Cron jobs, I opted for Northflank. Their support for Cron jobs is by far the best. Pricing aside, the developer experience is much better than Heroku Scheduler.

Heroku Add-ons — not applicable to my case. I was relying on Heroku Add-ons for monitoring and logging. Thankfully most of the modern PaaS today support that out of the box.

Some thoughts

Woefully, most Heroku alternatives’ free tiers aren’t as good as Heroku. For example, most of the free compute instances provided are 1 shared CPU and 256 MB RAM (Heroku Free Dynos started at 512 MB RAM). Not considering the limited number of apps allowed yet.

To my surprise, most of these PaaS still don’t support running Cron jobs natively. Shoutout to Northflank again for providing such capability with great developer experience.

For Koyeb, the supported regions for free tiers are quite limited at the time of writing this.

Lastly, render.com is another popular alternative out there in the market. You may want to check them out.

Cost Comparisons

Updated as of 1 Dec 2022.

Free tiers

Pricing

Free tiers aside, most of these Heroku alternatives bill you for the resources you use by the second/minute/hour.

One thing that I did not cover here is bandwidth cost (inbound + outbound). This is something you might want to seriously consider for bigger projects.

They feel… different

Perhaps I am biased. There aren’t any obvious drop-in replacements for Heroku free tiers (at least not on par with Heroku’s generosity). Throwing free tiers aside, most of them don’t offer the same kind of developer experience as Heroku.

Heroku free tiers are one of the best things that have ever happened to software engineering in my opinion.

I really appreciated Heroku’s free dyno hours. I didn’t mind the fact that dynos goes to sleep and only gets woken up when it’s needed — this means I could host multiple demos at once and only consume my free limits on demand.

Need to show something to someone quickly? Simply send them your URL. It’ll be up in a couple of seconds and I am totally cool with the cold start. This developer experience is something all the existing platforms cannot give, at least not with their free tiers.

While I’m a happy paying customer, it stings to pay $5/mo for something that I do not use 99% of the time.

Closing Thoughts

Sadly, it is how it is. The Internet is a gnarly place. If you are offering free service on the Internet, people will find a way to abuse it. I remember hosting my first URL shortener. On my first day of making it public, the service immediately got spammed. I had to implement CAPTCHA and expiring links to mitigate abuse.

Today, whenever I see something available for “truly free” (if you know what I mean) created by individuals, I can’t help but wonder how are they going to keep the lights on sustainably out of goodwill. I just hate to see someone’s good intentions fail. Maybe we should start paying for Internet stuff.

Though I use cloud services like AWS almost every day, PaaS like Heroku always holds a special place in my heart.

Heroku was amazing. I think it did a great job of raising the bar in the PaaS field besides indirectly advocating free education (in one way or another).

Thank you, Heroku, from the bottom of my heart.

This article was originally published on jerrynsh.com

Stop Comparing JWT vs Cookies

Jerry Ng — Thu, 01 Dec 2022 23:30:43 +0000

There is a lot of confusion about cookies, sessions, token-based authentication, and JWT.

Today, I want to clarify what people mean when they talk about “JWT vs Cookie, “Local Storage vs Cookies,” “Session vs token-based authentication,” and “Bearer token vs Cookie” once and for all.

Here’s a hint — we should stop comparing JWT and Cookies!

Along the line, I’ll go through what XSS and CSRF attacks are and how to prevent them using token-based authentication with JWT and CSRF tokens.

Let’s begin!

Overview

Terminology Speed Run
- Client
- Server
- Request/Response Headers
- Cookie
- XSS Attack
- CSRF Attack
- Cookies Storage
- Web Storage
- Cookies (Storage) vs Web Storage
JWT
- So, why use JWT?
- Stop comparing JWT and Cookie
Session-based vs Token-based Authentication
Cookie vs Bearer Tokens
CSRF Prevention
- Same-site Cookies
- Common CSRF prevention methods
- The modified “cookie-to-header token” approach
Conclusion
- A Demo, perhaps?
Reference

Terminology Speed Run

To start, it’s important to know the differences between some of these terminologies.

Without explicitly stating these, it would be unclear for us to compare things properly.

Client

Our client application. In this context, we are specifically talking about our web browsers, e.g., Firefox, Brave, Chrome, etc.

Server

Computers that are doing all the magic behind the curtains.

Request/Response Headers

HTTP headers. Note that they are case-insensitive.

Cookie

Aka “HTTP cookie,” “web cookie,” or “browser cookie.”

A small piece of information that a server sends back to the client.

Stored in the browser’s Cookies storage, cookies are typically used for authentication, personalization, and tracking.

A cookie is received in name-value pairs via the Set-Cookie response header in a request. With this, your cookie will automatically be kept in the browser’s Cookies storage (document.cookie).

An example of how a Cookie is received. You should never publicly share your JWT! (this JWT is no longer in use).

Cookies with HttpOnly, Secure, and SameSite=Strict flags are more secure.

For example, with the HttpOnly flag, the cookies are not accessible through JavaScript, thus making it immune to XSS attacks.

With HttpOnly, the cookie is not shown

Read more on MDN and check out what other flags do (they are handy).

XSS Attack

Aka “Cross-Site Scripting” attack.

For context, the web storage (e.g., local storage) is accessible through JavaScript on the same domain. Consequently, web storage is vulnerable to XSS attacks.

In short, XSS is a type of vulnerability where an attacker injects JavaScript that will run on your page.

Basic XSS attacks attempt to inject JavaScript through form inputs, where the attacker puts an alert(localStorage.getItem('your-secret-token')) into a form to see if it is run by the browser and can be viewed by other users.

If you still don’t get what XSS is, check out this “XSS Explained” video.

CSRF Attack

Aka “Cross-Site Request Forgery” attack.

Cookies are vulnerable to CSRF attacks. No cookies = no CSRF attacks.

As browsers automatically send Cookies with all requests, CSRF attacks use this to gain authenticated access to a trusted site. Need more? Read CSRF in simple words.

To protect your site against CSRF attacks while using Cookies (with SameSite=None), check out this StackOverflow answer. Next, read more about CSRF prevention on Wikipedia (Wikipedia is generally too dry to my liking, but this is good!).

Still don’t get what CSRF is? Check out this “CSRF Explained” video.

Cookies Storage

Aka “Cookie Jar” or “Cookies.” Yup. I can already see why it is confusing for many.

Client-side storage where HTTP cookies are stored.

Here’s an important note: Browsers automatically send cookies (no client-side code needed) along with every request via the cookie request header. This is exactly why Cookie (storage) is vulnerable to CSRF attacks.

XSS attacks can be prevented when using Cookies storage if the cookie is set with the HttpOnly flag.

To view Cookies: F12 → ‘Application’ → ‘Storage’ → ‘Cookies’

Web Storage

Aka “Local/Session Storage.”

Client-side storage. They are typically used to store data in key-value pairs.

Vulnerable to XSS attacks. Hence, not ideal for storing private/sensitive/authentication-related data.

You can get any item on your Local Storage using JavaScript

To view Local Storage items: F12 → ‘Application’ → ‘Storage’ → ‘Local Storage’ / ’Session Storage’

sessionStorage — data is persisted only for the duration of the page session

localStorage — data is persisted even when the browser is closed and reopened

Cookies (Storage) vs Web Storage

	Local/Session Storage	Cookies (Storage)
JavaScript	Accessible through JavaScript on the same domain	Cookies, when used with the HttpOnly cookie flag, are not accessible through JavaScript
XSS attacks	Vulnerable to XSS attacks	Immune to XSS (with HttpOnly flag)
CSRF attacks	Immune to CSRF attacks	Vulnerable to CSRF attacks
Mitigation	Do not store private/sensitive/authentication-related data here	Make use of CSRF tokens or other prevention methods

JWT

Aka “JSON Web Tokens.”

Commonly used for authentication and authorization.

JWT is an open standard (RFC 7519). Meaning all JWTs are tokens.

Typically, JWT is stored in Local Storage or Cookies (storage).

Remember, JWT is not encrypted by any means.

Rather, it is encoded in Base64. Try decoding any JWT on jwt.io.

So, why use JWT?

Often used with token-based authentication, horizontal scaling is easier when using JWT.

Why? The verification of JWT does not require any communication between the servers and databases. In other words, the authentication can be stateless.

Stop comparing JWT and Cookie

Neither JWT nor Cookie are authentication mechanisms on their own.

JWT is simply a token format.

A cookie is an HTTP state management mechanism.

As demonstrated, a web cookie can contain JWT and can be stored within your browser’s Cookies storage.

So, we need to stop comparing JWT and Cookie.

Session-based vs Token-based Authentication

Rather, the right question to ask is, “what is the difference between token-based authentication and session-based authentication?”

Token-based	Session-based
Stateless	Stateful
The authentication state is NOT stored anywhere on the server-side	The authentication state is stored on the server-side (DB)
Easier to scale horizontally	Harder to scale horizontally
Commonly uses JWT for authentication	Commonly uses Session ID
Typically sent to the server via an HTTP Request `Authorization` Header (e.g. `Bearer <token>)`. Can use `Cookie` too	Usually sent to the server in the `Cookie` request header
Harder to revoke a user session	Able to revoke user session with ease

Cookie vs Bearer Tokens

Now, we know how cookies work. Let’s take a stab at the term “Bearer tokens.” Let’s assume we’ll use JWT as our authentication token from hereon.

What people call a “Bearer token” is a string (e.g., JWT) that goes into the Authorization header of any HTTP request. Unlike a browser cookie, it is not automatically stored anywhere, thus making this CSRF impossible.

To make use of a “Bearer token,” we’ll need to explicitly store the JWT somewhere in our client (Cookies storage or Local Storage) and add that JWT to our HTTP Authorization header while making requests.

If your cookie containing a JWT is set with the HttpOnly flag, retrieving your token from the client side would be impossible with JavaScript.

“Wait, how about we use Local Storage then?”

Remember, using Local Storage makes our JWT vulnerable to XSS. As a result, you’ll often hear people advise strongly against storing JWT in Local Storage.

At this point, it may sound like using Cookie to store JWT is our only option. But remember, this makes our website vulnerable to CSRF attacks!

CSRF Prevention

Same-site Cookies

Same-site cookies can effectively prevent CSRF attacks. Though, it comes with other caveats. Read more here.

What follows assumes that we’re not going to use SameSite cookies.

Common CSRF prevention methods

Leaving JWT behind for a bit, these two are some of the most common CSRF prevention methods:

Check out this StackOverflow answer for a quick summary of the two approaches above.

Cool. But now comes the question — how can we do this with JWT?

The modified “cookie-to-header token” approach

Honestly, I’m not too sure if this approach has a proper name. I found this approach from this wonderful talk about 100% Stateless with JWT (JSON Web Token) by Hubert Sablonnière. I highly recommend you check it out. It’s worth the hour.

In short, the modified approach (in my opinion) looks similar to the original Cookie-to-header token approach except with a few tweaks:

the anti-CSRF token is returned in a separate response header (e.g. X-CSRF-Token) instead of the Set-Cookie response header
we sign and set a JWT on the Set-Cookie response header

This implementation can be found on this Cloudflare Worker demo and GitHub.

The user logs in, and the server signs a JWT with csrfToken as part of the JWT claim.

{
  "email": "your@email.com",
  "exp": 1666798498,
  "csrfToken": "1449bd3e-41c2-45cb-a538-73c7ad80ca2c",
  "iat": 1666794898
}

The generated csrfToken should be unpredictable and unique per-user session.

The JWT would then be stringified into a cookie set into the Set-Cookie response header. The randomly generated csrfToken , on the other hand, will be set in the X-CSRF-Token response header.
With the Set-Cookie header present in the response header, our browser would automatically store the JWT in the Cookies (storage). The csrfToken present in the X-CSRF-Token header will be extracted and set in the browser’s local storage.
When a request (e.g., GET /hello) is triggered, our browser will get the csrfToken from the local storage.
The JWT from the Cookies (storage) and the csrfToken retrieved from the local storage will be sent to the server in our request header.
The server will verify the JWT and check the csrfToken from the request header with the csrfToken claim inside the JWT to verify if the CSRF Token is valid.

Conclusion

In essence, as long as authentication isn’t automatic (such as in browsers with Cookies), you don’t have to worry about CSRF attacks.

For example, if your application attaches authentication credentials via a Authorization header, CSRF isn't possible as the browser can't authenticate the request automatically.

Lastly, we need to stop advocating that any of our comparisons is better. That is not how it works. Rather, we should think about the tradeoffs that we are making.

A Demo, perhaps?

The screenshots in this post are taken from this Cloudflare Worker demo I made. The source code of this demo can be found on GitHub.

To test a CSRF attack, check out this little tool (credits: Shrikant).

That aside, I created a branch name csrf-attack-demo where you could run that locally to simulate a CSRF attack on our demo site.

Reference

I owe my thanks to all of the following references:

This article was originally published on jerrynsh.com

JWT vs Cookie: Why Comparing the Two Is Misleading

Jerry Ng — Tue, 01 Nov 2022 00:00:19 +0000

There is a lot of confusion about cookies, sessions, token-based authentication, and JWT.

Today, I want to clarify what people mean when they talk about “JWT vs Cookie, “Local Storage vs Cookies”, “Session vs token-based authentication”, and “Bearer token vs Cookie” once and for all.

Here’s a hint — we should stop comparing JWT vs Cookies!

Along the line, I’ll go through what XSS and CSRF attacks are and how to prevent them using token-based authentication with JWT and CSRF tokens.

Let's begin!

Anatomy

To start, it’s important to know the differences between some of these terminologies.

Without explicitly stating these, it would be unclear for us to compare things properly.

Client

Our client application. In this context, we are specifically talking about our web browsers, e.g. Firefox, Brave, Chrome, etc.

Server

Computers that are doing all the magic behind the curtains.

Request/Response Headers

HTTP headers. Note that they are case-insensitive.

Cookie

a.k.a “HTTP cookie”, “web cookie”, or “browser cookie”.

A small piece of information that a server sends back to the client.

Stored in the browser’s Cookies storage, cookies are typically used for authentication, personalization, and tracking.

A cookie is received in name-value pairs via the Set-Cookie response header in a request. With this, your cookie will automatically be kept in the browser’s Cookies storage (document.cookie).

An example of how a Cookie is received. You should never publicly share your JWT! (this JWT is no longer in use).

Cookies with HttpOnly, Secure, and SameSite=Strict flags are more secure.

For example, with the HttpOnly flag, the cookies are not accessible through JavaScript, thus making it immune to XSS attacks.

With HttpOnly, the cookie is not shown

Read more on MDN and check out what other flags do (they are handy).

XSS Attack

a.k.a “ Cross - S ite S cripting” attack.

For context, the Web Storage (e.g. Local Storage) is accessible through JavaScript on the same domain. Consequently, Web Storage is vulnerable to XSS attacks.

In short, XSS is a type of vulnerability where an attacker injects JavaScript that will run on your page.

If you still don’t get what XSS is, check out this “XSS Explained” video.

CSRF Attack

a.k.a “ C ross- S ite R equest F orgery” attack.

Cookies are vulnerable to CSRF attacks. No cookies = no CSRF attacks.

As browsers automatically send Cookies with all requests, CSRF attacks make use of this to gain authenticated access to a trusted site. Need more? Read CSRF in simple words.

Still don’t get what CSRF is? Check out this “CSRF Explained” video.

Cookies Storage

a.k.a “Cookie Jar”, or “Cookies”. Yup. I can already see why it is confusing for many.

Client-side storage where HTTP cookies are stored.

Here’s an important note — browsers automatically send cookies (no client-side code needed) along with every request via the cookie request header. This is exactly why Cookie (storage) is vulnerable to CSRF attacks.

XSS attacks can be prevented when using Cookies storage if the cookie is set with the HttpOnly flag.

To view Cookies: F12 → ‘Application’ → ‘Storage’ → ‘Cookies’

Web Storage

a.k.a “Local/Session Storage”.

Client-side storage. They are typically used to store data in key-value pairs.

Vulnerable to XSS attacks. Hence, not ideal for storing private/sensitive/authentication-related data.

You can get any item on your Local Storage using JavaScript

To view Local Storage items: F12 → ‘Application’ → ‘Storage’ → ‘Local Storage’ / ’Session Storage’

sessionStorage — data is persisted only for the duration of the page session

localStorage — data is persisted even when the browser is closed and reopened

Cookies (Storage) vs Web Storage

	Local/Session Storage	Cookies (Storage)
JavaScript	Accessible through JavaScript on the same domain	Cookies, when used with the HttpOnly cookie flag, are not accessible through JavaScript
XSS attacks	Vulnerable to XSS attacks	Immune to XSS (with HttpOnly flag)
CSRF attacks	Immune to CSRF attacks	Vulnerable to CSRF attacks
Mitigation	Do not store private/sensitive/authentication-related data here	Make use of CSRF tokens or other prevention methods

JWT

a.k.a “JSON Web Tokens”.

Commonly used for authentication and authorization.

JWT is an open standard (RFC 7519). Meaning all JWTs are tokens.

Typically, JWT is stored in Local Storage or Cookies (storage).

Remember, JWT is not encrypted by any means. Rather, it is encoded in Base64. Try decoding any JWT on jwt.io.

So, why use JWT?

Often used with token-based authentication, horizontal scaling is easier when using JWT.

Why? The verification of JWT does not require any communication between the servers and databases. In other words, the authentication can be stateless.

Stop comparing JWT & Cookie

Neither JWT nor Cookie are authentication mechanisms on their own.

JWT is simply a token format.

A cookie is an HTTP state management mechanism really.

As demonstrated, a web cookie can contain JWT and can be stored within your browser’s Cookies storage.

So, we need to stop comparing JWT vs Cookie.

Session-based vs Token-based Authentication

Rather, the right question to ask is “What is the difference between token-based authentication and session-based authentication?”

Token-based	Session-based
Stateless	Stateful
The authentication state is NOT stored anywhere on the server-side	The authentication state is stored on the server side (DB)
Easier to scale horizontally	Harder to scale horizontally
Commonly uses JWT for authentication	Commonly uses Session ID
Typically sent to the server via an HTTP Request `Authorization` Header (e.g. `Bearer <token>)`. Can use `Cookie` too	Usually sent to the server in the `Cookie` request header
Harder to revoke a user session	Able to revoke user session with ease

Cookie vs Bearer Tokens

Now, we know how cookies work. Let’s take a stab at the term “Bearer tokens”. Let’s assume we’ll use JWT as our authentication token from hereon.

What people call a “Bearer token” is a string (e.g. JWT) that goes into the Authorization header of any HTTP request. Unlike a browser cookie, it is not automatically stored anywhere, thus making this CSRF impossible.

GET <http://www.example.com>
Authorization: Bearer my_bearer_token_value // HTTP Request Header

To make use of a “Bearer token”, we’ll need to explicitly store the JWT somewhere in our client (Cookies storage or Local Storage) and add that JWT to our HTTP Authorization header while making requests.

If your cookie (e.g. with a JWT) is set with the HttpOnly flag, retrieving your token from the client side would be impossible with JavaScript.

“Wait, how about we use Local Storage then?”

Remember, using Local Storage makes our JWT vulnerable to XSS. As a result, you’ll often hear people advise strongly against storing JWT in Local Storage.

At this point, it may sound like using Cookie to store JWT is our only option. But remember, this makes our website vulnerable to CSRF attacks!

CSRF Prevention

Same-site Cookies

Same-site cookies can effectively prevent CSRF attacks. Though, it comes with other caveats. Read more here.

What follows assumes that we're not going to use SameSite cookies.

Common CSRF prevention methods

Leaving JWT behind for a bit, these 2 are some of the most common CSRF prevention methods:

Check out this StackOverflow answer for a quick summary of the 2 approaches above.

Cool. But now comes the question — how can we do this with JWT?

The Modified "Cookie-to-header token" Approach

Honestly, I’m not too sure if this approach has a proper name. I found this approach from this wonderful talk about 100% Stateless with JWT (JSON Web Token) by Hubert Sablonnière. I highly, highly recommend you to check it out. It’s worth the hour.

In short, the modified approach (in my opinion) looks similar to the original Cookie-to-header token approach except with a few tweaks:

the anti-CSRF token is returned in a separate response header (e.g. X-CSRF-Token) instead of the Set-Cookie response header
we sign and set a JWT on the Set-Cookie response header

This implementation can be found on this Cloudflare Worker demo and GitHub.

Explanation:

The user logs in, the server would sign a JWT with csrfToken as part of the JWT claim (for verification in Step 6).

{
  "email": "your@email.com",
  "exp": 1666798498,
  "csrfToken": "1449bd3e-41c2-45cb-a538-73c7ad80ca2c",
  "iat": 1666794898
}

The generated csrfToken should be unpredictable and unique per-user session.

The JWT would then be stringified into a cookie which will be set into the Set-Cookie response header. The randomly generated csrfToken on the other hand will be set in the X-CSRF-Token response header.
With the Set-Cookie header present in the response header, our browser would automatically store the JWT in the Cookies (storage). The csrfToken present in the X-CSRF-Token header will be extracted and set in the browser’s Local Storage.
When a request (e.g. GET /hello) is triggered, our browser will fetch the csrfToken from the Local Storage.
The JWT from the Cookies (storage) and the csrfToken retrieved from the Local Storage will be sent to the server in the request header.
The server will verify the JWT and check csrfToken from the request header against the csrfToken claim inside the JWT to verify if the CSRF Token is valid.

Demo

The screenshots in this post are taken from this Cloudflare Worker demo I made. The source code of this demo can be found on GitHub.

[

GitHub - ngshiheng/worker-auth: To demonstrate and implement a PoC to protect your site against XSS & CSRF attacks.

To demonstrate and implement a PoC to protect your site against XSS & CSRF attacks. - GitHub - ngshiheng/worker-auth: To demonstrate and implement a PoC to protect your site against XSS & C...

GitHubngshiheng

](https://github.com/ngshiheng/worker-auth)

To test a CSRF attack, check out this little tool (credits: Shrikant).

I created a branch name csrf-attack-demo where you could run that locally to simulate a CSRF attack on your site.

Conclusion

In essence, as long as authentication isn't automatic (such as in browsers with Cookies), you don't have to worry about CSRF attacks.

For example, if your application attaches authentication credentials via a Authorization header, then CSRF isn't possible as the browser can't automatically authenticate the request.

Lastly, we need to stop advocating that any one of our comparisons is better than another. That is not how it works. Rather, we should think about the tradeoffs that we are making.

Reference

I owe my thanks to all of the following references:

4 Levels of How To Use Makefile

Jerry Ng — Mon, 31 Oct 2022 14:56:06 +0000

I love Makefile. Today, I use Makefile for most of the projects that I am working on. You may have seen Makefile in many open-source projects on GitHub (e.g. this). Probably like me, you have wondered what Makefiles are and what they do.

There is a bazillion of incredible Makefile tutorials out there. My goal is to get you interested enough to start using Makefile in under 5 minutes. By the end, you should know enough to start using and self-learn about Makefile.

TL;DR: read Level 1 & Level 2

What Are Make and Makefile

In short, Makefile is a special format file with rules that tells the GNU Make utility tool (i.e. make) how to execute commands that run on *nix. Typically, Make is used to compile, build, or install the software.

While Makefile is commonly used to compile C or C++, it is NOT limited to any particular programming language. You can use Make for all sorts of stuff:

Execute a chain of commands to set up your dev environment
Automate build
Run test suites
Deployments, etc.

Why Use Makefile

Compiling source code can be cumbersome, especially when you have to include multiple source files.

At its core, Makefile is a utility for writing and executing a series of command-line instructions for things like compiling code, testing code, formatting code, running code, etc.

Basically, it helps to automate your development workflows into simple commands (make build, make test, make format, make run).

Also:

make is preinstalled at most *nix systems that you can find
make is programming language/framework agnostic

Alright enough talking, let’s get to the real deal.

A Quick Guide to Makefile

While going through the levels below, I highly encourage you to create a Makefile (note: always name it Makefile) and try things out yourself.

Level 1: “Tell me all I need to know”

At this level, you’ll learn the basics of Makefile; probably all you ever need to know to benefit from it.

Suppose you have a project using that uses Docker. To build and run your app iteratively using a Docker container, you would typically:

Run docker build
Make sure there are no running container
Run docker run
Repeat

To do this yourself:

docker build -t image-name . --build-arg ENV_VAR=foo
docker stop container-name || true && docker rm container-name || true
docker run -d -e ANOTHER_VAR=bar--name container-name image-name

What a hassle! There is a lot to remember and type. On top of that, many chances to make silly mistakes.

Sure, you could just run all 3 commands every time you make changes. That would work. However, it is just so inefficient. Instead, we could write a Makefile just like the below:

all: build stop run # build -> stop -> run

build:
    @docker build -t image-name . --build-arg ENV_VAR=foo

stop:
    @docker stop container-name || true && docker rm container-name || true

run:
    @docker run -d -e ANOTHER_VAR=bar--name container-name image-name

Now to build and run a new Docker image, all you need is a single make all command. In the case above, we can just call make because all is the first rule (note: the first rule is selected by default).

To summarize, a rule within a Makefile generally looks like this:

# run `make <target>` to run this rule
<target>:   <prerequisite 1> <prerequisite 2> <prerequisite N> # a comment block is prefixed with a "#"
    <command 1>
    <command 2>
    <command N>

Takeaways for Level 1:

<target> — typically file name; could be any name
<command> — typically *nix commands/steps used to make the <target>. These MUST start with a TAB character
<prerequisite> — optional. This tells make that each prerequisite must exist before the commands are run. Therefore, prerequisites run in order of 1 to N (in the example above).
What is the @ syntax for? If a command line starts with @, the echoing of that command is suppressed (reference)
The first <target> is selected by default when running just make

That’s it. Simple right?

Now go forth and make a Makefile at work!

Level 2: “Cool, but I need a little bit more”

The use of variable substitution is rather common when it comes to all aspects of programming. Makefile is not exempted.

So, how to use environment variables (with default)?

Suppose you want to docker build your app with different build arguments (e.g. ENV_VAR), here’s how your Makefile would look like:

NAME := my-app
DOCKER := $(shell command -v docker 2> /dev/null)
ENV_VAR := $(shell echo $${ENV_VAR-development}) # NOTE: double $ for escaping
ENV_ANOTHER_VAR ?= bar

.PHONY: build
build:  ## build docker image based on shell ENV_VAR
    @if [ -z $(DOCKER) ]; then echo "docker is missing."; exit 2; fi # tip
    @docker build -t $(NAME) . --build-arg ENV_VAR=$(ENV_VAR)

Takeaways for Level 2:

.PHONY — by default make assumes your <target> is a file. So if they are not, just mark them with .PHONY in case you have a filename that is the same as <target> (reference and read this for more info)
Declare a Makefile variable with = or := syntax (reference)
:= — to execute a statement once
= — to execute a statement every time. An example use case is when you need a new date value every time you call a function.
?= — set the Makefile variable only if it's not set or doesn't have a value (reference)
Finally, you can use an environment variable to check if a command exists too as shown in the if statement above
Be careful about adding in-line comments as below. You may end up adding extra space to your variable:
```
FOO = /my/path/to # <- a white space!
```

Bonus:

echo $${ENV_VAR-development} sets ENV_VAR shell variable based on your current shell environment with a default value to development.
Alternatively, make allows you to pass variables and environment variables from the command line, e.g. ENV_VAR=development make build

By now, you probably have everything you need to create a Makefile for small and medium-sized projects.

Now go forth and make awesomeness with Makefile!

Level 3: “Now, show me the fancy stuff”

This is probably my favorite part. Everyone loves a good help message right? Right…?

Let’s create a useful help target in case our users need help on how to use Make in our project:

.DEFAULT_GOAL := help

.PHONY: help
help:
    @echo "Welcome to $(NAME)!"
    @echo "Use 'make <target>' where <target> is one of:"
    @echo ""
    @echo "  all    run build -> stop -> run"
    @echo "  build  build docker image based on shell ENV_VAR"
    @echo "  stop   stop docker container"
    @echo "  run    run docker container"
    @echo ""
    @echo "Go forth and make something great!"

all: build stop run

.PHONY: build
build:
    @docker build -t image-name . --build-arg ENV_VAR=foo

.PHONY: stop
stop:
    @docker stop container-name || true && docker rm container-name || true

.PHONY: run
run:
    @docker run -d -e ANOTHER_VAR=bar--name container-name image-name

.DEFAULT_GOAL — remember how I said the first rule is selected by default when you run just make? You can override that with this
With this, you can simply run make to display the error message every time

However, every time you add a new target to your Makefile, you’ll need to add a new line of echo.

Now, how does a self-documenting Makefile help message sound to you?

Modify your help target as follow (modify accordingly):

.DEFAULT_GOAL := help

.PHONY: help
help:  ## display this help message
    @awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n  make \033[36m<target>\033[0m\n\nTargets:\n"} /^[a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-10s\033[0m %s\n", $$1, $$2 }' $(MAKEFILE_LIST)

Next, add comments with the ## tag to print the comments as part of the help message:

all: build stop run ## run build -> stop -> run

.PHONY: build
build:  ## build docker image
    @docker build -t image-name . --build-arg ENV_VAR=foo

.PHONY: stop
stop:   ## stop running container
    @docker stop container-name || true && docker rm container-name || true

.PHONY: run
run:    ## run docker container
    @docker run -d -e ANOTHER_VAR=bar--name container-name image-name

Bam! Here you have it, a nice self-documenting help message:

$ make

Usage:
  make <target>

Targets:
  help        display this help
  all         run build -> stop -> run
  build       build docker image
  stop        stop running container
  run         run docker container

Credits: I would like to thank the author of make help - Well documented Makefiles for this.

Level 4: “Your article does not provide the level of details I need!”

Here are some Makefile references that I’ve curated that I highly recommend you to check out; I owe my thanks to all of them:

Learn Makefiles With the tastiest examples — a really good general Makefile tutorial; I prefer to rather than going through the 180+ pages long manual
Tips for your Makefile with Python — if you’re working on a Python project
Need a decent example? Here’s a Makefile with 1.8k+ Stars on GitHub
If all else fails — GNU Make manual (you probably won’t need most of it)

Final Thoughts

Even if you’re writing good documentation, chances are Makefile is much more reliable. The beauty of Makefile is that it’s simply a rigorous way of documenting what you’re already doing.

Contributing to an open-source project? Refer to the Makefile to identify the development workflow.

Current State of Makefile

First appearing in 1976, the use of Makefile is known to have its quirks.

Some of the most common complaints about Make is that it has rather complicated syntax. As a result, you’ll find people who absolutely love or hate Makefile.

I think this kind of utility gets very personal. Everyone has their favorite. I for one am less interested in having to download yet another fancy program/tool/framework/library for many other reasons.

Makefile “just works”.

How To Automate Ghost Blog Backup

Jerry Ng — Mon, 22 Aug 2022 00:06:00 +0000

I’ve been writing on my self-hosted Ghost blog for some time now. In case you’re wondering, this site is hosted on a Digital Ocean Droplet.

For the most part, I felt like I was doing something inconsequential that only meant much for myself. Today, the site has grown to a size that it’d feel like a hat-flying slap to my face if I were to lose all my content.

If you’re looking for a backup solution for your self-hosted Ghost blog, you’ve come to the right place.

TL;DR: How to automate backup for your self-hosted Ghost blog to cloud storage like Google Drive

Context

Getting started with Ghost is easy. You would typically pick between:

Ghost (Pro) managed service
Self-hosted on a VPS or serverless platform like Railway

I’d recommend anyone (especially non-developers) to opt for the managed version.

Yes, it’s relatively more expensive (so is every managed service). However, it’d most likely save you a bunch of headaches (and time) that come along with self-hosting any other sites:

Backups
Maintenance
Downtime recovery
Security, etc.

In short, you’d sleep better at night.

On top of that, 100% of the revenue goes to funding the development of the open source project itself — a win-win.

“Uh, why are you self-hosting Ghost then?”

Price — nothing beats the price affordability of hosting on your own dedicated server
Knowledge gain — I’ve learned a lot from hosting and managing my own VPS

Other perks of self-hosting include customizability, control, privacy, etc. — which are great, albeit not my primary reasons.

Most importantly, all the hassles above of self-hosting came to me as fun.

Until it isn’t, I guess.

The pain of backing up Ghost

Setting up Ghost on Digital Ocean is as easy as a click of a button. Yet, there isn’t any proper in-house solution to back up your Ghost site.

From Ghost’s documentation, you can manually backup your Ghost site through Ghost Admin. Alternatively, you could use the ghost backup command.

Even so, there was no mention of database backup as of the time of writing this.

Backing up with Bash

Why Bash

Simplicity. On top of that, Bash is great for command line interaction.

What are we backing up

Two things:

Ghost content/ — which includes your site/blog content in JSON, member CSV export, themes, images, and some configuration files
MySQL database

Overview

In this article, we’re going to write a simple Bash script that does all the following steps for us.

Assuming that we already have Rclone set up, here’s an overview of what our Bash script should cover:

Optional: run requirement checks to ensure that the CLIs that we need are installed. E.g. mysqldump, rclone, etc.
Back up the content/ folder
Back up our MySQL database
Copy the backup files over to our cloud storage (e.g. Google Drive) using Rclone
Optional: clean up the generated backup files

Utility functions

Let’s create util.sh which contains a set of helper functions for our backup script.

Personally, I really like having timestamps printed on my logs, so:

#!/bin/bash

log() {
    echo "$(date -u): $1"
}

With this, we can now use log instead of echo to print text; with the timestamp using:

$ log 'Hola Jerry!'
Sun Jul 22 03:01:52 UTC 2022: Hola Jerry!

Next, we’ll create a utility function that helps to check if a command is installed:

# util.sh

# ...

check_command_installation() {
    if ! command -v $1 &>/dev/null; then
        log "$1 is not installed"
        exit 0
    fi
}

We can use this function in Step 1 to ensure that we have ghost, mysqldump, etc. installed before we start our backup process. If the CLI is not installed, we would just log and exit.

The backup script

In this section, we’ll create a backup.sh file as our main backup Bash script.

To keep our code organized, we break the steps in the overview into individual functions.

Before we begin, we’ll need to declare some variables and source our util.sh so that we can use the utility functions that we defined earlier:

#!/bin/bash

set -e

source util.sh

GHOST_DIR="/var/www/ghost/"

REMOTE_BACKUP_LOCATION="ghost_backups/"

TIMESTAMP=$(date +%Y_%m_%d_%H%M)
GHOST_CONTENT_BACKUP_FILENAME="ghost_content_$TIMESTAMP.tar.gz"
GHOST_MYSQL_BACKUP_FILENAME="ghost_mysql_$TIMESTAMP.sql.gz"

Step 1: Run checks

Check if the default /var/www/ghost directory exists. ghost CLI can only be invoked within a folder where Ghost was installed
Check if the required CLIs to run our backup are installed

# backup.sh

# ...

pre_backup_checks() {
    if [ ! -d "$GHOST_DIR" ]; then
        log "Ghost directory does not exist"
        exit 0
    fi

    log "Running pre-backup checks"
    cd $GHOST_DIR

    cli=("tar" "gzip" "mysql" "mysqldump" "ghost" "rclone")
    for c in "${cli[@]}"; do
        check_command_installation "$c"
    done
}

Step 2: Backup the content directory

Compress the content/ directory into a .gz file

# backup.sh

# ...

backup_ghost_content() {
    log "Dumping Ghost content..."
    cd $GHOST_DIR

    tar -czf "$GHOST_CONTENT_BACKUP_FILENAME" content/
}

Step 3: Backup MySQL database

Fetch all the necessary database credentials (username, password, DB name) from the Ghost CLI
Run a check to ensure that we are able to connect to our MySQL database using the credentials above
Create a MySQL dump and compress it into a .gz ****file

# backup.sh

# ...

check_mysql_connection() {
    log "Checking MySQL connection..."
    if ! mysql -u"$mysql_user" -p"$mysql_password" -e ";" &>/dev/null; then
        log "Could not connect to MySQL"
        exit 0
    fi
    log "MySQL connection OK"
}

backup_mysql() {
    log "Backing up MySQL database"
    cd $GHOST_DIR

    mysql_user=$(ghost config get database.connection.user | tail -n1)
    mysql_password=$(ghost config get database.connection.password | tail -n1)
    mysql_database=$(ghost config get database.connection.database | tail -n1)

    check_mysql_connection

    log "Dumping MySQL database..."
    mysqldump -u"$mysql_user" -p"$mysql_password" "$mysql_database" --no-tablespaces | gzip >"$GHOST_MYSQL_BACKUP_FILENAME"
}

Step 4: Copying the compressed backup files to a cloud storage

# backup.sh

# ...

rclone_to_cloud_storage() {
    log "Rclone backup..."
    cd $GHOST_DIR

    rclone_remote_name="remote"

    rclone copy "$GHOST_DIR/$GHOST_CONTENT_BACKUP_FILENAME" "$rclone_remote_name:$REMOTE_BACKUP_LOCATION"
    rclone copy "$GHOST_DIR/$GHOST_MYSQL_BACKUP_FILENAME" "$rclone_remote_name:$REMOTE_BACKUP_LOCATION"
}

Step 5: Clean up the backup files

# backup.sh

# ...

clean_up() {
    log "Cleaning up old backups..."
    cd $GHOST_DIR

    rm -r "$GHOST_CONTENT_BACKUP_FILENAME"
    rm -r "$GHOST_MYSQL_BACKUP_FILENAME"
}

Finally, we shall invoke all of the functions defined for Steps 1 — 5.

# At the end of the backup.sh

# ...

log "Welcome to Wraith"
pre_backup_checks
backup_ghost_content
backup_mysql
rclone_to_cloud_storage
clean_up
log "Completed backup to $REMOTE_BACKUP_LOCATION"

And… we’re done!

The final code

You may find the code at github.com/ngshiheng/wraith.

To use this project directly:

SSH into your VPS where you host your Ghost site
Set up Rclone (important)
Clone this repository
Run ./backup.sh from the wraith directory

Automating Backup with Cron

I despise doing manual maintenance and administrative tasks. Let’s schedule a regular backup for our Ghost site to ease our pain using Crontab:

Run crontab -e
For example, you can run a backup at 5 a.m every Monday with:

# m h  dom mon dow   command
0 5 * * 1 cd /path/to/backup_script/ && ./backup.sh

Do take timezone into consideration when you set your Cron schedule.

Summary

Regardless of the fact that whether you’re just running a simple personal website or a proper business, having a proper backup is critical.

If a large, well-architected distributed system can go down for days, so can your $5/month Digital Ocean droplet.

I Export PostgreSQL Queries to Google Sheets For Free. Here’s How

Jerry Ng — Tue, 05 Jul 2022 23:58:55 +0000

A couple of years ago, I wouldn’t have known that connecting a PostgreSQL database to Google Sheets could be this expensive.
Despite being a trivial problem, existing market solutions such as Zapier, KPIBees, etc. require us to pay a premium for it.

TL;DR In this article, I am writing about how I was able to export PostgreSQL queries to Google Sheets via GitHub (and a little bit of Bash scripting).

Why Did I Need It

Here’s a little bit of context.

I maintain a tiny side project named Fareview — a commercial beer price monitoring tool that scrapes commercial beer data from Singapore e-commerce sites and stores it in a PostgreSQL database.
The summary of the data gathered is then synced daily to Google Sheets for users to view.

Instead of paying for a monthly premium, I’ve decided to use GitHub Action to help me with such a task for free.

Here’s How It Works

This method should also work with any other SQL databases (e.g. MySQL) with a CLI like psql.

Create a simple Bash script that uses Postgres client CLI (psql) to run the SQL queries and output them in CSV file format from our PostgreSQL database server
Set up a GitHub Actions workflow that runs the Bash script in Step 1 and commits the generated file into our repository on a Cron schedule
On Google Sheets, use the =IMPORTDATA("<url-of-csv-file>") function to import our CSV data from our repository to our Google Sheets

It is important to note that the IMPORTDATA function updates data automatically at up to 1-hour intervals. In case you need a shorter interval use case, you may need to work around it.

Here Are the Steps To Do It

Bash Script

Depending on your use case, you may not even need a Bash script. For instance, you could just run the psql command as one of the steps within your GitHub Actions workflow.
Using a Bash script here provides more flexibility as you could also run this manually outside of GitHub Actions in case you need it.

This is the Bash script (generate_csv.sh) that I’m running:

#!/bin/bash
BRANDS=("carlsberg" "tiger" "heineken" "guinness" "asahi")

PGDATABASE="${PGDATABASE-fareview}"
PGHOST="${PGHOST-localhost}"
PGPASSWORD="${PGPASSWORD-}"
PGPORT="${PGPORT-5432}"
PGUSER="${PGUSER-postgres}"

mkdir -p data/
for brand in "${BRANDS[@]}"; do
    PGPASSWORD=$PGPASSWORD psql -v brand="'${brand}'" -h "$PGHOST" -U "$PGUSER" -d "$PGDATABASE" -F ',' -A --pset footer -f alembic/examples/get_all_by_brand.sql >"data/${brand}.csv"
done

A simple script uses the psql command to run SQL query from a .sql file with CSV table output mode (-A flag)
The output of this command is saved in a CSV file within the data directory of the Git repository
The script gets all the necessary database settings from the environment variables
In our GitHub Actions, we’re going to set these environment variables from our repository secrets (note: we’ll have to add these environment variables into our repository ourselves)

Here’s the permalink to the Bash script.

GitHub Actions Workflow

Why GitHub Actions?

GitHub Actions workflow supports running on a Cron schedule. Essentially, what this means is that we can schedule our job (i.e. our script in this case) to run as short as 5 minutes intervals.

In our use case, we can use this to export our PostgreSQL query to our Google Sheets daily.

Let’s start by creating a generate_csv.yml workflow file inside the .github/workflows folder of our project directory:

name: Generate CSV
on:
    workflow_dispatch:
    schedule:
        - cron: "30 23 * * *" # At 23:30 UTC daily

workflow_dispatch is added so that we can manually trigger our workflow from GitHub API, CLI, or browser UI
Check out Crontab Guru for Cron schedule syntax

Next, to connect to any database, we’ll need to pass the connection settings:

name: Generate CSV
on:
    workflow_dispatch:
    schedule:
        - cron: "30 23 * * *" # <https://crontab.guru/#30_23_*_*_*>
env:
    PGDATABASE: ${{ secrets.PGDATABASE }}
    PGHOST: ${{ secrets.PGHOST }}
    PGPASSWORD: ${{ secrets.PGPASSWORD }}
    PGPORT: ${{ secrets.PGPORT }}
    PGUSER: ${{ secrets.PGUSER }}

Under the same GitHub project repository Secrets setting, enter the respective environment variables under Actions:

Finally, let's create the job for us to export our PostgreSQL queries and commit them to our repository

name: Generate CSV
on:
    workflow_dispatch:
    schedule:
        - cron: "30 23 * * *" # <https://crontab.guru/#30_23_*_*_*>
env:
    PGDATABASE: ${{ secrets.PGDATABASE }}
    PGHOST: ${{ secrets.PGHOST }}
    PGPASSWORD: ${{ secrets.PGPASSWORD }}
    PGPORT: ${{ secrets.PGPORT }}
    PGUSER: ${{ secrets.PGUSER }}
jobs:
    generate:
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v3
            - name: Install PostgreSQL # Step 1
              run: |
                  sudo apt-get update
                  sudo apt-get install --yes postgresql
            - name: Generate CSV # Step 2
              run: scripts/generate_csv.sh
              shell: bash
            - name: Get current date # Step 3
              id: date
              run: echo "::set-output name=date::$(TZ=Asia/Singapore date +'%d %b %Y')"
            - name: Commit changes # Step 4
              if: ${{ success() }}
              uses: EndBug/add-and-commit@v9
              with:
                  pull: "--rebase --autostash ."
                  message: "chore(data): update generated csv automatically for ${{ steps.date.outputs.date }} data"
                  default_author: github_actions

A job in a GitHub Actions workflow can contain many steps. Different steps in GitHub Actions run in different containers as well.

The first step is to set up and install PostgreSQL (with psql) onto the step container
Next, we’ll add a step to run our Bash script that runs an SQL query from a file
Optional: Get the local date-time so that we can use it as part of our commit message in Step 4
Commit the generated CSV file from Step 2 into our repository. Here, I am using the Add & Commit GitHub Actions to commit my CSV file changes.

Why use the --autostash flag with the git pull? This allows us to automatically stash and pop the pending CSV file changes before committing and pushing them to the repository. This helps us to work around Git commit issues whereby other developers could be pushing new code changes while this job runs.

That's it! we now have a Cron job that runs every day to update our CSV file for us so that our Google Sheets can import them.

Closing Throughs

Having GitHub — a highly available service to host our CSV file for us feels great. What’s more, having GitHub hosting this for free almost feels like some sort of a cheat.

I have also used this similar approach to run a scraper job to fetch Michelin Guide restaurants from the Michelin Guide website.

Alternatively, I have also considered using Google Sheets API to sync my data directly to Google Sheets. Given the integration efforts required, I’m glad that I stick to this very simple method.

Thanks for reading, and here's the final link to the example project.

This article was originally published on jerrynsh.com

How I Setup CI/CD Pipeline For Cloudflare Worker

Jerry Ng — Wed, 01 Jun 2022 00:09:26 +0000

Today, most of my projects on GitHub are kept up to date with Renovate. With auto-merging enabled, I want to have enough confidence that the automated dependencies update would not cause any regressions.

Testing Cloudflare Worker is a bit out of whack. Don’t get me wrong, I love Cloudflare Worker. However, the existing solution out there feels a little bit unintuitive. On top of that, dollarshaveclub/cloudworker is no longer actively maintained, which is a bummer.

TL;DR: This post talks about how I set up a CI/CD pipeline for a Cloudflare Worker project using GitHub Action and Grafana k6.

Overview

Previously, I built a URL shortener clone with Cloudflare Worker. Using this existing project (GitHub link), we shall look into setting up a CI/CD pipeline for it along with simple integration tests.

Our GitHub Action CI/CD pipeline is rather straightforward. The stages (jobs) are as follows:

Example CI workflow on GitHub

Lint check or unit testing
Deploy to the Staging environment
Run the integration tests on our Staging environment
Run semantic release and deploy to the Production environment

Wrangler Config

To start, we’ll have to modify our existing wrangler.toml file. Remember, we need a Staging environment to deploy to run our integration test against it:

[env.staging]
name = "atomic-url-staging"
workers_dev = true
kv_namespaces = [
  { binding = "URL_DB", id = "ca7936b380a840908c035a88d1e76584" },
]

Here are the things to take note of:

name — Make sure that the name has to be unique and alphanumerical (- are allowed) for each environment. Let’s name our Staging environment atomic-url-staging.
worker_dev — Our integration tests run on <NAME>.<SUBDOMAIN>.workers.dev endpoint. Hence, we need to deploy our Cloudflare Worker by setting workers_dev = true (reference).
kv_namespaces — Atomic URL uses KV as its database to store shortened URLs. Here, I chose to use a preview namespace KV as the test database. Why? Simply because it is the same development KV namespace that I use during local development (when running wrangler dev). Of course, you could just use a regular KV namespace. Just make sure that you’re not using Production KV id. Read how to create a KV namespace.

Next, we’ll also need to create a Production environment to deploy to with our production KV namespace:

[env.production]
name = "atomic-url"
route = "s.jerrynsh.com/*"
workers_dev = false
kv_namespaces = [
  { binding = "URL_DB", id = "7da8f192d2c1443a8b2ca76b22a8069f" },
]

The Production environment section would be similar to before except — we’ll be setting worker_dev = false and route for production.

Deployment

To deploy manually from your local machine to their respective environment, run:

wrangler publish -e staging
wrangler publish -e production

Though, we’ll look into how to do this automatically via our CI/CD pipeline using GitHub Actions.

Before we move on, you may find the complete wrangler.toml configuration here.

Oh, here’s a cheat sheet to configure wrangler.toml. I highly recommend you make use of this!

Grafana k6

For integration testing, we’ll be using a tool known as k6. Generally, k6 is used as a tool for performance and load testing. Now, bear with me, we’re not going to integrate any load tests into our CI/CD pipeline; not today.

Here, we’ll be running smoke tests for this project whenever new commits are pushed to our main branch. A smoke test is essentially a type of integration test that performs a sanity check on a system.

In this case, running a smoke test is sufficient enough for me to determine that our system is deployed without any regression and can run on minimal load.

What to test

Basically, here are a couple of things that we want to check as part of our smoke test for our URL shortener app in groups:

The main page should load as expected with a 200 response status

group("visit main page", function () {
    const res = http.get(BASE_URL);

    check(res, {
        "is status 200": (r) => r.status === 200,
        "verify homepage text": (r) =>
            r.body.includes(
                "A URL shortener POC built using Cloudflare Worker",
            ),
    });
});

Our main POST API endpoint /api/url should create a short URL with the original URL

group("visit rest endpoint", function () {
    const res = http.post(
        `${BASE_URL}/api/url`,
        JSON.stringify({ originalUrl: DUMMY_ORIGINAL_URL }),
        { headers: { "Content-Type": "application/json" } },
    );

    check(res, {
        "is status 200": (r) => r.status === 200,
        "verify createShortUrl": (r) => {
            const { urlKey, shortUrl, originalUrl } = JSON.parse(r.body);
            shortenLink = shortUrl;
            return urlKey.length === 8 && originalUrl === DUMMY_ORIGINAL_URL;
        },
    });
});

Make sure that when we visit the generated short URL, it redirects us to the original URL

group("visit shortUrl", function () {
    const res = http.get(shortenLink);

    check(res, {
        "is status 200": (r) => r.status === 200,
        "verify original url": (r) => r.url === DUMMY_ORIGINAL_URL,
    });
});

Finally, we configure out BASE_URL to point to our newly created Staging environment in the previous section.

To test locally, simply run k6 path/to/test.js. That’s all! You may find the full test script here.

In case you are thinking about running load tests, read how to determine concurrent users in your load test.

GitHub Actions

I’ll be glossing over this section as it is pretty straightforward. You may refer to the final GitHub Actions workflow file here.

Let’s piece together everything we have. Below are the GitHub Actions that we’ll need to use:

wrangler-action — For deploying Cloudflare Worker using Wrangler CLI
k6-action — For running k6

One thing to note about the workflow file — to make a job depend (need) on another job, we’ll make use of the need syntax.

Actions Secrets

This project requires 2 Actions secrets:

CF_API_TOKEN — To be used by Wrangler GitHub Action to automatically publish our Cloudflare Worker to its respective environment. You can create your API token using the Edit Cloudflare Workers template.
NPM_TOKEN — This project also uses semantic-release to automatically publish to NPM. To enable this, you will need to create an NPM_TOKEN via npm create token.

To add it to your GitHub repository secrets, check out this guide.

If you had taken a look at the final workflow file, you may have noticed the syntax ${{ secrets.GITHUB_TOKEN }} and wondered why I didn’t mention anything about adding GITHUB_TOKEN to our project Actions secrets. Turns out, it is automatically created and added to all of your workflows.

Closing Remark

Understandably, serverless platforms are generally known to be hard to test and debug. However, that doesn’t mean that we should ignore it.

So, what’s next? Right on top of my head, we could do better. Here are a couple of improvements that we can make:

Add a job/stage that automatically rolls back and reverts commit when the smoke tests fail
Create an individual testing environment upon PR creation so that we can run smoke tests on them
Probably overkill for this project — implementing canary deployment sounds like a good challenge

References

If you’re looking into unit testing Cloudflare Workers, here are my recommendations:

Here’s a decent video about setting up an ideal yet practical CI/CD pipeline:

https://www.youtube.com/watch?v=OPwU3UWCxhw

This article was originally published on jerrynsh.com

Saying Goodbye to Heroku Postgres for Now

Jerry Ng — Thu, 28 Apr 2022 14:39:11 +0000

A little less than a year ago, I built Burplist, a free search engine for craft beer in Singapore. With the goal to keep my infrastructure cost as low as possible, I started off with Heroku Postgres free tier.

With a row limit of 10,000 and a storage capacity of 1 GB, I thought that it would last me for at least a year — it didn’t.

Despite having a garbage collector service that runs on a weekly basis to remove staled rows, Heroku Postgres’s free tier simply wasn’t enough. I started looking for alternatives.

TL;DR: I’ve migrated my side projects’ PostgreSQL to Railway because of its pricing and ease of migration.

Heroku Postgres Alternatives

After some Googling, I came across a couple of PaaS similar to Heroku, each with its own Postgres offerings:

One stood out

I chose Railway because it makes the most economical sense for my use case. Below are some of the things that I like about Railway:

Incredibly generous pricing, where you would only start paying for resources usage after $10. In my case, this is a lot cheaper than using Heroku.
They have a Discord community where you can easily get help from.
Rather slick and intuitive UI. On top of that, you can view and make SQL queries directly via the dashboard. Though, I would argue that using a database tool like TablePlus or pgAdmin would be much more convenient.

You can only view and make SQL queries on the default database name railway
The dashboard also provides CPU, memory, and network metrics on your database usage; which is something that is unavailable on Heroku’s free tier.

Railway PostgreSQL dashboard metrics

How to Migrate

The migration was a piece of cake. This is easily one of the push factors that made the decision of migrating to Railway Postgres.

Pre-requisite

Installed postgresql locally on your machine. For e.g. if you’re on Mac — brew install postgresql
Make sure you can run the [pg_restore](http://www.postgresql.org/docs/current/static/app-pgrestore.html) command

Setup Railway

Setup a Railway account (referral link)
Go to your Railway dashboard and create a new project and provision PostgreSQL > Take note of your database credentials under the "Connect" or "Variables" tab. You'll need them later

5 Migration steps

Export Heroku Postgres (reference).

heroku pg:backups:capture -a <heroku_app_name>
heroku pg:backups:download -a <heroku_app_name>

At your current working directory, you should see latest.dump file

$ heroku pg:backups:capture -a <heroku_app_name>

Starting backup of postgresql-encircled-90125... done
Use Ctrl-C at any time to stop monitoring progress; the backup will continue running.
Use heroku pg:backups:info to check progress.
Stop a running backup with heroku pg:backups:cancel.
Backing up DATABASE to b001... done

$ heroku pg:backups:download -a <heroku_app_name>
Getting backup from ⬢ <heroku_app_name>... done, #1
Downloading latest.dump... ████████████████████████▏  100% 00:00 309.99KB

$ ls
latest.dump

In the same working directory with latest.dump, run the following command to import your downloaded database dump to your Railway Postgres. Update accordingly with your own Postgres credentials:

# NOTE:
# Keep PGDATABASE as `railway` if you want to view your tables via the Railway dashboard.
# If you insist on another database name, you will have to create your own database via psql command.
#
# E.g.:
# PGPASSWORD=$PGPASSWORD psql -h $PGHOST -U $PGUSER -p $PGPORT -d $PGDATABASE
# CREATE DATABSE your_database_name

PGPASSWORD=$PGPASSWORD pg_restore -h $PGHOST -U $PGUSER -p $PGPORT -d $PGDATABASE < latest.dump

Verify that your data is imported correctly. To do so, you can use the psql command as shown in the example in Step 4; or connect to your own database via a tool like pgAdmin.
Go to your existing apps, e.g. your Heroku app, and update the database connection URL/credentials accordingly.

That’s it!

I Miss Heroku Dataclip

One thing that I really miss about Heroku Postgres is its ability to share query results with Dataclips. Heroku Dataclips is incredibly useful as you can easily:

Make a SQL query via the UI
Share the output in JSON or CSV format via a link

Once you have a sharable link to CSV, you easily import it to other SaaS such as Google Sheets. On top of that, most data processing SaaS supports CSV out of the box; this makes Dataclip incredibly useful.

Now, I’ll either have to generate my own CSV on another server or pray that the SaaS that I’m using has PostgreSQL integration (hint: most don’t).

Having this said, I am still more than happy to make this tradeoff.

Other Uses

Besides using Railway as my PostgreSQL database server for my projects, I am also using it to host my own umami analytics site in combination with Vercel.

I’d also recommend you to check out their other offerings such as Redis, MongoDB, and also their starter project templates.

Closing Thoughts

No, I’m not leaving Heroku entirely. Heroku has stood the test of time. I would expect a company owned by Salesforce to provide better reliability and stability.

On top of that Heroku’s add-ons often come in handy. Add-ons provide useful integration out of the box e.g. logging, search, and many more!

On the contrary, I can’t speak for the reliability and uptime of the Railway. Having that said, the projects that I am running allow me to take on this level of risk, and my experience so far has been great.

I’m rooting for Railway. I simply enjoy seeing competitions in the market.

Thanks for reading!

This article was originally published on jerrynsh.com

12 Tips to Self-host Renovate Bot

Jerry Ng — Mon, 04 Apr 2022 00:38:03 +0000

Updating dependencies is boring. Despite its importance, we always find excuses to avoid updating them in the phrase of “if it ain't broke, don't fix it” or “there are more important features to work on”. Over time, the maintainability of the projects deteriorates. The team ends up with a 3-year-old dependency where no one is brave enough to bump it.

So, what’s the solution? Let the robots do our boring job!

In this article, you will learn tips on running a self-hosted Renovate bot with GitLab as an example. If you’re looking for a guide on how to start using Renovate on GitHub, I’d highly recommend you to read this.

TL;DR

Why use Renovate
Getting started on self-hosted Renovate (GitLab as an example)
How to run Renovate locally
Debugging Renovate jobs
12 useful Renovate tips

Disclaimer: these tips are my own opinions from lessons gathered from hours working. Enjoy!

Why Use Renovate

In short, Renovate (official doc) helps to update project dependencies (private or third-party) automatically.

How? Renovate parses our projects’ dependency management files such as package.json, pyproject.toml, go.mod file and raises a pull/merge request (PR/MR) with the updated dependencies accordingly.

Renovate is highly customizable via a simple configuration file (config.js). With rather intuitive configuration settings, it also supports a wide range of package managers.

Today, Renovate supports many other Git platforms such as Bitbucket, Gitea, or even Azure DevOps. On top of that, it’s used by a lot of popular development communities or companies such as Uber, GitLab, Prisma, Netlify, etc.

Highlight

The best part about Renovate is its ability to auto-merge PRs/MRs.

That aside, the flexibility provided by the [packageRules feature](https://docs.renovatebot.com/configuration-options/#packagerules) which is used to apply rules to specific dependencies (in individual or group) is incredibly handy.

How To Run Self-hosted Renovate on GitLab

To run Renovate on self-hosted GitLab, you’ll need a private GitLab project (i.e. repository in the following). This bot repository will be used to host GitLab runners which run Renovate for your other projects.

Next, assuming that you already have GitLab runners installed and set up, you’ll need the following in the bot project:

Configure the following CI/CD variables:
1. RENOVATE_TOKEN — GitLab Personal Access Token (PAT) with scopes: read_user, api, and write_repository.
2. GITHUB_COM_TOKEN — GitHub PAT with minimum scope.

config.js (NOTE: here’s a more complex example).

// Example config.js from https://github.com/renovatebot/renovate/blob/main/docs/usage/examples/self-hosting.md

module.exports = {
  endpoint: 'https://self-hosted.gitlab/api/v4/',
  token: '**gitlab_token**',
  platform: 'gitlab',
  onboardingConfig: {
    extends: ['config:base'],
  },
  repositories: ['username/repo', 'orgname/repo'],
};

.gitlab-ci.yml. If you need a more complex example, check out this example from the GitLab team. Otherwise, here’s a minimal example; do update accordingly:

image: renovate/renovate:32.6.12

variables:
    LOG_LEVEL: debug

renovate:on-schedule:
    tags:
        - your-gitlab-runner-tag-if-any
    only:
        - schedules
    script:
        - renovate $RENOVATE_EXTRA_FLAGS

renovate:
    tags:
        - your-gitlab-runner-tag-if-any
    except:
        - schedules
    script:
        - renovate $RENOVATE_EXTRA_FLAGS

Lastly, you’ll need to run Renovate at regular intervals (e.g. every hour) using the GitLab project’s CI/CD Schedules feature. Do note that this is different from Renovate’s own schedule.

Check the source for more details. Though, the steps above should be sufficient.

How To Run Renovate Locally

While I was working with a large repository (>6GB for full clone), each Renovate job may take hours to complete. Having the ability to run Renovate locally saves me a bunch of time when it comes to experimentation and debugging.

First, create a minimum reproducible example (MRE) repository. Then, update your config.js to target or discover the MRE repository. To run Renovate locally:

It’s a lot easier to just use Docker. So, make sure you’ve installed Docker.

To start, you’ll need to ensure that these environment variables are being exported in your shell:

export RENOVATE_TOKEN="aa11bb22cc" # GitLab Personal Access token (PAT)
export GITHUB_COM_TOKEN="cc33dd44ee" # GitHub PAT

Next, update your config.js accordingly. You’ll need to update your target repositories accordingly.
Finally, you can run Renovate using the following. To grab the latest Renovate version, check out Docker hub. Do change the following command accordingly:
```
docker run \
    --rm \
    -e LOG_LEVEL="debug" \
    -e GITHUB_COM_TOKEN="$GITHUB_COM_TOKEN" \
    -v "/path/to/local/config.js:/usr/src/app/config.js" \
    renovate/renovate:"32.6.12" \
    --token "$RENOVATE_TOKEN" \
    --dry-run="true"
```
With this, testing Renovate configuration in a fast-feedback loop manner is now possible. In case you need more comprehensive logs, try setting LOG_LEVEL=”trace” instead.

To perform an actual run, update --dry-run="false".

12 Useful Renovate Bot Tips

General tips

Not sure how to get the most out of Renovate quickly? Check out this Renovate bot cheat sheet instead of the verboseness of the official Renovate doc.
If you are unsure whether Renovate supports a certain functionality, always check out their FAQ page first. Chances are it’s already there.

To disable updates for a specific package or library simply set enabled: false under the respective [packageRule](https://docs.renovatebot.com/configuration-options/#packagerules). Example:

// Example config.js

module.exports = {
    endpoint: 'https://your.domain.com/api/v4/',
    token: '**gitlab_token**',
    platform: 'gitlab',
    onboardingConfig: {
        extends: ['config:base'],
    },
    repositories: [
        {
            repository: 'john/next-generation-repo',
            packageRules: [
                {
                    description: "'Disable MAJOR update types',"
                    matchUpdateTypes: ['major'],
                    enabled: false,
                },
            ],
        },
    ],
}

Need to run some custom task or script after upgrading (e.g. a script that posts messages to Slack)? Try postUpgradeTasks.
Keep project-specific Renovate configs on the bot repository instead of having renovate.json in every other repository. For this, set [onboarding: false](https://docs.renovatebot.com/self-hosted-configuration/#onboarding) under module.exports. This allows for Renovate-related configs to be abstracted away to a single repository only.

Tips on debugging Renovate jobs

When it comes to self-hosted solutions, there’s no running away from debugging your own jobs. There could be scenarios where you’ll need to test connections to your private registry, proxy, etc.

Here are a couple of helpful tips:

Try running Renovate locally to have a faster feedback loop instead of relying on your CI/CD pipelines. Plus, running locally is practically free!
Use trace LOG_LEVEL in case debug doesn’t give you enough information.
In case you’re working on some bespoke use cases or facing odd encounters/bugs, try Google searching with the prefix “site:github.com/renovatebot/renovate ”. (example). In most cases, you’ll find that there are already others who have filed a similar discussion or issue.

Dealing with a large repository (e.g. GB in size)

If you attempt to run Renovate on a large repository, you may encounter a SIGTERM signal (which can be seen in your Renovate job log) due to timeout. To cope with this, increase the [executionTimeout setting](https://docs.renovatebot.com/self-hosted-configuration/#executiontimeout) in your config.js.

// Example config.js

module.exports = {
  endpoint: 'https://your.domain.com/api/v4/',
    executionTimeout: 1440, // minutes
  token: '**gitlab_token**',
  platform: 'gitlab',
  onboardingConfig: {
    extends: ['config:base'],
  },
  repositories: ['john/large-repo', 'doe/huge-repo'],
};

Under your GitLab project’s CI/CD General pipelines settings, you may want to increase your job timeout (e.g. 10 hours) as Renovate may take a long time on an uncached run on large repositories.

You may want to set persistRepoData: true for faster git fetch between runs.

// Example config.js

module.exports = {
    endpoint: 'https://your.domain.com/api/v4/',
    token: '**gitlab_token**',
    platform: 'gitlab',
    persistRepoData: true,
    onboardingConfig: {
        extends: ['config:base'],
    },
    repositories: ['john/repo'],
}

On some occasions, you may run into ERROR: Disk space error - skipping. Here, you may want to provision a runner with increased disk size. E.g. if you are using AWS EC2, try to increase the size of the volume.

Conclusion

Over the past few months, I have been using Renovate bot extensively; on GitHub, and self-hosted GitLab. Today, I am using Renovate on most of my active projects on GitHub to automate dependency updates. So far, Renovate was able to fit 99% of my use cases.

Keeping our project dependencies up-to-date is often overlooked. Yet, overlooking such work often comes at a great cost. The fact that we can leverage Renovate’s ability to do such grunt work is quite a blessing.

I hope this article saves you enough time instead of having you groom through pages of GitHub discussions and issues. Thanks for reading!

This article was originally published on jerrynsh.com