DEV Community: Mahadevan Sreenivasan

Building a fast URL Shortener with Go and Redis

Mahadevan Sreenivasan — Tue, 18 Oct 2022 14:36:55 +0000

Why build a URL shortener now?

URLs shorteners have been around for a while now and one could just pick any of the hundreds of URL Shorteners available on the web and start using it.

However building our own URL shortener has its own benefits -

Most URL shorteners require you to pay for creating custom URL shorts (or slugs)
Many mature URL shorteners will allow you to hold your redirects only for a short period of time.
URL shorteners generally make money by selling URL redirects and customers' IP addresses to third parties. If you are concerned about data privacy, you would build your own URL shorteners.

Functional Requirements

We should be able to generate unique short-URLs for every valid HTTPS URL
We should be able to resolve short-URLs quickly and redirect users to actual URLs
Short-URLs should have a lifetime / expiry.
We should be able to specifiy an optional custom short-URL. If such a custom slug is provided, we will try to map the original URL to the custom short, else we will generate a new one.

Non Functional Requirements

The system will be read heavy i.e the ratio of redirects will be much higher than generating short-urls. Let's assume this ratio to be 100 : 1
Resolving short-urls and redirecting users to the original URL should happen instantaneously with minimal delay.

Traffic Estimates

Let's assume that users generate 100K short-urls every month. That means we can expect 100K * 100 redirects -> 10M redirects every month.
10M redirects would translate to 10M / (30d * 24h * 3600s) = ~3 redirects a second.

Storage Estimates

To compute storage, let's assume that we use a key-value store with the key being the short-url and the value as the original URL and the entire record to have an expiry.

custom-short: ~10 characters
url: ~1000 characters
ttl: int32

With this, we can assume that a record would be ~1KB. With 100K unique short-urls every month, we can assume our memory capacity at 100K * 1KB = ~100 MB. For an entire year, we can assume ~1.2GB / year

Generating the short-URL or Slug

If the user has already provided a custom-short, we will try to set that as the short-URL. Otherwise, we will have to generate a unique short-URL.

Remember that we are generating 100K short-urls per month or 1.2M short-urls a year or ~15M short-urls in 12 years :)

How do we generate unique URLs with minimal collision for 15M urls over time?

Base62 Encoding

Our custom-short URLs have to be reasonably short 1 - 10 characters. To generate such URLS, initially we can generate a random int64 number within a specific range (0, 15M). The idea is to convert this number from Base10 to Base62.

But before that, lets try to get some intuition -

Assume that the random int64 that we generate is 234556 .

The equivalent binary or Base2 -> 001111000010100111.

The equivalent Hex or Base16 -> C3493

The equivalent Base64 encoded string would be -> 8q5

The Base64 alphabet consists of the following characters

const (
    alphabet      = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789/="
)

However, we don't want URLs to contain special characters. So we remove the slash and the equals sign and create a Base62 alphabet and encode our int64 to a Base62 string.

const (
    alphabet      = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ012345678"
)

Remember, our assumption is that we would store at-most 15M shorts. So if we generate 15,000,000 as a random integer for our short, the equivalent Base62 encoded string would be -> El6ab

Thats only 5 characters! What about 100,000,000 (100M)? The equivalent Base62 would be -> oJKVg

You see, our short-urls / slugs are < 5 characters, which is exactly what we want.

Here is a version of a int64 -> Base62 algorithm written in go and the algorithm is pretty straightforward.

const (
    alphabet      = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
)

func Base62Encode(number uint64) string {
    length := len(alphabet)
    var encodedBuilder strings.Builder
    encodedBuilder.Grow(10)
    for ; number > 0; number = number / uint64(length) {
        encodedBuilder.WriteByte(alphabet[(number % uint64(length))])
    }

    return encodedBuilder.String()
}

System Design

Since the title of the post has Golang and Redis, we are obviously going to use both of them to build our URL shortener service.

)

Redis is a fast key-value store and for our purposes, it seems like an ideal solution. For the webserver in go, I used go-fiber inspried by Akhil Sharma's YT channel. If you are from a Node.JS / Express background, you should feel at home with go-fiber.

Our primary request and response can be defined like so -

Data Models

type request struct {
    URL         string        `json:"url"`
    CustomShort string        `json:"short"`
    Expiry      time.Duration `json:"expiry"`
}

type response struct {
    URL             string        `json:"url"`
    CustomShort     string        `json:"short"`
    Expiry          time.Duration `json:"expiry"`
    XRateRemaining  int           `json:"rate_limit"`
    XRateLimitReset time.Duration `json:"rate_limit_reset"`
}

Connecting to Redis

We will connect with Redis with a helper like so -

package database

import (
    "context"
    "os"

    "github.com/go-redis/redis/v8"
)

var Ctx = context.Background()

func CreateClient(dbNo int) *redis.Client {
    rdb := redis.NewClient(&redis.Options{
        Addr:     os.Getenv("DB_ADDR"),
        Password: os.Getenv("DB_PASS"),
        DB:       dbNo,
    })

    return rdb
}

The shorten endpoint

The Shorten endpoint takes in the following payload -

{
  "url": "https://your-really-long-url/"
  "custom-short": "optional",
  "expiry": 24
}

The shorten URL is mapped to a shorten.go file. We initially validate the request by performing the following checks

Rate Limiting

    r2 := database.CreateClient(1)
    defer r2.Close()

    val, err := r2.Get(database.Ctx, ctx.IP()).Result()
    limit, _ := r2.TTL(database.Ctx, ctx.IP()).Result()

    if err == redis.Nil {
    // Set quota for the current IP Address
        _ = r2.Set(database.Ctx, ctx.IP(), os.Getenv("API_QUOTA"), 30*60*time.Second).Err()
    } else if err == nil {
        valInt, _ := strconv.Atoi(val)
    // If Quota has been exhausted
        if valInt <= 0 {
            return ctx.Status(fiber.StatusServiceUnavailable).JSON(fiber.Map{
                "error":            "Rate limit exceeded",
                "rate_limit_reset": limit / time.Nanosecond / time.Minute,
            })
        }
    }

We use Redis-0 for storing URL Shorts and Redis-1 for managing IP Addresses & Rate limiting. Our Assumption is that the same IP Address will not be allowed to hit the /shorten end-point more than 10 times (default value in .env)

Generating the Base-62 short or store the custom-short

We simply check the db for a collision before storing the generated / custom-short

if body.CustomShort == "" {
        id = helpers.Base62Encode(rand.Uint64())
    } else {
        id = body.CustomShort
    }

    r := database.CreateClient(0)
    defer r.Close()

    val, _ = r.Get(database.Ctx, id).Result()

    if val != "" {
        return ctx.Status(fiber.StatusForbidden).JSON(fiber.Map{
            "error": "URL Custom short is already in use",
        })
    }

    if body.Expiry == 0 {
        body.Expiry = 24
    }

    err = r.Set(database.Ctx, id, body.URL, body.Expiry*3600*time.Second).Err()

After generating the custom-short, we simply return the response using the struct defined above.

The Resolve endpoint

URL resolution is pretty straightforward. We simply check for the path param against our keys in redis and redirect to the original value (URL) using a HTTP 301 Redirect

func Resolve(ctx *fiber.Ctx) error {
    url := ctx.Params("url")

    r := database.CreateClient(0)
    defer r.Close()

    value, err := r.Get(database.Ctx, url).Result()
    if err == redis.Nil {
        return ctx.Status(fiber.StatusNotFound).JSON(fiber.Map{"error": "short-url not found in db"})
    } else if err != nil {
        return ctx.Status(fiber.StatusInternalServerError).JSON(fiber.Map{"error": "Internal error"})
    }

    return ctx.Redirect(value, 301)
}

And that's it. We now have a fast working URL Shortener up and running. Check out the complete source code here. Do let me know your thoughts and feeback :)

Mock REST APIs with Mockatron

Mahadevan Sreenivasan — Sun, 20 Mar 2022 05:42:37 +0000

Why Mock APIs?

As a mobile / front-end developer, have you ever been blocked due to a dependency on a back-end API that is not readily available for consumption? This scenario is very frequent in fast paced development teams where multiple front-end and back-end teams work in parallel whilst striving to resolve dependencies with one another.

What is Mockatron

I am building Mockatron - a simple tool that helps front-end / mobile developers setup a quick Mock HTTP API Server without having to wait for actual back-end APIs to be available.

The goal is to help parallelize development efforts of teams so that once the actual APIs are available, consumers can simply switch to the original implementation without having to make a whole lot of changes.

By defining a simple set of configuration rules, you can stand up a portable mock HTTP server that can return **static **as well as **dynamic **response data. Some of the features that are offered today include:

Handlebars style templates to support dynamic generation of responses from object templates.
Proxy support to mock specific routes and redirect others to your actual back-end server.
Support for static responses (e.g plain old JSON data).

Example

Let us assume that we are going to build a mobile / web application like Amazon that will list products from a back-end API. Our goal is to use Mockatron and setup a mock API server that will return mock products each time a call to /api/products is called from the front-end / mobile app.

Install Mockatron

Pre-requisities:

Make sure that you have Node and npm installed.

$ npm install -g mockatron

Setup your mock definitions

Create an empty folder for us to configure our mock server. Give it an arbitrary name (e.g products).

Inside the products folder, create a configuration folder named config. We will be configuring our mock end points and constraints (more on this later) in this folder.

Inside the config folder, create a main.json file. Here is my folder structure that I have created

products
└── config
    ├── main.json

Let us start by writing a simple config in our main.json file.

{
     "contextPath": "/api", 
     "routes": [ 
         { 
             "path": "/products", 
             "method": "GET", 
             "responses": [ 
                 { 
                     "body": "'no results found'", 
                     "statusCode": 200 
                 } 
             ] 
         } 
     ] 
}

The above configuration should be straightforward. The contextPath key specifies where the defined routes will be mounted while running our mock server. For e.g if we are hosting our mock server at localhost:8080, http://localhost:8080/api/products would match the /products route that we have defined above.

The responses array specifies the different response bodies that this route can potentially return. In this case, we are going to return a simple string body with a status code of 200.

Generate a mock server

Now that we have setup our configuration, go ahead and run the below command inside the products directory.

$ mockatron --config config --out output

If everything went well without errors, an output directory will be created in the specified path.

Run the mock server

cd into the output path
$ npm i - This will install all dependencies for the mock server
$ npm start - This will run a mock server on port 8080

Ideally, you should see the following messages in stdout

> mockatron_server@1.0.0 start
> node index.js

Started application on port 8080

Open a tab in your browser or send a curl request to http://localhost:8080/api/products and you should get no results found as the response with a status code of 200.

And that's it! We have a mock API server up and running in less than 10 lines of code.

Dynamic responses

The above config returns the same static text every time we hit /api/products. However, most of the time while building a front-end or mobile app, we would need to generate variability in our responses to simulate a real world scenario. To achieve this go back to the configuration folder and create another file named products.json

products
└── config
    ├── main.json
    ├── products.json

Inside the products.json file, we will try to create a template for our response.

{
    "productList": [
            {{#array 5 20}}
            {
                "price": "{{float 50.0 5000.0}}",
                "rating": {{int 1 5}},
                "id": {{@index}},
                "sku": "{{uuid}}",
                "name": "{{word 2}}"
            }
            {{/array}}
    ]
}

We will use the productList object to return an array of products based on the template that we have defined above.

The #array helper generates a random number of objects (between 5 & 20) that it encompasses. The array is closed out with the {{/array}} tag.

Each object in the array can again be randomized. The float and int helpers take min and max arguments and generate a random number in between. Similarly there are helpers for generating a random uuid, random text using the word helper and so on. A complete list of response helpers can be found here.

To hook up the productList definition array to our /api/products route, head back to our main.json file and modify the existing config like so.

{
     "contextPath": "/api", 
     "routes": [ 
         { 
             "path": "/products", 
             "method": "GET", 
             "responses": [ 
                 { 
                     "body": "{{{def 'products' 'productList'}}}", 
                     "statusCode": 200 
                 } 
             ] 
         } 
     ] 
}

We have only changed the body to load the productList definition that we defined in the products.json file instead of static text. For this, we use the {{{def <json-file-name> <definition>.

Now, lets rebuild our mock server. Go back to the root folder (the products directory) and run the following command

$ mockatron --config config --out output

$ cd output

$ npm i && npm start

Now open a browser and try hitting http://localhost:8080/api/products multiple times. You will see that you get a different response each time!

Adding Constraints to Routes

Now that we are able to generate dynamic response data, let us look at adding constraints that determine when the response should be generated.

Constraints are synonymous to if (condition) then return response in Mockatron. Think of a constraint as logical blocks that we will put inside an if block.

Constraints can be added to each element of the responses key inside a route.

As an example, assume that the products API that we are building mandates a search query parameter in the URL and we should return a response only if the search query parameter is not null.

Let's head back to our main.json file and add the above rule as a constraint.

{
  "contextPath": "/api",
  "routes": [
    {
      "path": "/products",
      "method": "GET",
      "responses": [
        {
          "constraint": "{{neq (query 'search') undefined}}",
          "body": "{{{def 'products' 'productList'}}}",
          "statusCode": 200
        },
        {
          "body": "'No Results found'",
          "statusCode": 200
        }
      ]
    }
  ]
}

We see that the responses array now contains 2 elements. The first element contains a constraint key that evaluates if the incoming request's query parameter - search !== undefined. Only if this condition is satisfied, will the body get executed.

Otherwise, Mockatron will fall back to the next object in the responses array which basically returns a static string like before.

Remember: Contents in the responses array are evaluated in the order in which they are listed in the configuration

A full list of Mockatron constraint helpers are available here .

Nesting Constraints

We can nest multiple constraints into a single constraint. For e.g what if we want to return the response only if the search query param is not null and the price param > 0?

Modify the constraint section like so.

"constraint": "{{and 
                    (neq (query 'search') undefined)
                    (gt (query 'price') 0) 
               }}",

Static Response Data

Sometimes we may not want to deal with templating or needing dynamic response data for all our routes and a simple plain JSON object would be sufficient. To return static JSON content as response, use the {{file <json_file_name>}} helper instead of the {{def}} helper that we were using till now.

Let's assume that we add a new route called /static-product to our list of routes. In main.json, lets add the following configuration to the existing routes array.

{
  "path": "/static-product",
  "method": "GET",
  "responses": [
     {
       "statusCode": 200,
       "body": "{{file 'static-products'}}"
     }
   ]
}

All you need is a valid JSON response defined in a file named static-products.json in the config folder alongside the main.json file.

What's next

Try adding multiple routes, responses and constraints and see what works for you. If you want to report a bug / discuss a new feature request, reach out to me / raise an issue in Github.

In the next post, I will talk about configuring proxy support in Mockatron. This could be really powerful if you want to use Mockatron as a proxy server that will mock only specific routes while proxying other requests to your actual back-end APIs. Stay tuned!