DEV Community: Maurício Linhares

Building and distributing a command line tool in Golang

Maurício Linhares — Fri, 01 Apr 2022 17:42:51 +0000

Thanks to its static binaries and cross-compiling, building and distributing command-line apps in Go is a breeze. You can quickly build an app and produce binaries for multiple platforms without leaving your current environment, and we'll learn how to do just that here.

We'll build a simple command-line HTTP client, produce binaries for Windows, Mac, and Linux and publish them to Github. You can find the repo for this post here.

We'll try to build this client by separating the actual command line operation from the rest of the code as much as we can, so to kick it off, let's look at the code that performs the HTTP request:

type Config struct {
   Headers            http.Header
   UserAgent          string
   Data               string
   Method             string
   Insecure           bool
   Url                *url.URL
   ControlOutput      io.Writer
   ResponseBodyOutput io.Writer
}

The config type here encapsulates the options we offer for the command-line client in a way that is independent of who is making the calls. We allow it to set custom headers, user agent, request body data, HTTP method, if we allow requests to servers with certificates we don't trust, the URL called, and where we will print details of the process.

Next, the code that takes this config to execute the request:

package gurl

import (
   "bytes"
   "crypto/tls"
   "fmt"
   "github.com/rs/zerolog/log"
   "io"
   "net/http"
   "net/url"
   "strings"
)

type Config struct {
   Headers            http.Header
   UserAgent          string
   Data               string
   Method             string
   Insecure           bool
   Url                *url.URL
   ControlOutput      io.Writer
   ResponseBodyOutput io.Writer
}

func Execute(c *Config) error {
   var r io.Reader
   var tlsConfig *tls.Config

   if c.Data != "" {
      r = bytes.NewBufferString(c.Data)
   }

   if c.Insecure {
      tlsConfig = &tls.Config{
         InsecureSkipVerify: true,
      }
   }

   request, err := http.NewRequest(c.Method, c.Url.String(), r)
   if err != nil {
      return err
   }

   if c.UserAgent != "" {
      request.Header.Set("User-Agent", c.UserAgent)
   }

   for key, values := range c.Headers {
      for _, value := range values {
         request.Header.Add(key, value)
      }
   }

   client := http.Client{
      Transport: &http.Transport{
         TLSClientConfig: tlsConfig,
      },
      CheckRedirect: func(req *http.Request, via []*http.Request) error {
         return http.ErrUseLastResponse
      },
   }

   requestBuilder := &wrappedBuilder{
      prefix: ">",
   }

   requestBuilder.Printf("%v %v", request.Method, request.URL.String())
   requestBuilder.WriteHeaders(request.Header)
   requestBuilder.Println()

   if _, err := io.Copy(c.ControlOutput, strings.NewReader(requestBuilder.String())); err != nil {
      return err
   }

   response, err := client.Do(request)
   if err != nil {
      return err
   }

   defer func() {
      if err := response.Body.Close(); err != nil {
         log.Warn().Err(err).Str("url", c.Url.String()).Msg("failed to close response body")
      }
   }()

   responseBuilder := &wrappedBuilder{
      prefix: "<",
   }

   responseBuilder.Printf("%v %v", response.Proto, response.Status)
   responseBuilder.WriteHeaders(response.Header)
   responseBuilder.Printf("")
   responseBuilder.Println()

   if _, err := io.Copy(c.ControlOutput, strings.NewReader(responseBuilder.String())); err != nil {
      return err
   }

   _, err = io.Copy(c.ResponseBodyOutput, response.Body)
   return err
}

Here we get the config object and set up the HTTP request and client objects to perform it. The code outputs general information to ControlOutput on the config (like what curl does by adding < and > to the output to see the outgoing and incoming data). This function doesn't care how it gets called, you can call it from a command-line operation or unit tests, and it still works the same way.

We also have a small helper object, the wrappedBuilder, to avoid repeating the prints (and simplify error handling, as it sucks to handle errors on every write to ControlOutput):

type wrappedBuilder struct {
   prefix string
   strings.Builder
}

func (w *wrappedBuilder) WriteHeaders(headers http.Header) {
   for key, values := range headers {
      for _, value := range values {
         w.Printf("%v: %v", key, value)
      }
   }
}

func (w *wrappedBuilder) Println() {
   w.WriteString("\n")
}

func (w *wrappedBuilder) Printf(s string, a ...any) {
   w.WriteString(fmt.Sprintf("%v %v\n", w.prefix, fmt.Sprintf(s, a...)))
}

Building the command line bridge

We'll use cobra, one of the best command-line libraries available in Go, to parse the commands. In cobra everything is a cobra.Command, and you can have commands with subcommands (like git status, status here is a subcommand of git).

In our specific case, all we need is a single command, so here's how we'll build it:

func CreateCommand() *cobra.Command {
   config := &Config{
      Headers:            map[string][]string{},
      ResponseBodyOutput: os.Stdout,
      ControlOutput:      os.Stdout,
   }

   headers := make([]string, 0, 255)

   command := &cobra.Command{
      Use:     `gurl URL`,
      Short:   `gurl is an HTTP client`,
      Long:    `gurl is an HTTP client for a tutorial on how to build command line clients in go`,
      Args:    ArgsValidator(config),
      PreRunE: OptionsValidator(config, headers),
      RunE: func(cmd *cobra.Command, args []string) error {
         return Execute(config)
      },
   }

   command.PersistentFlags().StringSliceVarP(&headers, "headers", "H", nil, `custom headers headers to be sent with the request, headers are separated by "," as in "HeaderName: Header content,OtherHeader: Some other value"`)
   command.PersistentFlags().StringVarP(&config.UserAgent, "user-agent", "u", "gurl", "the user agent to be used for requests")
   command.PersistentFlags().StringVarP(&config.Data, "data", "d", "", "data to be sent as the request body")
   command.PersistentFlags().StringVarP(&config.Method, "method", "m", http.MethodGet, "HTTP method to be used for the request")
   command.PersistentFlags().BoolVarP(&config.Insecure, "insecure", "k", false, "allows insecure server connections over HTTPS")

   return command
}

We start by creating a Config struct with the basic options setup (and pointing to stdout) and then create a cobra.Command with the options and documentation we need. You could add more examples and better documentation to the program here. It will be up to you how much you want to add, but the more, the merrier.

With the command created, we can now add flags. They are special options to change its behavior. These all point back to fields at the Config struct, so you can add more fields to it and then add flags to control the value here. One doesn't point directly, and it's the headers. Let's look at why that is:

func OptionsValidator(c *Config, headers []string) func(cmd *cobra.Command, args []string) error {
   return func(cmd *cobra.Command, args []string) error {
      for _, h := range headers {
         if name, value, found := strings.Cut(h, ":"); found {
            c.Headers.Add(strings.TrimSpace(name), strings.TrimSpace(value))
         } else {
            return newErrorWithCode(3, "header is not a valid http header separated by `:`, value was: [%v]", h)
         }
      }

      return nil
   }
}

Unfortunately, cobra doesn't offer an easy way to parse map[string][]string from the command line (it does offer map[string]string, though). We use the PreRunE with the OptionsValidator function to parse the headers and add them to the Headers property at Config. If the header values are invalid, we return an error right away before calling the HTTP client code.

The other validation we have in place is for the arguments provided, the program requires a single argument to be called (as in gurl http://example.com/), so we verify there is exactly one parameter, and it is a valid URL before proceeding:

func ArgsValidator(c *Config) func(cmd *cobra.Command, args []string) error {
   return func(cmd *cobra.Command, args []string) error {
      if l := len(args); l != 1 {
         return newErrorWithCode(2, "you must provide a single URL to be called but you provided %v", l)
      }

      u, err := url.Parse(args[0])
      if err != nil {
         return errors.Wrapf(err, "the URL provided is invalid: %v", args[0])
      }

      c.Url = u

      return nil
   }
}

With commands and implementation done, here's the main function that starts the program (this file lives at cmd/gurl):

package main

import (
   "github.com/mauricio/gurl"
   "github.com/rs/zerolog"
   "os"
)

func main() {
   zerolog.TimeFieldFormat = zerolog.TimeFormatUnix

   if err := gurl.CreateCommand().Execute(); err != nil {
      switch e := err.(type) {
      case gurl.ReturnCodeError:
         os.Exit(e.Code())
      default:
         os.Exit(1)
      }
   }
}

It creates the command, runs it, checks the error, and exits. Here's the help it generates:

$   go run cmd/gurl/main.go -h
gurl is an HTTP client for a tutorial on how to build command line clients in go

Usage:
 gurl URL [flags]

Flags:
 -d, --data string     data to be sent as the request body
 -H, --headers strings   custom headers headers to be sent with the request, headers are separated by "," as in "HeaderName: Header content,OtherHeader: Some other value"
 -h, --help        help for gurl
 -k, --insecure      allows insecure server connections over HTTPS
 -m, --method string    HTTP method to be used for the request (default "GET")
 -u, --user-agent string  the user agent to be used for requests (default "gurl")

cobra formats and includes all the information we provided when setting up the command and flags. You don't even have to set up a help command. It just adds it automatically and prints the details.

Building for multiple platforms

Now that we have all the pieces together, it's time to produce binaries for multiple platforms so we can distribute them. Now it's easier to clone the repo to follow along, so you don't have to type everything.

If you don't have any native dependencies, you can cross-compile your go app to any supported platform. Here's what it looks like (all commands assume you have cloned the repo and are at the repo's root folder):

# building the program for intel macs
GOOS=darwin GOARCH=amd64 go build -o gurl-mac-amd64 cmd/gurl/main.go 
# building the program for M1 macs
GOOS=darwin GOARCH=arm64 go build -o gurl-mac-arm64 cmd/gurl/main.go 
# building the program for 64 bits amd/intel linux
GOOS=linux GOARCH=amd64 go build -o gurl-linux-amd64 cmd/gurl/main.go

You can set the GOOS environment variable to multiple operating systems (like Linux, windows, darwin, and others) and the GOARCH to multiple architectures (like amd64, arm64, and others) and go will produce a binary for that specific version that you can use. So if you're working on a Linux box, you can make binaries for Macs and Windows without having a Mac or Windows box available. And users don't have to install virtual machines, language runtimes, or any other dependencies to run these programs. Build and send them the binaries, and they can be used right away.

To automate it to the next level, you can also use a tool like goreleaser that can automatically build for multiple targets and even push the binaries as a release to GitHub, GitLab, or other source control repositories.

Golang's use of static binaries for its builds makes it an incredible tool to distribute command-line programs. You can build programs for multiple platforms from any platform, and they have no dependencies, making it a breeze to distribute them. Next time you're planning on building a tool like that, make sure you take a serious look at Go as its language.

Should you document your code?

Maurício Linhares — Wed, 02 Mar 2022 21:39:22 +0000

Time and again, the old "well-written code does not need documentation" rears its ugly head in software circles. Like all the other silver bullets and magical thinking solutions, it ignores context and the realities of building software in the real world. While well-meaning, as good code should be straightforward to follow, it's hard to express all the decisions that led to that code directly on it, and a couple of weeks from when you wrote, none of it will make sense.

In most codebases, we have tacit and explicit knowledge. Explicit knowledge is what is documented, structured, and well defined. You don't need to ask other people on the team about it. It's already there and available for you to consume. Tacit knowledge is the general understanding people on the team have about the code instinctively. It is knowing changing some part of the project usually leads to bugs or that you shouldn't change some specific configuration because it causes it to break or misbehave in exceptional cases.

The main goal of documenting code is to turn tacit knowledge into explicit, so you and everyone else don't have to keep all this context in memory. Humans are known to be bad at memorization. We routinely misplace objects we use day-to-day, create false memories because of stories we've heard, and forget stuff. As much as you might think you're going to remember the decisions and reasoning that led to the code you're writing, you won't. So, be selfish, think about you a month from now, looking at this code and not knowing why and how it works, you won't have a great time.

Why did I write this?

One of the most critical effects of documenting code is defining the context you wrote it. It's easy to look at code today and think who wrote this shit and later running git-blame finding out you wrote it. Why did I do that?

It's likely whoever wrote the shitty code you're looking at had good reasons to do so, and documenting such reasons makes them available to whoever is reading and touching it in the future. Document why the code does what it does, what was the context and reasoning to do it, so it's clear to whoever is reading it why it works the way it does so they know the requirements and expected behavior.

While going through a transition for one of our projects, I noticed code that performed health checking of other services (sending HTTP requests to verify these services are reachable) did something strange. The health checker is configured with multiple endpoints for each service. If one endpoint fails, it goes out of rotation, and traffic is sent to the other endpoints that are available (if a service has HostA, HostB, and HostC and HostA is failing, we only send traffic to HostB and HostC), but if all endpoints fall all hosts go back in the rotation.

There were no comments about why it behaved like this, so it was time to go some git spelunking. An incident a long time ago happened because someone made a mistake on their service configuration and added a health check that would never succeed, taking the whole service down. In response to this incident, we decided that if all endpoints for a service fail, we should assume it was just a misconfiguration and not that it is down and bring all endpoints back. This is now an obvious comment in the code that performs health checking. There is no need to dig into git history or go through JIRA tickets to figure out why something behaves the way it does.

You want to have the reasoning that led to the code explained, so no one needs to perform code archeology to figure out why something works the way it does. Having it clear will lead to fewer incidents and issues when people think they can change the code they see as it might look wrong, stupid, or unnecessary. Until we invent crystal balls where people can find out right away what the effects of changing code will be, it's best to say why it does what it does.

How does it do it?

Now on to the pernicious explanation of how the code works. People will frequently say you don't need to say what it does because it is right there in front of you, but there are valid reasons for you to explain the code as it is. One is to make it easier for people to associate the how with the abstraction you're using.

Is it using a design pattern? Make it part of the name of the class/function and add in the docs that this is the X pattern. Is it using a known algorithm? Say it in the comments that it uses that specific algorithm.

This makes it much easier for whoever is writing the code to dig into it as they don't have to find out you used specific patterns or algorithms. It's right there in the comments for them to prime their expectation on what they're looking at.

Internal use vs library code

Comments also change if you're dealing with code you're only using once, internally in a project, or code that will be widely shared inside an organization or to the broader public. If it's just internal code, it might make sense to mainly document the reasoning and context and be done with it. If what you're working on is a library that other people on other teams will use, the requirements will change.

A great example is Go's http package. Most types are documented with examples on using them, like the http.Get function. There is also a full separate package that allows you to unit test code that uses the http package, httptest.

So when writing libraries, you have to think both about how people are going to learn how to use them and what you can do to make the experience more straightforward. Will users need a particular way to perform unit testing? Should you provide mocks? Are there use cases that need specific setups you could simplify with a testing package? These are all issues you have to consider when writing code for other people to consume.

Yes, you should write docs

So, yes, you should document your code. Not only for the people that will be working on it in the future once you've left your job but also for you a couple of weeks from now trying to figure out why you wrote these strange lines of code you have no memory of. The less context you have to carry with you while working, the better the experience will be.

Transitioning a project to new owners

Maurício Linhares — Wed, 23 Feb 2022 01:48:30 +0000

If you've been long enough on the same team or job, you've been either the recipient or the originator of a transition. Maybe the team mission has changed, people have moved, or you are moving somewhere else, a new job or a new team and existing software need a new home. Frequently, this is a short engagement. Someone is leaving, so they schedule a couple of meetings to share knowledge, write a bunch of docs, and that's it.

But it doesn't have to be like this. We're just going through disbanding a team and finding new homes for our existing projects, and we're trying to make this less frustrating for everyone involved. Our main goal with this process is to make it easier for the other side to learn and maintain these systems.

The process involves:

What to do before the transition starts;
The actual transition work;
What happens once the transition completes;

Pre-transition

The first question you should ask of every project planned for transition is do we need this?. You're going to invest time in moving a resource to another team that will also have to invest in understanding and maintaining it, most likely without any previous experience. Can you kill it now and not even move?

Can you replace it with some other tool that is already available, a SaaS solution, or some other integration that would require less work than keeping the whole thing running?

Looking for an existing solution for the problem so you don't have to maintain it could be better than keeping a project alive. You'll have to invest time in setting up the project for the new team, and they will have to learn how it works and then maintain it (the most expensive part of running software). If you can use a tool that someone else will support at an acceptable price, a better solution might be to go that route instead of keeping the project alive.

If there's no other available solution or it would be too risky for the business to outsource this to another company, it's time to prepare the transition work.

Start by building an inventory of all the pieces to be moved. Source code repositories, dashboards, databases, deployments, CI pipelines, service accounts, tokens, and any other resource that the service uses. The more stuff you can find and document, the less risky the move, especially for service accounts and tokens. It's surprisingly common for service resources to be tied directly to personal accounts, and when you move ownership somewhere else, the service breaks, and no one knows why.

With the inventory in hand, it's time for a cleanup. Are there any resources that belong to specific people instead of a shared service account or shared roles? If there are, it's time to move them into a shared service account or roles. This will make the transition more manageable as the new team will join the service account or role and own the resources.

Shore up documentation, playbooks, incident reviews, and standard practices the team performs. All teams have rituals to deal with their day-to-day work that might not be documented. This is the time to turn these rituals into well-defined processes the new people should know. It doesn't mean they will have to perform the same practices the same way. You're just letting them know how the team operates right now.

Is there a typical sequence of steps people usually follow when they think an incident is going on? Make it a playbook. Got a specific sequence you perform database maintenances? Document it. Are deployments supposed to run in a particular order? Either encode that on your CI config or document it.

Review existing documentation to verify it's still valid and up to date. As projects age, explanations, diagrams and the like grow stale, this is the time for you to update or delete documentation that is not valid anymore. Old or wrong documentation is generally worse than not having any documentation at all. Code that performs complex computation should also be documented and thoroughly tested with examples of known failure scenarios. The new people joining have no context on the existing code or the decisions that led to its current state. Make sure it is clear why it does the work the way it does.

Define them now if you don't have clearly defined SLOs. Odds are you have clear expectations for what the service has to do and how long it takes to do the work (including alerting when it's not doing that on time), but you might not have them written down. Make them visible now, so the new team knows what to expect when they take this on.

Transitioning

So there's no way for you to throw this project in the trash, eh? Time to move it elsewhere. Build a timeline with the work that needs to be done for transition. There will be meetings, coding, presentations, so make sure you're accounting for them and, most importantly, it should have an end date. The time by which the project now belongs to the new team.

While knowledge sharing sessions are somewhat helpful, what is useful is to work on actual code building real stuff on the project. Find features that touch the code and have people on both teams pair up on building them, with the person on the original team as passengers.

You want the new team to get as much exposure to the codebase as possible, so make sure the work is done mainly with guidance from the current team but not active work. This will force the new team to go through a development environment setup and, hopefully, a complete release cycle. Building the feature, producing artifacts, and pushing them to production.

If there are on-call rotations, start with the new folks as secondaries but always reach out to them if incidents happen during working hours. You also want them to have exposure to the project when stuff goes sideways to learn how to debug issues and react to incidents.

Suppose incidents are infrequent enough that you don't expect them to happen during the transition, roleplay something that has occurred recently. When roleplaying, make the new team the drivers. You provide the problem and let them ask questions on what is going on to get more information and respond to their actions. Your main goal here is to let them feel the system and associate how the pieces fit together to make it work.

Start moving resource ownership to the new team. Source code, machines, tokens, teams, should all either include the new team or be moved to be owned by them. It's essential to move resources now, during the transition, as outages or issues caused by these moves will be easier to fix while the old team is still active and working on the code. If you let this happen only after all the work is done, there might be surprising behavior lurking around. Also, don't move it all at once. Move resources one at a time, make sure they work at their new home, and go to the next one. Making multiple changes at once on a project is a surefire way to cause issues.

Present the new people to the teams you're used to interacting with as the project's new owners. The existing team needs to make it visible that new people are working on this, and they are the ones that people should communicate with instead of their usual contacts. There is always a lot of informal communication between teams, and it's crucial to get the new folks into the flow, get them invited to scheduled meetings, and present them to people frequently in contact.

After transitioning

At this point, the old team has disengaged from the project entirely. They should not be on call and should not be involved in the day-to-day operation. You want a clean break for the old team because you want them to focus on the new work they're doing and not worry about what is going on at the old thing.

That's why it's essential to have a firm date to when it ends, so the people leaving don't feel like they have to wear multiple hats because they left this other project behind. It doesn't mean they shouldn't engage when needed, answer questions, or help out, but they should no longer have to worry about what's going on at the old project.

I hope these tips can lead you to better transitions in and out of projects!

Optimizing your MySQL queries

Maurício Linhares — Wed, 16 Feb 2022 19:10:22 +0000

Odds are, if you're doing any commercial programming, you've had to interact with a SQL database. They're a staple in programming due to how easy it is to store and retrieve data on them and the many free and open-source options available. Still, people have trouble figuring out how to optimize their queries correctly, so let's fix that.

While this text is focused on MySQL specifically, most of the discussion is also true for other databases. So, even if you don't use MySQL, you should learn something new that is going to be helpful on the SQL database you're using right now.

The database we'll use here is the employees db from MySQL itself. This repo has a docker-compose setup on how to start and run the database so you can run the commands along.

EXPLAIN to me

The first thing you have to learn to optimize queries in relational databases is to use the EXPLAIN command. EXPLAIN runs the query you've created through the database engine to check what it thinks it's going to take to run the command. It won't run the command, but it checks how many rows it believes it will need to access to satisfy the query.

So given the following table:

CREATE TABLE employees (
    emp_no      INT             NOT NULL,
    birth_date  DATE            NOT NULL,
    first_name  VARCHAR(14)     NOT NULL,
    last_name   VARCHAR(16)     NOT NULL,
    gender      ENUM ('M','F')  NOT NULL,    
    hire_date   DATE            NOT NULL,
    PRIMARY KEY (emp_no)
);

And a query that finds all the employees named Joe:

SELECT * FROM employees WHERE first_name = "Joe";

You running a query with EXPLAIN before it all, so an EXPLAIN on the previous query would be:

EXPLAIN SELECT * FROM employees WHERE first_name = "Joe"\G

The \G here is specific to MySQL to say you want the data printed in rows. Here's what it prints:

mysql> EXPLAIN SELECT * FROM employees WHERE first_name = "Joe"\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: ALL
possible_keys: NULL
          key: NULL
      key_len: NULL
          ref: NULL
         rows: 299645
     filtered: 10.00
        Extra: Using where
1 row in set, 1 warning (0.00 sec)

So when we ask the database what it thinks about this query, its response is a bit harsh. It says it has to look for 299645 rows (this table has 300024 entries, so it's almost every single row) to find out there are no employees named Joe. There are no keys to be used (keys here would be database indexes), and we're using a WHERE clause to filter the results.

So, we know we have to find users given the first name, and the EXPLAIN response shows there are no indexes on this specific column, let's add one:

CREATE INDEX employees_first_name_idx ON employees (first_name);

Now running EXPLAIN again:

mysql> EXPLAIN SELECT * FROM employees WHERE first_name = "Joe"\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: ref
possible_keys: employees_first_name_idx
          key: employees_first_name_idx
      key_len: 58
          ref: const
         rows: 1
     filtered: 100.00
        Extra: NULL
1 row in set, 1 warning (0.00 sec)

A single row checked!

How did this happen?

When you ask for the database to create an index on a column, it creates an optimized structure that allows you to find all rows associated with a specific value quickly. Think about it as if it maps a value ("Joe") to all rows that have that value on the column you created the index.

Indexes are the magical solution. We should create indexes for every column in the table, and we should be good, right?

"You should have an index for all columns you use for querying a table"

This is a pretty common misconception of how indexes work in relational databases. If you have an index for every column, every query is automatically optimized as the database can use all these indexes to find the rows, but this is not true.

MySQL can, in some cases, use more than one index when querying data. If we create separate indexes on first_name and last_name and try to find a specific user with values for both, we get this:

mysql> EXPLAIN SELECT * FROM employees WHERE last_name = 'Halloran' AND first_name = 'Aleksander'\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: index_merge
possible_keys: employees_first_name_idx,employees_last_name_idx
          key: employees_last_name_idx,employees_first_name_idx
      key_len: 66,58
          ref: NULL
         rows: 1
     filtered: 100.00
        Extra: Using intersect(employees_last_name_idx,employees_first_name_idx); Using where

So, we're still doing good, only one row checked, but the extra field has an interesting reference, intersect(employees_last_name_idx,employees_first_name_idx). MySQL sees two indexes here and decides to use them both for the query. It's still better than not having an index, but we're going through two separate data structures to find the value instead of a single one.

If we now get an index on both last_name and first_name, this is the output:

mysql> EXPLAIN SELECT * FROM employees WHERE last_name = 'Halloran' AND first_name = 'Aleksander'\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: ref
possible_keys: employees_first_name_idx,employees_last_name_idx,employees_last_first_name_idx
          key: employees_last_first_name_idx
      key_len: 124
          ref: const,const
         rows: 1
     filtered: 100.00
        Extra: NULL
1 row in set, 1 warning (0.00 sec)

The server digs into a single index to find the row that matches the expected value instead of two. Having multiple rows in an index also helps with covering indexes, a feature we'll discuss later.

Picking index columns

Picking indexes is about how you query the table and what fields are part of the queries. When looking for employees, we want to find them by first and last name, so we have an index on both of them. When creating an index for multiple columns, the order of the columns matters.

Given we have an employee called Georgi Facello, our index would have it referenced under the entry Facello-Georgi (the index is last_name, first_name), so it only works for queries where you are looking for last_name and first_name or last_name alone. A query that only looks for first_name can't use this index as the index matches left-to-right. For that, you'd need an index that starts with first_name.

It's also not valid for a query where you need to find someone given a last_name or a first_name, as you'd need both columns to be the leftmost columns in the indexes used, this would be a case where having separate indexes on the columns would be helpful. MySQL would run a union on both indexes to perform the query.

Now, if you're querying all or most columns in the index, what order should they have?

The columns with the biggest variety in values should come first. You can quickly calculate an average of how common values are in a column with a query like the following:

mysql> SELECT COUNT(DISTINCT emp_no)/COUNT(*) FROM employees;
+---------------------------------+
| COUNT(DISTINCT emp_no)/COUNT(*) |
+---------------------------------+
|             1.0000              |
+---------------------------------+

The primary key is the perfect example. Every row has a unique value, so we get 1. We now want the columns we'll use in the index to be as close to 1 as we can, let's look at first_name:

mysql> SELECT COUNT(DISTINCT first_name)/COUNT(*) FROM employees;
+-------------------------------------+
| COUNT(DISTINCT first_name)/COUNT(*) |
+-------------------------------------+
|               0.0042 |
+-------------------------------------+

Then look at last_name:

mysql> SELECT COUNT(DISTINCT last_name)/COUNT(*) FROM employees;
+------------------------------------+
| COUNT(DISTINCT last_name)/COUNT(*) |
+------------------------------------+
|               0.0055               |
+------------------------------------+

So last_name gives us better filtering than first_name, making it the best column on the index's left side. Still, this is an average, and as you might have found out the hard way in other places, averages are good at hiding outliers. One here would impact query performance directly so we also want to check if there are visible outliers on these columns:

mysql> SELECT last_name, COUNT(*) AS total FROM employees GROUP BY last_name ORDER BY total DESC LIMIT 10;
+-----------+-------+
| last_name | total |
+-----------+-------+
| Baba      |  226  |
| Gelosh    |  223  |
| Coorg     |  223  |
| Sudbeck   |  222  |
| Farris    |  222  |
| Adachi    |  221  |
| Osgood    |  220  |
| Mandell   |  218  |
| Neiman    |  218  |
| Masada    |  218  |
+-----------+-------+

So no outliers here. Values seem to be pretty close to each other. Let's look at first_name:

mysql> SELECT first_name, COUNT(*) AS total FROM employees GROUP BY first_name ORDER BY total DESC LIMIT 10;
+-------------+-------+
| first_name | total |
+-------------+-------+
| Shahab      |  295 |
| Tetsushi    |  291 |
| Elgin       |  279 |
| Anyuan      |  278 |
| Huican      |  276 |
| Make        |  275 |
| Sreekrishna |  272 |
| Panayotis   |  272 |
| Hatem       |  271 |
| Giri        |  270 |
+-------------+-------+

Not bad either. Values aren't that far from each other. The order we decided on for the index is a pretty good one.

Now, something else you need to account for when creating indexes is how you will sort the results?

Just like the database uses the index to find rows quickly, it can also sort the results if you're sorting on the same columns in the index order. For our last_name, first_name index it would mean to ORDER BY last_name or ORDER BY last_name, first_name. If multiple columns are sorted, they all have to be in the same direction, so either all ASC or all DESC. If they're not all in the same order the database can't use the index itself to sort the results and would have to resort to temporary tables to load the results and sort them.

Covering indexes

And another important reason to have indexes that hold multiple columns is the covering indexes feature that MySQL provides. When your query only loads the primary key and columns in the index, the database doesn't even have to look at the tables to read the results. It reads everything from the index alone. Here's what it looks like:

mysql> explain select emp_no, last_name, first_name from employees where last_name = "Baba"\G
*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: employees
   partitions: NULL
         type: ref
possible_keys: employees_last_name_idx,employees_last_first_name_idx
          key: employees_last_first_name_idx
      key_len: 66
          ref: const
         rows: 226
     filtered: 100.00
        Extra: Using index

The hint here is Extra: Using index, which means all the data is read from the index alone. As the index already contains all the information we need (all indexes include the primary key for the table), the database loads everything from it and returns, not even reaching out to the table. This is the best-case scenario for queries, especially if your index fits into memory.

Creating multiple indexes

Creating indexes isn't free. While they make it faster for us to find data, they also slow down any changes to the table as writing to columns with indexes will cause these indexes to be updated. So you have to strike a balance between making as many queries as possible fast but also allowing for fast INSERT/UPDATE/DELETE commands.

Summary

So, when optimizing, remember to:

Use EXPLAIN to find out how the database thinks your query will perform - check the command docs here;
Create indexes that cover the filtering and ordering you want to perform on your queries;
Evaluate the best order for the columns when creating indexes;
Try to load as much information from covering indexes as possible;

One of the best references to optimizing MySQL databases is the High Performance MySQL that is in its 4th edition and covers from database internals to how to design your database schema to make the most of MySQL. If you're running apps on MySQL, you should read it. If you're not using MySQL, it's very likely there is a book just like this one for it as well, and you should invest some time in reading it.

GoF Design patterns that still make sense in Go

Maurício Linhares — Mon, 07 Feb 2022 19:45:21 +0000

Since its release in 1994, the Design Patterns book continues to be a seminal work in building software. The book created a new shared vocabulary and named these repeated solutions we see all over different codebases. So much so there have been multiple other books on design patterns, documenting even more examples. You can quickly explain that your solution is an Adapter for anyone who has read about it without detailing what an Adapter is.

The material, as expected, isn't without critique, Peter Novig wrote his analysis on how many of these patterns are unnecessary or replaceable by simpler constructs in dynamic languages. The book was written mainly using C++ (and some Smalltalk), focusing on covering what was available in C++, so dynamic languages change or remove the need for some of these patterns.

A couple of weeks ago, a Twitter thread caught my attention, saying people shouldn't read this book anymore as many of the patterns are old or don't make much sense in most mainstream programming languages. I still have fond memories of when I read it back in college (2003, many moons ago) and wondered if it is true? Have languages evolved past this book, or do we still use the patterns and vocabulary defined there?

So, coming from a Golang perspective, what are some of the patterns in the book that are still usable nowadays? Let's have a look!

Builder

Builders are very much alive. They might have fancy names like functional options or fluent interfaces but the goal is still the same of the honored builder, simplify the creation of a complex object so that you don't end up with a single function call receiving dozens of parameters.

We'll now look at how you could set up a builder that produces *http.Request objects:

package gof_go

import (
   "context"
   "io"
   "net/http"
)

// NewBuilder creates a builder given a URL, we're going to use this so we don't leak
// the actual builder and have to worry about null/empty values on the builder itself.
// You could just use a struct directly here but it makes it a bit harder to validate
// defaults so we'll go for the simpler interface based solution.
func NewBuilder(url string) HTTPBuilder {
   return &builder{
      headers: map[string][]string{},
      url:     url,
      body:    nil,
      method:  http.MethodGet,
      ctx:     context.Background(),
      close:   false,
   }
}

// HTTPBuilder defines the fields we want to set on this builder, you could add/remove
// fields here.
type HTTPBuilder interface {
   AddHeader(name, value string) HTTPBuilder
   Body(r io.Reader) HTTPBuilder
   Method(method string) HTTPBuilder
   Close(close bool) HTTPBuilder
   Build() (*http.Request, error)
}

type builder struct {
   headers map[string][]string
   url     string
   method  string
   body    io.Reader
   close   bool
   ctx     context.Context
}

func (b *builder) Close(close bool) HTTPBuilder {
   b.close = close

   return b
}

func (b *builder) Method(method string) HTTPBuilder {
   b.method = method

   return b
}

func (b *builder) AddHeader(name, value string) HTTPBuilder {
   values, found := b.headers[name]

   if !found {
      values = make([]string, 0, 10)
   }

   b.headers[name] = append(values, value)

   return b
}

func (b *builder) Body(r io.Reader) HTTPBuilder {
   b.body = r

   return b
}

func (b *builder) Build() (*http.Request, error) {
   r, err := http.NewRequestWithContext(b.ctx, b.method, b.url, b.body)
   if err != nil {
      return nil, err
   }

   for key, values := range b.headers {
      for _, value := range values {
         r.Header.Add(key, value)
      }
   }

   r.Close = b.close

   return r, nil
}

So we have a builder that sets up some sane defaults (the body is empty, the method is GET, there's a default context). Here's what it looks like in use:

func TestBuilder_Build(t *testing.T) {
   request, err := NewBuilder("https://example.com/").
      AddHeader("User-Agent", "Golang patterns").
      Build()

   assert.NoError(t, err)

   assert.Equal(t, "Golang patterns", request.Header.Get("User-Agent"))
   assert.Equal(t, http.MethodGet, request.Method)
   assert.Equal(t, "https://example.com/", request.URL.String())
}

We could go complex with multiple headers, change the method, set a custom context, and it would still look clean and easy to read. That's the main advantage of using a builder to instantiate complex objects. You get to go as deep as needed but should still be allowed to create something with sane defaults quickly. While this builder follows the fluent API style, it's not the only way to produce a builder, as long as you find a way to separate the parametrization from the creation of a complex object that would still be a builder (like using the function opts linked above). The main goal here is to make it easier to build complex objects correctly.

Abstract factory/factory method

As the language lacks inheritance and focuses on composition, the primary way you'd represent factories is by having a function that produces an object as a parameter instead of abstract classes or methods.

A common reason you'd do this is to simplify unit tests. Imagine I'm testing a type that opens a socket connection to some other service, while you could use a net.Dialer directly, which would make testing it harder. If I give the object a method that produces a net.Conn object instead, I can easily replace the implementation by changing the factory function.

Here's what it would look like:

package gof_go

import (
   "bytes"
   "context"
   "fmt"
   "github.com/stretchr/testify/assert"
   "github.com/stretchr/testify/require"
   "net"
   "testing"
   "time"
)

type mockAddress struct {
   network string
   address string
}

func (m *mockAddress) Network() string {
   return m.network
}

func (m *mockAddress) String() string {
   return m.address
}

// mockConnection represents a connection used for testing only
type mockConnection struct {
   closed  bool
   buffer  *bytes.Buffer
   address *mockAddress
}

func (m *mockConnection) Read(b []byte) (n int, err error) {
   return m.buffer.Read(b)
}

func (m *mockConnection) Write(b []byte) (n int, err error) {
   return m.buffer.Write(b)
}

func (m *mockConnection) LocalAddr() net.Addr {
   return m.address
}

func (m *mockConnection) RemoteAddr() net.Addr {
   return m.address
}

func (m *mockConnection) SetDeadline(t time.Time) error {
   return nil
}

func (m *mockConnection) SetReadDeadline(t time.Time) error {
   return nil
}

func (m *mockConnection) SetWriteDeadline(t time.Time) error {
   return nil
}

func (m *mockConnection) Close() error {
   m.closed = true
   return nil
}

type socketClient struct {
   address string
   factory func(ctx context.Context, network, address string) (net.Conn, error)
}

func (s *socketClient) ping(ctx context.Context) error {
   c, err := s.factory(ctx, "tcp4", s.address)
   if err != nil {
      return err
   }

   defer func() {
      if err := c.Close(); err != nil {
         fmt.Printf("failed to close socket: %v\n", err)
      }
   }()

   if _, err := c.Write([]byte("PING")); err != nil {
      return err
   }

   return nil
}

func TestSocketClient(t *testing.T) {
   connection := &mockConnection{
      buffer: bytes.NewBuffer(make([]byte, 0, 1024)),
   }

   c := &socketClient{
      address: "example.com:40",
      factory: func(ctx context.Context, network, address string) (net.Conn, error) {
         connection.address = &mockAddress{
            address: address,
            network: network,
         }

         return connection, nil
      },
   }

   require.NoError(t, c.ping(context.Background()))
   assert.True(t, connection.closed)
   assert.Equal(t, "PING", connection.buffer.String())
}

We need mocks for net.Addr and net.Conn as we want to return in memory structs for those. Then our socketClient type has a factory property with type func(ctx context.Context, network, address string) (net.Conn, error), which is exactly the same function signature as net.Dialer#DialContext so the concrete implementation here would be the net.Dialer function.

Instead of thinking about factories as classes you use or extend, you can have them be a function you call that produces the object. There is no need for inheritance, abstract classes, or interfaces, just a function signature.

Adapter

It continues to be a widely used pattern all over, the database/sql package is an excellent example of the pattern. Every database driver implements the driver.Driver interface and registers it with the database/sql package. So it doesn't matter what database you're using, it all looks the same as you'll be interacting with objects from the database/sql package only most of the time.

We also see this same pattern on libraries like go-cloud where the specific details of cloud providers are hidden behind the standard interface the library presents.

Decorator

Decorators add functionality to an existing object you might have no control over without breaking its interface. You build a decorator by creating an object that wraps another but has the same methods and forwards calls of these methods to the object being decorated, adding some functionality on top of them.

Typical use cases are adding buffers to IO classes (which is the example we'll see in a bit), adding metrics or tracing to types you don't have control over, like external libraries, and many more. The main goal is hiding from the calling code that something has changed.

The lack of inheritance makes decorators in Go a bit more problematic, as you can see on this ancient issue on collecting metrics when resolving DNS entries. Given that the code that calls the net.Resolver code expects the exact type and there is no inheritance, we can't wrap the resolver as we would in languages that allow inheriting. You can only use decorators in Go if you're dealing with interfaces or function signatures. If you have to wrap a struct, the only way is to change the code that depends on the struct to an interface with the same methods.

Now we'll look at a decorator that adds a buffer to io.Reader objects (this is not supposed to be the most efficient and optimized version, it's just an example):

func NewBufferedReader(wrapped io.Reader, length int) io.Reader {
   if length <= 0 {
      length = 1024
   }

   return &bufferedReader{
      currentIndex:  0,
      lastIndex:     0,
      buffer:        make([]byte, length, length),
      wrappedReader: wrapped,
      err:           nil,
   }
}

type bufferedReader struct {
   currentIndex  int
   lastIndex     int
   buffer        []byte
   wrappedReader io.Reader
   err           error
}

func (b *bufferedReader) Read(p []byte) (n int, err error) {
   if len(p) == 0 {
      return 0, errors.New("an empty slice was provided to Read")
   }

   availableBytes := b.lastIndex - b.currentIndex

   if availableBytes == 0 {
      if b.err != nil {
         return 0, b.err
      }

      if read, err := b.wrappedReader.Read(b.buffer); err == nil || err == io.EOF {
         b.err = err
         b.currentIndex = 0
         b.lastIndex = read
         availableBytes = read

         if availableBytes == 0 {
            return 0, b.err
         }
      } else {
         b.err = err
         return 0, err
      }
   }

   expectedBytes := len(p)

   bytesToRead := availableBytes
   if availableBytes > expectedBytes {
      bytesToRead = expectedBytes
   }

   copy(p, b.buffer[b.currentIndex:b.currentIndex+bytesToRead])
   b.currentIndex += bytesToRead

   return bytesToRead, b.err
}

We wrap any existing io.Reader and add a buffer on top of it. For the code that was using a reader before, nothing has changed, it is still an io.Reader object, but we have introduced new functionality without breaking the contract. It's also how the IO package works, with decorators adding features on top of reader and writer objects. This pattern is still alive in Go, albeit not in as many places as in other languages.

Facade

Still here! Facades hide a complex or extensive API behind a small interface you can interact with without understanding all the details behind the scenes. Like we mentioned before, go-cloud is an excellent example of both adapter and facade, as it hides the complex details of talking to cloud providers behind a straightforward interface.

Proxy

Proxies are an interface to some other object that might be expensive to create or interact with directly. A typical case for using proxies is in ORM tools when loading an association. Think a User has Posts. You don't necessarily need to load the posts for the user every time you're loading a user so that ORM tools will hide the Posts behind a proxy object. They will wait until you try to do something with Posts that requires looking at the objects. They'll now load them from the data source where they live, instantiating the Posts collection.

If you never try to access them, you won't pay for the cost of loading them, and that's the advantage of using a proxy. You don't have to have that object directly available to you all the time. You can apply these lazy load techniques to produce more efficient code and use fewer resources.

Like the Decorator example above, the lack of inheritance also forces proxies to only be available for interface or function types. Reflection in Go also doesn't offer dynamic proxies (creating a proxy object from scratch in runtime) as you'd find in languages like Java and C# or a method_missing capability as we have in Ruby, you have to generate the proxy code at compilation time.

Chain of responsibility and Command

Two patterns together? Yes!

Almost every single case of Chain of responsibility I've ever seen has been with Command. I'm not even sure if it makes sense to implement a chain of responsibility without Command objects (or functions). If you've done any HTTP server development in go you've seen them together already:

package http

type Handler interface {
    ServeHTTP(ResponseWriter, *Request)
}

This is our Command for handling HTTP requests. It doesn't matter how you're going to implement your functionality. As long as it matches this interface, it's going to work. We also have the same interface defined as a function signature:

package http

type HandlerFunc func(ResponseWriter, *Request)

Both should be interchangeable. The only thing that matters is that they have the same inputs and outputs. Both take an http.ResponseWriter and a http.Request and have no return types.

Good, we have the Command. Where does Chain of Responsibility come from then? Middlewares!

Here's the middleware interface:

package gof_go

import (
   "fmt"
   "net/http"
)

type HTTPMiddleware func(w http.ResponseWriter, r *http.Request, next http.Handler)

func LoggingMiddleware(w http.ResponseWriter, r *http.Request, next http.Handler) {
   fmt.Printf("REQUEST %v\n", r.URL.String())
   next.ServeHTTP(w, r)
}

func OkHandler(w http.ResponseWriter, r *http.Request) {
   w.WriteHeader(http.StatusOK)
   if _, err := w.Write([]byte("OK")); err != nil {
      fmt.Printf("failed to write to body: %v", err)
   }
}

func MidddlewareToHandler(middleware HTTPMiddleware, next http.Handler) http.Handler {
   return http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
      middleware(writer, request, next)
   })
}

Almost the same as the http.Handler, the only difference is that it also takes a next parameter that will be called if the current handler thinks it should. Let's see what it looks like being used:

package gof_go

import (
   "github.com/stretchr/testify/assert"
   "github.com/stretchr/testify/require"
   "io"
   "net/http"
   "net/http/httptest"
   "testing"
)

func TestOkHandler(t *testing.T) {
   ts := httptest.NewServer(MidddlewareToHandler(LoggingMiddleware, http.HandlerFunc(OkHandler)))
   defer ts.Close()

   res, err := http.Get(ts.URL)
   require.NoError(t, err)

   ok, err := io.ReadAll(res.Body)
   require.NoError(t, err)
   require.NoError(t, res.Body.Close())

   assert.Equal(t, "OK", string(ok))
}

Here we use the MiddlewareToHandler method to add the logging middleware to the OkHandler, but you could add any number of features with other middlewares here. Authorization, feature flipping, rate limiting, and all without changing the commands that handle the requests and in any order you'd like. As the HTTPMiddleware implementors have complete control over if the request continues or not, you can add functionality (almost like decorators) and decide if the request flow continues or not.

While this example only adds a single middleware, the actual stack could go as deep and add as many middlewares as you'd need by making more calls to MiddlewareToHandler. You can even go fancy and use a builder object to create the actual Chain of Responsibility, like the gorilla mux project does.

Iterator

While you can build iterators in Go for specific use cases, the fact that the language doesn't have a collections library like other languages makes it less common generally. With generics coming along in 1.18 we might see collection libraries showing up and there might be a standard iterator or an easier way for objects that are not in the language itself (like arrays, slices, and maps) to be iterated over using for ... range loops.

The most famous iterator in the language might be sql.Rows, which is an object everyone sees once they're dealing with SQL in Go.

Observer

Observers are almost first-class citizens in go, given the existence of channels. Most of the code you'd write to support observers, especially one producer, one consumer pattern, is already baked in how we use channels in the language. It starts to get a bit more complicated when you need to publish a message and have multiple consumers see the same message, as you'd need various channels to make a fan-out behavior work.

Strategy

Strategies are known as classes that implement multiple algorithms but expose the same interface. You could have numerous sorting methods (merge sort, bubble sort, quick sort) all implemented in separate classes but with the same interface (a single method that takes an array, for instance).

While you could build strategies in Go using single-method interfaces, that's not very idiomatic. You're better off defining a function signature and having each algorithm being a function that implements the same signature.

package gof_go

import "testing"

type Sorter func(a []int)

func MergeSort(a []int) {
   // implementation
}

func QuickSort(a []int) {
   // implementation
}

func TestSorter(t *testing.T) {
   var sorter Sorter
   sorter = MergeSort
   sorter([]int{10, 7, 5, 2, 4})
}

Using functions we remove the need to have classes for every algorithm when it would just be a method in the class anyway.

Template Method

Due to the lack of inheritance and the availability of functions as first-class objects, becomes the Template Function. An abstract method available for you to implement in a subclass becomes a function taken as a parameter just like we had with factories before.

Sorting objects is a pretty typical example of Template Method still in action in Go:

package gof_go

import (
   "github.com/stretchr/testify/assert"
   "sort"
   "testing"
)

type Racer struct {
   Position int
   Name     string
}

func TestSortRacers(t *testing.T) {
   racers := []*Racer{
      {
         Position: 10,
         Name:     "Alonso",
      },
      {
         Position: 2,
         Name:     "Verstappen",
      },
      {
         Position: 1,
         Name:     "Hamilton",
      },
      {
         Position: 12,
         Name:     "Vettel",
      },
   }

   sort.SliceStable(racers, func(i, j int) bool {
      return racers[i].Position < racers[j].Position
   })

   assert.Equal(t, []*Racer{
      {
         Position: 1,
         Name:     "Hamilton",
      },
      {
         Position: 2,
         Name:     "Verstappen",
      },
      {
         Position: 10,
         Name:     "Alonso",
      },
      {
         Position: 12,
         Name:     "Vettel",
      },
   }, racers)
}

Like other implementations, we don't know what algorithm is used to sort the objects. All we do is provide a way to compare them with a function. It removes the need to use a separate class as you'd do with implementations of Template Method in languages that don't have functions as first-class citizens.

What Is Dead May Never Die

As we've seen, many of the patterns are still here with us, in the standard library, shared third-party libraries, and frameworks. The patterns I haven't mentioned are sometimes too specific for the problems they're solving (like interpreter and visitor), so it's a bit harder to find their usage in the wild, which doesn't mean they're unnecessary.

While it might be harder to read the book nowadays if you have no C++ or Smalltalk experience, other books cover the same patterns in a newer format, like the Head First Design Patterns.

So while languages have evolved, they haven't removed the need for these patterns and the advantages they can bring to your codebase when used correctly. Keep them on your toolbelt, so you know when to use and how to spot them in code, its going to improve your coding vocabulary and the solutions you'll build.

What to ask at job interviews when you are being interviewed

Maurício Linhares — Wed, 12 Jan 2022 18:42:17 +0000

There are many guides on how to ace job interviews, you'll find books, videos, courses on multiple subjects covering every aspect possible, including even prep that is just for specific companies (like checking questions Facebook is currently asking), but we talk very little on how you should also interview the people you're talking to make sure you do want to work at that place. Switching jobs to find that you ended up in a bad work environment on a dead-end team is demoralizing and might affect your future career prospects.

Most of the questions here assume the people you're talking to will be at least a little bit truthful in their answers. I'd also recommend exercising your network to find friends or friends of friends that work at the place to verify if the answers are accurate or if they're painting a rosy picture to make sure you accept the offer. Most businesses will have some of the problems we will be discussing next, so it's going to be up to you to decide if you do want to sign up for these issues or not, there is no perfect place, but you should go in being comfortable on what the challenges are going to be. Glassdoor, Blind, and Fishbowl are usually hit and miss, with stuff being, either too rosy or just hell, so I can't say I find them very effective.

Also, unless you're going for a director-level/decision-making position, don't trick yourself into thinking you're going to change the whole environment and make it bearable, odds are you won't and the energy you'll spend trying to fix it without having the actual power to do so would be better spent on a job that doesn't require you do pull a Sisyphus every day. The job market is red hot right now, and you should be able to give that bad-looking job a pass.

Now let's look at a couple of questions you could ask to figure out if it is a good job or a tar pit.

What is the on-call schedule like? How many incidents does the team handle every week on average?

In most sane places, the teams that build apps are also responsible for ensuring they're running as expected in production. This means that when something bad happens, someone on the team will be called/paged and get the dreaded PagerDuty alert, you have one incident phone call. Most of us have been there, and pager pain is very real. Working on a team that is firefighting all the time is the quickest path to burnout you could take as you'll be constantly worried about the next time you'll be fast asleep and the phone is going to ring, destroy your night and the next day.

What you're looking for here is a clear on-call schedule with multiple people on rotation and a secondary that also rotates. The schedules should usually be a couple of days long (I'd say a week is generally good enough), with at least two weeks off until you're on call again. Back-to-back on-call schedules are demanding, especially if incidents are frequent because there is very little time you're not worried about incidents.

How much time does the team spend firefighting or doing unplanned work? If a team spends over 30% of its time just fixing bugs and working through incidents instead of building features, there are deeper issues on the team itself. The root causes here could be lack of investment in reliability, automation, lousy planning, death marches, none of which are a good signs. You could dig deeper here with the people you're talking to as to what is going to be done to bring the team back into a more manageable schedule and lower the amount of toil/bugs/incidents if there's no plan or they think this isn't a problem (it's just how we do work), it's a pass.

What is the engineering onboarding process like? What about onboarding on the team itself?

It might sound like a first world problem thing, but having a well-defined onboarding process is critical to having employees that start contributing to the bottom line as fast as possible. Businesses that are hiring but have no plan to get the people they're hiring into actually doing the job aren't doing a good job at hiring and might not be doing an excellent job at other places.

Depending on the company's size, they might not have a general engineering and a specific team onboarding; there could just be one, that's not a problem. What you want to know is what will happen once you join and the expectations. Is there going to be a buddy taking you through the process, someone you can ask questions and get unstuck? How will you get used to the projects you'll be working on? Is there documentation? What are the first things you're going to be working on?

The expectation here is that they won't just say here's the source code, now figure it out but that there is some well-defined way for you to go along (preferably with someone else on the team as a pair) to get you used to the tooling, code, development and deployment process. If there is no onboarding process at all, be ready just to be thrown at stuff and be left to fend for yourself and hope for the best. While it's entirely possible to make this work (as I have had to make this work multiple times) I'd rather be at a place that is going to guide and help me get productive faster instead of having to ask please can you help me to multiple people that might or might not care about helping me out.

How long has the person managing the team been a manager?

It might sound a bit unfair, but, especially as you get more senior, joining a team in a new company with a manager that does not have much experience managing people is risky. While having a manager with a lot of experience isn't a guarantee that everything will be unicorns and rainbows, a new manager is more likely to lack many skills required to build and lead an effective team.

I've heard horror stories from friends that got a new job with a manager that had just been promoted after a stellar stint as an Individual Contributor, which led to a lot of clout in the business, but that couldn't manage people at all. Given the new manager's success in his previous career, he was given a lot of leeway on running the team, and most of the criticism would fall on deaf ears, making it hard to move the needle in any direction. This leads to team members not being promoted, not being assigned to important/impactful projects, and not being promoted.

Being under a bad manager is not only bad for your mental health, but it's also bad for your career as it might stall your professional development and keep you from moving at the pace you expected you'd go. So, beware, joining a team with a newly minted manager carries many risks. Make sure you know what you're signing up for if this is the case.

What is the vision for the team? What are the important projects it is taking on in the next 12 months? What would be my goals/projects if I joined?

Here what you're looking for is to make sure the team you're joining is healthy and doing important work for the business. Teams should have large, impactful projects they have planned and are working on, they should have a vision of what the team's job is, and as they're hiring new people, they need to have a reason for why they would be hiring you.

If the team doesn't have important projects to work on, odds are it might not be an essential part of the business, or the manager hasn't been able to pull important work or doesn't want to do meaningful work. It's not unusual for, in larger companies, there to be teams that are just coasting and delivering small things here and there. If that's what you're looking for, the answers to these questions might give you hints this is it.

Otherwise, if you want to do impactful and essential work, you want to make sure the folks you're talking to have good answers. They should have a clear vision for what the job is, it can't just be "deliver features", every team is working on some domain-specific work, so it has to be "improve the way warehouses are managed", "deliver code faster to production" (if they're a platform team, for instance). And on that vision, they should also have cool and exciting projects they're working on that have a tangible impact on the business, like building a new database-as-a-service product to capture a growing market. This kind of team allows you to grow into a better professional and offers challenges that will get you more experience and promotions.

And last but not least, ask why they're hiring a new person and what would be the work you'd be doing, at what level you're expected to deliver. Are you here to lead a new project? Continue an existing project? What are the opportunities available for you to grow? Having good answers for these questions means they've done their homework on why they need to hire someone else, and you'll get a feel of the expectations they're placing on you if you join.

Is there a career development plan, and what does it look like?

You might be in for the love of writing code and stuff, but we all want to be paid and eventually make more money next year than we do right now, and to do so, you have to progress on your career in some way. Ideally, you want the place to have a ladder with multiple levels and well-defined expectations for what you should be doing to get to the next level. Not knowing what you have to do to be promoted leaves you at the whims of your manager, and that is going to be the luck of the draw, the manager might work to get you promoted or not, taking most of the possibility of you influencing this out of your hands.

Having a known set of activities you have to attain to move up allows you to plan your work accordingly, ticking the boxes that need to be ticked instead of just randomly doing stuff and hoping for the best. That also allows you to decide the crucial pieces of work that need to be done and focus on them instead of over-engineering or investing too much time on stuff that won't matter that much for your career. This gives you some power to decide your path instead of leaving it all in someone else's hands.

What are the self-development benefits provided?

No tech company survives with a workforce that doesn't improve, so most of them will offer a stipend or access to education related to the work. You want to be at a place that will buy you that book you need for some new project you're working on or pay for a course or event you feel like will be essential for your productivity at work.

Places that require you to invest in yourself even when your productivity improvement goes mainly to the business aren't the best place to stay.

How long does it take for code that is feature complete to reach production in some way?

Here you're looking for minutes as the answer. In some businesses, hours would be acceptable. More than that is typically bad news unless you're applying for a job at NASA and your code will ship in a satellite. The speed code arrives in production signals maturity of the business development process. The faster it is, the more mature and uneventful shipping code is, and this is what you're looking for at your job. The scarier it is for a team to push changes, the more likely they will cause incidents and break stuff.

Teams with a well-oiled deployment process most likely practice continuous integration, have a considerable mass of automated testing (unit, integration, smoke, black box), observability tools, and ways to rollout code that doesn't update everything all at once. That's why we have the in some way there in the question, you don't have to necessarily update everything to everyone at once (it might even be impossible, given the scale of the work), but you still have a way to put that code in front of actual users and verify it is doing the job. This process should also be fully automated and executed from a CI system instead of a developer's local environment.

Most of the time, not delivering code fast to production happens because the business doesn't trust the team. New processes or steps are added, like change request approvals, manual QA steps, staging environments, so the changes are less likely to break. Unfortunately, adding more processes instead of improving the engineering practices doesn't work. You end up with a very long cycle time to get changes out, with larger changesets that are much more likely to break once deployed, and the cycle begins anew, with even more processes to try to build trust.

How do you measure and maintain code quality?

How does the team make sure the code shipped is of quality and meets the standards defined? Are there standards well defined? Is there time for code gardening, refactors, improvements, and upgrades? Are there automated tools running on the codebases to ensure standards are kept?

We expect a review process or pair programming, static code analyzers, automated formatters, test coverage requirements (at different levels, it shouldn't just be unit or integration tests), and essential documentation. Coding should also be a group exercise, and people should not be working in a vacuum. The team should be working together to deliver on its commitments.

For instance, if there is a pull request review process, how long does it usually take to review a pull request? Minutes? Hours? Days? The longer it takes, the more time wasted waiting, making conflicts more likely. Long review cycles are also a sign the team might not work very well together, and instead of being a team, it might just be a collection of people that happen to be together.

Pick your poison

No place will ever be the perfect workplace; you're going to find issues, problems, walls that can't be breached, and people that can't be reasoned with; still, getting more information allows you to make a better decision when accepting a job or not. You should always ask as many questions as you can about the company, the business, the team, and the people surrounding the team to ensure that it is the place you want to work. Don't forget your job is most likely the single thing you're going to spend most of your time in for a long while. You have to make sure this time is worth it.

Building a Redis client from scratch in Go

Maurício Linhares — Fri, 07 Jan 2022 16:07:33 +0000

Writing applications that talk to each other over the network is something we do almost daily but we usually don't worry too much about how the data moves along or what the communication protocol looks like, as we use HTTP clients and servers to do most of the work. Still, it's good to get dirty every once in a while with sockets and byte streams and learn a little about how we communicate with other applications. Today we're doing that with Redis.

First, why Redis? Redis is a widely used data structures store that has a straightforward protocol format we can cover in a single blog post but that is still very powerful in what it allows us to do. With Redis, you can deal with the common key-value store use case but you can also use other tools, like lists, sets, maps so the simplicity of the protocol also shows that you don't have to complicate the way you access data even if the implementation you might have on the server-side is indeed complex.

Our main goal here is to understand the basics of network programming in Go, write code that parses messages from and writes them to a Redis server and understand some options we have when writing apps that operate with sockets in the language. We are not going to cover optimizations, TCP/IP, or other advanced concepts, we're just going to write a simple Redis client. This client also doesn't work for Redis pub-sub, only for the general request-response interface.

The full source code for this post is available on github.

The protocol

The Redis wire protocol is what we usually call a terminator-based protocol as you know that a message has ended once you find a \r\n. For instance, if you want to define a simple string in Redis it's going to be written as:

+OK\r\n

The + marks this as a simple string and the content of the simple string is all the bytes until you find a \r\n. The other basic types are errors (-ERR failed to do something\r\n) and integers (:1245\r\n).

But what if the string you want to send has a \r\n in it, you might ask, this is where bulk strings some into place. This type is different than the others as it doesn't use the \r\n to mark the end alone but it also has a length of bytes that should be read as the value. Here's an example:

$6\r\ngolang\r\n

A bulk string stats with the $ symbol, then a number as a string until a \r\n that is the length in bytes that you must read until you find another \r\n marking the end of the bulk string. Given we have the length we should read, it doesn't matter what are the contents inside the bulk string as we're not checking anything there, we're just reading everything until the end. So whenever you have to write a string that could have a \r\n in it you have to use bulk strings. Bulk strings have a special case that is a null bulk string, a bulk string with a -1 length should be represented by clients as a null value and not an empty string, these are returned as "$-1\r\n"

And last but not least we have the array type that is a collection of the previously defined types (including arrays of arrays). They start with a * followed by an integer length (the number of items the array holds) and then each item in order, an array of two simple strings would look like "*2\r\n+foo\r\n+bar\r\n", the *2\r\n piece says this is a two items array and then we get two simple strings. Any of the previously defined values are valid on arrays, including arrays themselves, if we wanted to have an array that is two separate arrays of simple strings we could define it like:

*2\r\n*3\r\n+foo\r\n+bar\r\n+sample\r\n*1\r\n+nope\r\n

Which would be equivalent to having:

[
    [ "foo", "bar", "sample"],
    ["nope"]
]

So you can mix and match values for arrays as you see fit. Arrays also have a special case, the null array, which just like the bulk string case is different from having an empty array in Redis, null arrays are defined as *-1\r\n", an array with a -1 length.

Reading data

Now that we've covered the protocol basics, we'll start with the reader part of our implementation. Given we know this is mostly a terminator-based protocol (with only bulk strings as a special case) we can use a bufio.Scanner to parse the stream of data, breaking up the lines on \r\n and if we do see a bulk string do a bit more magic to return it to the code reading from the scanner as a single line.

When using a bufio.Scanner you provide a SplitFunction with the following signature:

type SplitFunc func(data []byte, atEOF bool) (advance int, token []byte, err error)

So you get the data in bytes that are available and a bool value signaling if we're at the end of the stream or not. You then have to return how many bytes to advance on the stream, the data that should be returned to whoever is reading the scanner, and any error if it has happened. If you still don't have enough data to read, you just return a nil token value to ask the scanner to read more data (for instance, if there are no \r\n anywhere you know right away this isn't a full Redis response). Also, atEOF can be true and data not be empty, as you could be at the end of the stream because the server has closed the connection but it did write a full response back so even if you're at the end of a stream on a scanner you should make sure you check what the data is and return it to whoever is consuming from the scanner.

Our implementation here is done in two parts, one is the split function we use for the scanner, which breaks the Redis responses in lines, and the code that reads the lines, interprets them, and turns them into strings, integers, or arrays. Here's what it looks like:

package redis_client

import (
    "bufio"
    "bytes"
    "encoding/base64"
    "errors"
    "fmt"
    pkgerrors "github.com/pkg/errors"
    "io"
    "strconv"
)

const (
    defaultBufferLength = 10140
    typeSimpleString    = '+'
    typeErorr           = '-'
    typeInteger         = ':'
    typeBulkString      = '$'
    typeArray           = '*'
)

var (
    separator = []byte("\r\n")
)

type Reader struct {
    scanner *bufio.Scanner
}

func NewReader(r io.Reader) *Reader {
    scanner := bufio.NewScanner(bufio.NewReaderSize(r, defaultBufferLength))
    scanner.Split(redisSplitter)

    return &Reader{
        scanner: scanner,
    }
}

func (r *Reader) Read() (*Result, error) {
    return readRESP(r.scanner)
}

// redisSplitter splits a byte stream into full lines in the redis protocol format. it reads a whole item
// (string, bulk string, error, integer, or array) and then returns it as a line. code reading from scanners
// created with this function can safely assume that if they got a line it's a full line that does not need
// any extra checking.
func redisSplitter(data []byte, atEOF bool) (advance int, token []byte, err error) {
    // a valid redis message has at least 3 characters, if less than that ask for more stuff
    if len(data) < 3 {

        if atEOF {
            return 0, nil, fmt.Errorf("unexpected end of stream, a redis message needs at least 3 characters to be valid, actual content in base64: [%v]", base64.RawStdEncoding.EncodeToString(data))
        }

        return 0, nil, nil
    }

    found := bytes.Index(data, separator)
    // if we could not find a \r\n and the stream is at its end, this stream is broken and can't be recovered
    if found == -1 && atEOF {
        return 0, nil, fmt.Errorf("unexpected end of stream, there should have been a \\r\\n before the end, actual content in base64: [%v]", base64.RawStdEncoding.EncodeToString(data))
    }

    // if there is no \r\n we need to read more data, this means this result isn't finished yet
    if found == -1 {
        return 0, nil, nil
    }

    // bulk strings are special, they change the binary format from a terminator based one (\r\n to close messages)
    // to a length based one. so once we figure out this message is a bulk string (like `$6\r\nfoobar\r\n`),
    // we have to read the length and make sure there are at least length + 2 bytes after the first \r\n to
    // show that we do have the whole bulk string here. it is not safe to just find all \r\n in a bulk string
    // because there could be \r\n tokens as part of the string itself, so we always have to make sure we consume
    // the length and use it to read the whole value.
    if data[0] == typeBulkString {
        length, err := strconv.ParseInt(string(data[1:found]), 10, 64)
        if err != nil {
            return 0, nil, fmt.Errorf("message starts as bulk string but length is not a valid int, actual content in base64: [%v]", base64.RawStdEncoding.EncodeToString(data[0:found]))
        }

        // a -1 length means this is a null string and should be returned as such to clients, null and empty
        // strings are different things in redis. this is the only time we return a bulk string sign ($) here
        // as we'll use it as a marker for null strings. for someone reading from a scanner
        // there should be no difference between a simple or a bulk string as we have already
        // parsed the lengh and we'll return only the actual string contents.
        if length == -1 {
            return 5, []byte("$"), nil
        }

        // a 0 length means an empty string, an empty string is not the same as a null string on redis
        if length == 0 {
            return 6, []byte("+"), nil
        }

        // this is the position of the first \r\n + the expected length + 4 which is the \r\n twice we have on bulk strings
        expectedEnding := found + int(length) + 4
        if len(data) >= expectedEnding {
            // given here we already have all the information we need to return this as a string,
            // we don't return the length anymore, we return this as if it was a normal string.
            // now we set the first `\n` we have to `+` so the code parses it as a simple string
            // as we have already capped the returned slice do the length of the string.

            start := found + 1
            data[start] = '+'
            return expectedEnding, data[start : expectedEnding-2], nil
        }

        if atEOF {
            return 0, nil, fmt.Errorf("unexpected end of stream, stream ends before bulk string has ended, expected there to be %v total bytes but there were only %v, actual content in base64: %v", expectedEnding, len(data), base64.RawStdEncoding.EncodeToString(data))
        }

        // not enough data, ask for more data to be read
        return 0, nil, err
    }

    return found + 2, data[:found], nil
}

// readRESP reads from a scanner that was initiated with `redisSplitter`. it expects every scanned line to be
// a full line of a data type redis supports (unless it's an array, arrays start with the length of the array only).
func readRESP(r *bufio.Scanner) (*Result, error) {

    for r.Scan() {
        line := r.Text()
        switch line[0] {
        case typeSimpleString:
            // if a string, just remove the marker and return it
            return &Result{
                content: line[1:],
            }, nil
        case typeBulkString:
            // a bulk string is only returned if it is nil, otherwise it is turned as a simple string
            return &Result{
                content: nil,
            }, nil
        case typeErorr:
            // if an error just wrap the error and return it
            return &Result{
                content: errors.New(line[1:]),
            }, nil
        case typeInteger:
            content, err := strconv.ParseInt(line[1:], 10, 64)
            if err != nil {
                return nil, fmt.Errorf("failed to parse returned integer: %v (value: %v)", err, line)
            }
            return &Result{
                content: content,
            }, nil
        case typeArray:
            // the first thing to be done when we find an array is to find its length, if not `-1` we then
            // read items from the scanner until we've read all items on the array.
            length, err := strconv.ParseInt(line[1:], 10, 64)
            if err != nil {
                return nil, fmt.Errorf("failed to parse array length: %v (value: %v)", err, line)
            }

            if length == -1 {
                return &Result{content: nil}, nil
            }

            contents := make([]interface{}, 0, length)

            for x := int64(0); x < length; x++ {
                result, err := readRESP(r)
                if err != nil {
                    return nil, pkgerrors.Wrapf(err, "failed to read item %v from array", x)
                }

                contents = append(contents, result.content)
            }

            return &Result{
                content: contents,
            }, nil
        }
    }

    if r.Err() == nil {
        return nil, errors.New("scanner was empty")
    }

    return nil, r.Err()
}

Most of the complexity here lies at the redisSplitter function, that is breaking up the bytes into lines with a full item to be read. We also cheat a little on the bulk strings by turning them into simple strings on the fly. This is done mostly because we have the code that splits lines separate from the code that parses the values, if we weren't using a bufio.Scanner and were reading straight from the byte stream (the io.Reader) we could remove the special case for bulk strings but that would also require us to do a lot of the work that the scanner is doing. Doing it like this pushes the complexity of reading enough bytes and buffering them into the scanner instead of into our code.

We're also using this Result type here all over but it's just a wrapper to make it easier to transform a value from Redis if you know what it is supposed to be, here's the code:

package redis_client

import "fmt"

type Result struct {
    content interface{}
}

func (r *Result) Err() error {
    err, ok := r.content.(error)
    if !ok {
        return nil
    }

    return err
}

func (r *Result) Int64() (int64, error) {
    if err := r.Err(); err != nil {
        return 0, err
    }

    result, ok := r.content.(int64)
    if !ok {
        return 0, fmt.Errorf("content is not an int64: %#v", r.content)
    }

    return result, nil
}

func (r *Result) String() (string, bool, error) {
    if err := r.Err(); err != nil {
        return "", false, err
    }

    if r.content == nil {
        return "", true, nil
    }

    result, ok := r.content.(string)
    if !ok {
        return "", false, fmt.Errorf("content is not an string: %#v", r.content)
    }

    return result, false, nil
}

func (r *Result) Slice() ([]interface{}, error) {
    if err := r.Err(); err != nil {
        return nil, err
    }

    if r.content == nil {
        return nil, nil
    }

    result, ok := r.content.([]interface{})
    if !ok {
        return nil, fmt.Errorf("content is not a slice: %#v", r.content)
    }

    return result, nil
}

func (r *Result) Content() interface{} {
    return r.content
}

It just holds the reponse that was read from redis and makes sure it is the right type before returning it.

Writer

The writer is a simpler inverse version of the reader. Given we already know the protocol, what we have to do here is to write the data in the correct format:

package redis_client

import (
    "encoding/base64"
    "fmt"
    "github.com/pkg/errors"
    "io"
    "strconv"
)

type Writer struct {
    writer io.Writer
}

func NewWriter(w io.Writer) *Writer {
    return &Writer{
        writer: w,
    }
}

func (w *Writer) write(messageType byte, contents ...[]byte) error {
    if _, err := w.writer.Write([]byte{messageType}); err != nil {
        return errors.Wrapf(err, "failed to write message type: %v", messageType)
    }

    for _, b := range contents {
        if _, err := w.writer.Write(b); err != nil {
            return errors.Wrapf(err, "failed to write bytes, content in base64: [%v]", base64.RawStdEncoding.EncodeToString(b))
        }
    }

    return nil
}

func (w *Writer) WriteBulkString(value []byte) error {
    return w.write(
        typeBulkString,
        []byte(strconv.FormatInt(int64(len(value)), 10)),
        separator,
        value,
        separator,
    )
}

// WriteNil writes a nil bulk string
func (w *Writer) WriteNil() error {
    return w.write(typeBulkString, []byte("-1"), separator)
}

func (w *Writer) WriteInt64(v int64) error {
    return w.write(
        typeInteger,
        []byte(strconv.FormatInt(v, 10)),
        separator)
}

// WriteArray writes an array that contains int8 to int64, strings, []byte, []interface{} or nil.
// Any other values inside the array will cause this method to return an error.
func (w *Writer) WriteArray(values []interface{}) error {
    if values == nil {
        return w.write(typeArray, []byte("-1"), separator)
    }

    if err := w.write(
        typeArray,
        []byte(strconv.FormatInt(int64(len(values)), 10)),
        separator,
    ); err != nil {
        return err
    }

    for _, v := range values {
        switch t := v.(type) {
        case int8:
            if err := w.WriteInt64(int64(t)); err != nil {
                return err
            }
        case int16:
            if err := w.WriteInt64(int64(t)); err != nil {
                return err
            }
        case int:
            if err := w.WriteInt64(int64(t)); err != nil {
                return err
            }
        case int32:
            if err := w.WriteInt64(int64(t)); err != nil {
                return err
            }
        case int64:
            if err := w.WriteInt64(t); err != nil {
                return err
            }
        case string:
            if err := w.WriteBulkString([]byte(t)); err != nil {
                return err
            }
        case []byte:
            if err := w.WriteBulkString(t); err != nil {
                return err
            }
        case []interface{}:
            if err := w.WriteArray(t); err != nil {
                return err
            }
        case nil:
            if err := w.WriteNil(); err != nil {
                return err
            }
        default:
            return fmt.Errorf("unsupported type: the value [%#v] is not supported by this client, supported types are int8 to int64, strings, []byte, nil, and []interface{} of these same types", v)
        }
    }

    return nil
}

Our writer is a bit strict, it only writes numbers from int8 to int64, strings and arrays with the previous values. We also don't bother with writing simple strings at all, we just assume all strings are always bulk strings as that allows us to ignore if there is a \r\n at all inside of them and just write them with the length directly. We also introduce methods to write null strings and null arrays as those are both valid values for redis.

The client

So we can read and write the Redis protocol from streams of bytes, the last step is to actually create a client that can open a socket connection given an address and talk to Redis, here's what it would look like:

package redis_client

import (
    "context"
    "github.com/pkg/errors"
    "io"
    "net"
    "time"
)

var (
    _ io.Closer = &Client{}
)

type Client struct {
    conn   net.Conn
    reader *Reader
    writer *Writer
}

func (c *Client) Close() error {
    return c.conn.Close()
}

func (c *Client) Send(values []interface{}) (*Result, error) {
    c.conn.SetDeadline(time.Now().Add(time.Second * 5))

    if err := c.writer.WriteArray(values); err != nil {
        return nil, errors.Wrapf(err, "failed to execute operation: %v", values[0])
    }

    return c.reader.Read()
}

func Connect(ctx context.Context, address string) (*Client, error) {
    dialer := net.Dialer{
        Timeout:   time.Second * 5,
        KeepAlive: time.Second * 10,
    }

    conn, err := dialer.DialContext(ctx, "tcp4", address)
    if err != nil {
        return nil, errors.Wrapf(err, "failed to connect to %v", address)
    }

    return &Client{
        conn:   conn,
        reader: NewReader(conn),
        writer: NewWriter(conn),
    }, nil
}

Given we've done most of the actual protocol parsing work at the Reader and Writer structs the only thing this client does is to create a dialer and open the TCP connection to Redis. The important pieces here are to make sure you setup a timeout on the dialer and a deadline on every operation. Here we're just hardcoding a 5 seconds deadline on every read/write but you can make this configurable, what is important here is to make sure that we do have timeouts/deadlines configured on every operations so we don't wait forever or until an error happens on the connection to give up the operation.

And with this you have a fully functional redis client that can send and receive any commands from the server. We built it by creating separate structs for the different operaitons we need (reading and writing) and when composed them into the actual client. By building them as separate pieces we can easily write unit tests for them (check the repo for the tests) and then assemble them into the actual client, this simplifies development and even allows us to build upon these as needed, we could, for instance, create a pooling client that holds onto multiple connections and the only piece that would change is the client itself, the reader and writer code would stay the same, thus requiring less code to be written. Now off you go building more clients for the apps you use!

Rate limiting HTTP requests in Go using Redis

Maurício Linhares — Thu, 30 Dec 2021 19:00:19 +0000

So you've created this awesome API, that offers a feature a lot of customers would be interested in using, but they're using it so much you can't handle the load in an effective way for everyone. While scaling up and making the service more reliable is an option, just doing that is not enough, the load might be uneven, the usage patterns might not be what your application was made to handle or you might just have limits you can't fix at this point in time, that's where rate-limiting comes in.

The idea behind rate limiting is that you're going to have a maximum amount of requests you'll be taking from a client and once that limit is reached, over a defined period, you'll start dropping requests until you reach the end of the period and restart the counter. For instance, clients are allowed to make 60 requests every minute, once they go over 60 requests you start denying the requests letting them know they're over their quota and need to wait to continue to have their requests served again.

The goal here is to make your service more reliable, rate-limiting is a protection you implement on your side to make sure bad actors or misconfigured clients can't take the whole service down or cause outages because the service is going over its expected usage limits. In an ideal scenario, you're not punishing good clients, as you have given them enough requests to do their usual work, but you'd block bad clients from wreaking havoc on your service.

You can find the whole source code for the project we'll be discussing below here on Github.

Building a rate-limiter

We'll have multiple implementations for the rate limiter to discuss the pros and cons of the different implementations, let's start then with the stuff that will be common across the implementations:

package redis_rate_limiter

import (
    "context"
    "time"
)

// Request defines a request that needs to be checked if it will be rate-limited or not.
// The `Key` is the identifier you're using for the client making calls. This could be a user/account ID if the user is
// signed into your application, the IP of the client making requests (this might not be reliable if you're not behind a
// proxy like Cloudflare, as clients can try to spoof IPs). The `Key` should be the same for multiple calls of the
// same client so we can correctly identify that this is the same app calling anywhere.
// `Limit` is the number of requests the client is allowed to make over the `Duration` period. If you set this to
// 100 and `Duration` to `1m` you'd have at most 100 requests over a minute.
type Request struct {
    Key      string
    Limit    uint64
    Duration time.Duration
}

// State is the result of evaluating the rate limit, either `Deny` or `Allow` a request.
type State int64

const (
    Deny  State = 0
    Allow       = 1
)

// Result represents the response to a check if a client should be rate-limited or not. The `State` will be either
// `Allow` or `Deny`, `TotalRequests` holds the number of requests this specific caller has already made over
// the current period and `ExpiresAt` defines when the rate limit will expire/roll over for clients that
// have gone over the limit.
type Result struct {
    State         State
    TotalRequests uint64
    ExpiresAt     time.Time
}

// Strategy is the interface the rate limit implementations must implement to be used, it takes a `Request` and
// returns a `Result` and an `error`, any errors the rate-limiter finds should be bubbled up so the code can make a
// decision about what it wants to do with the request.
type Strategy interface {
    Run(ctx context.Context, r *Request) (*Result, error)
}

A counter-based implementation

So we have the basics in place, what is the input, the output, and a small interface that needs to be implemented by every counter, let's look at the first counter, that just uses counters on Redis:

package redis_rate_limiter

import (
    "context"
    "github.com/go-redis/redis/v8"
    "github.com/pkg/errors"
    "time"
)

var (
    _ Strategy = &counterStrategy{}
)

const (
    keyWithoutExpire = -1
)

func NewCounterStrategy(client *redis.Client, now func() time.Time) *counterStrategy {
    return &counterStrategy{
        client: client,
        now:    now,
    }
}

type counterStrategy struct {
    client *redis.Client
    now    func() time.Time
}

// Run this implementation uses a simple counter with an expiration set to the rate limit duration.
// This implementation is functional but not very effective if you have to deal with bursty traffic as
// it will still allow a client to burn through its full limit quickly once the key expires.
func (c *counterStrategy) Run(ctx context.Context, r *Request) (*Result, error) {
    // a pipeline in Redis is a way to send multiple commands that will all be run together.
    // this is not a transaction and there are many ways in which these commands could fail
    // (only the first, only the second) so we have to make sure all errors are handled, this
    // is a network performance optimization.

    p := c.client.Pipeline()
    incrResult := p.Incr(ctx, r.Key)
    ttlResult := p.TTL(ctx, r.Key)

    if _, err := p.Exec(ctx); err != nil {
        return nil, errors.Wrapf(err, "failed to execute increment to key %v", r.Key)
    }

    totalRequests, err := incrResult.Result()
    if err != nil {
        return nil, errors.Wrapf(err, "failed to increment key %v", r.Key)
    }

    var ttlDuration time.Duration

    // we want to make sure there is always an expiration set on the key, so on every
    // increment we check again to make sure it has a TTl and if it doesn't we add one.
    // a duration of -1 means that the key has no expiration so we need to make sure there
    // is one set, this should, most of the time, happen when we increment for the
    // first time but there could be cases where we fail at the previous commands so we should
    // check for the TTL on every request.
    if d, err := ttlResult.Result(); err != nil || d == keyWithoutExpire {
        ttlDuration = r.Duration
        if err := c.client.Expire(ctx, r.Key, r.Duration).Err(); err != nil {
            return nil, errors.Wrapf(err, "failed to set an expiration to key %v", r.Key)
        }
    } else {
        ttlDuration = d
    }

    expiresAt := c.now().Add(ttlDuration)

    requests := uint64(totalRequests)

    if requests > r.Limit {
        return &Result{
            State:         Deny,
            TotalRequests: requests,
            ExpiresAt:     expiresAt,
        }, nil
    }

    return &Result{
        State:         Allow,
        TotalRequests: requests,
        ExpiresAt:     expiresAt,
    }, nil
}

This implementation is what would usually be called a fixed window strategy, it means that once the expiration has been set, a client that reaches the limit will be blocked from making further requests until the expiration time arrives. If a client has a limit of 50 requests every minute and makes all 50 requests in the first 5 seconds of the minute, it will have to wait 55 seconds to make another request. This is also the main downside of this implementation, it would still let a client burn through its whole limit quickly (bursty traffic) and that could still overload your service, as it could be expecting this traffic to be spread out throughout the whole limiting period.

A sorted set implementation to react better to bursty traffic

A rolling window strategy offers better protection for bursty traffic as it doesn't reset the counter completely but keeps the history for the duration of the time window. If you have a 5 minutes window, it will always count the amount of traffic that was generated over the last 5 minutes to decide if a client should be blocked or not instead of just waiting for a key to expire. This implementation is more CPU and memory intensive as you have to keep more information in memory(every request with its timestamp) but provides better protection against quick bursts of traffic.

Here's what it looks like:

package redis_rate_limiter

import (
    "context"
    "github.com/go-redis/redis/v8"
    "github.com/google/uuid"
    "github.com/pkg/errors"
    "strconv"
    "time"
)

var (
    _ Strategy = &sortedSetCounter{}
)

func NewSortedSetCounterStrategy(client *redis.Client, now func() time.Time) Strategy {
    return &sortedSetCounter{
        client: client,
        now:    now,
    }
}

type sortedSetCounter struct {
    client *redis.Client
    now    func() time.Time
}

// Run this implementation uses a sorted set that holds a UUID for every request with a score that is the
// time the request has happened. This allows us to delete events from *before* the current window if the window
// is 5 minutes, we want to remove all events from before 5 minutes ago, this way we make sure we roll old
// requests off the counters creating a rolling window for the rate limiter. So, if your window is 100 requests
// in 5 minutes and you spread the load evenly across the minutes, once you hit 6 minutes the requests you made
// on the first minute have now expired but the other 4 minutes of requests are still valid.
// A rolling window counter is usually never 0 if traffic is consistent so it is very effective at preventing
// bursts of traffic as the counter won't ever expire.
func (s *sortedSetCounter) Run(ctx context.Context, r *Request) (*Result, error) {
    now := s.now()
    // every request needs an UUID
    item := uuid.New()

    minimum := now.Add(-r.Duration)

    p := s.client.Pipeline()

    // we then remove all requests that have already expired on this set
    removeByScore := p.ZRemRangeByScore(ctx, r.Key, "0", strconv.FormatInt(minimum.UnixMilli(), 10))

    // we add the current request
    add := p.ZAdd(ctx, r.Key, &redis.Z{
        Score:  float64(now.UnixMilli()),
        Member: item.String(),
    })

    // count how many non-expired requests we have on the sorted set
    count := p.ZCount(ctx, r.Key, "-inf", "+inf")

    if _, err := p.Exec(ctx); err != nil {
        return nil, errors.Wrapf(err, "failed to execute sorted set pipeline for key: %v", r.Key)
    }

    if err := removeByScore.Err(); err != nil {
        return nil, errors.Wrapf(err, "failed to remove items from key %v", r.Key)
    }

    if err := add.Err(); err != nil {
        return nil, errors.Wrapf(err, "failed to add item to key %v", r.Key)
    }

    totalRequests, err := count.Result()
    if err != nil {
        return nil, errors.Wrapf(err, "failed to count items for key %v", r.Key)
    }

    expiresAt := now.Add(r.Duration)
    requests := uint64(totalRequests)

    if requests > r.Limit {
        return &Result{
            State:         Deny,
            TotalRequests: requests,
            ExpiresAt:     expiresAt,
        }, nil
    }

    return &Result{
        State:         Allow,
        TotalRequests: requests,
        ExpiresAt:     expiresAt,
    }, nil
}

The goal here is to use the sorted set with the current timestamp for the request as the sorting key so we can quickly ask for all requests under a specific time range to be deleted, cleaning up requests that were already expired. As we're using an UUID as the value the odds we'll have these requests conflict with each other on the sorted set are very low and having a conflict every once in a while wouldn't be too much trouble, even with millions of requests being served.

Write heavy solutions

Both implementations here are write first, they first write stuff into Redis and then check if the request should be accepted or not. While this makes the code more straightforward and produces less network traffic from the client to the server, they make it much more expensive for the server to operate, as it is always writing something and changing data in memory, even unbounded data, as in the sorted set case. We're not making sure these sorted sets are cleaned up or that there is a limit on how many requests can be stored and this is a general red flag you should not let happen on the systems you're working on.

We have to make sure the resource usage here is bounded, that is, it has a well known limit, you should always assume anything you're writing is going to be abused in some way (either intentionally or by accident) and make sure that your code forces a bound on these resources in some way. On Redis itself, there are many ways to make sure you're not going to run out of memory by setting a maxmemory value for how much memory it can use and maxmemory-policy to a value that is not no-eviction. For a rate limiter like the one we're building here allkeys-lru is a pretty decent option.

Bounded counter implementations

The updated counter implementation with a read before write would look like this:

package redis_rate_limiter

import (
    "context"
    "github.com/go-redis/redis/v8"
    "github.com/pkg/errors"
    "time"
)

var (
    _ Strategy = &counterStrategy{}
)

const (
    keyThatDoesNotExist = -2
    keyWithoutExpire    = -1
)

func NewCounterStrategy(client *redis.Client, now func() time.Time) *counterStrategy {
    return &counterStrategy{
        client: client,
        now:    now,
    }
}

type counterStrategy struct {
    client *redis.Client
    now    func() time.Time
}

// Run this implementation uses a simple counter with an expiration set to the rate limit duration.
// This implementation is functional but not very effective if you have to deal with bursty traffic as
// it will still allow a client to burn through its full limit quickly once the key expires.
func (c *counterStrategy) Run(ctx context.Context, r *Request) (*Result, error) {

    // a pipeline in redis is a way to send multiple commands that will all be run together.
    // this is not a transaction and there are many ways in which these commands could fail
    // (only the first, only the second) so we have to make sure all errors are handled, this
    // is a network performance optimization.

    // here we try to get the current value and also try to set an expiration on it
    getPipeline := c.client.Pipeline()
    getResult := getPipeline.Get(ctx, r.Key)
    ttlResult := getPipeline.TTL(ctx, r.Key)

    if _, err := getPipeline.Exec(ctx); err != nil && !errors.Is(err, redis.Nil) {
        return nil, errors.Wrapf(err, "failed to execute pipeline with get and ttl to key %v", r.Key)
    }

    var ttlDuration time.Duration

    // we want to make sure there is always an expiration set on the key, so on every
    // increment we check again to make sure it has a TTl and if it doesn't we add one.
    // a duration of -1 means that the key has no expiration so we need to make sure there
    // is one set, this should, most of the time, happen when we increment for the
    // first time but there could be cases where we fail at the previous commands so we should
    // check for the TTL on every request.
    // a duration of -2 means that the key does not exist, given we're already here we should set an expiration
    // to it anyway as it means this is a new key that will be incremented below.
    if d, err := ttlResult.Result(); err != nil || d == keyWithoutExpire || d == keyThatDoesNotExist {
        ttlDuration = r.Duration
        if err := c.client.Expire(ctx, r.Key, r.Duration).Err(); err != nil {
            return nil, errors.Wrapf(err, "failed to set an expiration to key %v", r.Key)
        }
    } else {
        ttlDuration = d
    }

    expiresAt := c.now().Add(ttlDuration)

    if total, err := getResult.Uint64(); err != nil && errors.Is(err, redis.Nil) {

    } else if total >= r.Limit {
        return &Result{
            State:         Deny,
            TotalRequests: total,
            ExpiresAt:     expiresAt,
        }, nil
    }

    incrResult := c.client.Incr(ctx, r.Key)

    totalRequests, err := incrResult.Uint64()
    if err != nil {
        return nil, errors.Wrapf(err, "failed to increment key %v", r.Key)
    }

    if totalRequests > r.Limit {
        return &Result{
            State:         Deny,
            TotalRequests: totalRequests,
            ExpiresAt:     expiresAt,
        }, nil
    }

    return &Result{
        State:         Allow,
        TotalRequests: totalRequests,
        ExpiresAt:     expiresAt,
    }, nil
}

Now we read before we increment, making sure we only increment if this is a good request that will be allowed. Otherwise just bail and deny the request before trying anything.

Now we'll look at the updated sorted set implementation:

package redis_rate_limiter

import (
    "context"
    "github.com/go-redis/redis/v8"
    "github.com/google/uuid"
    "github.com/pkg/errors"
    "strconv"
    "time"
)

var (
    _ Strategy = &sortedSetCounter{}
)

const (
    sortedSetMax = "+inf"
    sortedSetMin = "-inf"
)

func NewSortedSetCounterStrategy(client *redis.Client, now func() time.Time) Strategy {
    return &sortedSetCounter{
        client: client,
        now:    now,
    }
}

type sortedSetCounter struct {
    client *redis.Client
    now    func() time.Time
}

// Run this implementation uses a sorted set that holds an UUID for every request with a score that is the
// time the request has happened. This allows us to delete events from *before* the current window, if the window
// is 5 minutes, we want to remove all events from before 5 minutes ago, this way we make sure we roll old
// requests off the counters creating a rolling window for the rate limiter. So, if your window is 100 requests
// in 5 minutes and you spread the load evenly across the minutes, once you hit 6 minutes the requests you made
// on the first minute have now expired but the other 4 minutes of requests are still valid.
// A rolling window counter is usually never 0 if traffic is consistent so it is very effective at preventing
// bursts of traffic as the counter won't ever expire.
func (s *sortedSetCounter) Run(ctx context.Context, r *Request) (*Result, error) {
    now := s.now()
    expiresAt := now.Add(r.Duration)
    minimum := now.Add(-r.Duration)

    // first count how many requests over the period we're tracking on this rolling window so check wether
    // we're already over the limit or not. this prevents new requests from being added if a client is already
    // rate limited, not allowing it to add an infinite amount of requests to the system overloading redis.
    // if the client continues to send requests it also means that the memory for this specific key will not
    // be reclaimed (as we're not writing data here) so make sure there is an eviction policy that will
    // clear up the memory if the redis starts to get close to its memory limit.
    result, err := s.client.ZCount(ctx, r.Key, strconv.FormatInt(minimum.UnixMilli(), 10), sortedSetMax).Uint64()
    if err == nil && result >= r.Limit {
        return &Result{
            State:         Deny,
            TotalRequests: result,
            ExpiresAt:     expiresAt,
        }, nil
    }

    // every request needs an UUID
    item := uuid.New()

    p := s.client.Pipeline()

    // we then remove all requests that have already expired on this set
    removeByScore := p.ZRemRangeByScore(ctx, r.Key, "0", strconv.FormatInt(minimum.UnixMilli(), 10))

    // we add the current request
    add := p.ZAdd(ctx, r.Key, &redis.Z{
        Score:  float64(now.UnixMilli()),
        Member: item.String(),
    })

    // count how many non-expired requests we have on the sorted set
    count := p.ZCount(ctx, r.Key, sortedSetMin, sortedSetMax)

    if _, err := p.Exec(ctx); err != nil {
        return nil, errors.Wrapf(err, "failed to execute sorted set pipeline for key: %v", r.Key)
    }

    if err := removeByScore.Err(); err != nil {
        return nil, errors.Wrapf(err, "failed to remove items from key %v", r.Key)
    }

    if err := add.Err(); err != nil {
        return nil, errors.Wrapf(err, "failed to add item to key %v", r.Key)
    }

    totalRequests, err := count.Result()
    if err != nil {
        return nil, errors.Wrapf(err, "failed to count items for key %v", r.Key)
    }

    requests := uint64(totalRequests)

    if requests > r.Limit {
        return &Result{
            State:         Deny,
            TotalRequests: requests,
            ExpiresAt:     expiresAt,
        }, nil
    }

    return &Result{
        State:         Allow,
        TotalRequests: requests,
        ExpiresAt:     expiresAt,
    }, nil
}

Now before we add a request to the redis server we check whether there have been too many requests for the period already and if there were we just deny the request before adding a new one to Redis.

Integrating it as an HTTP middleware

Now that we have implemented two separate counters, how do we make use of them?

One of the ways of doing it is by creating a middleware handler that wraps an existing HTTP handler and adds the rate limiting capability to it so we can compose the handlers with rate limiting when it's needed or just leave them without it if we'd like to, here's what it looks like:

package redis_rate_limiter

import (
    "fmt"
    "net/http"
    "strconv"
    "strings"
    "time"
)

var (
    _            http.Handler = &httpRateLimiterHandler{}
    _            Extractor    = &httpHeaderExtractor{}
    stateStrings              = map[State]string{
        Allow: "Allow",
        Deny:  "Deny",
    }
)

const (
    rateLimitingTotalRequests = "Rate-Limiting-Total-Requests"
    rateLimitingState         = "Rate-Limiting-State"
    rateLimitingExpiresAt     = "Rate-Limiting-Expires-At"
)

// Extractor represents the way we will extract a key from an HTTP request, this could be
// a value from a header, request path, method used, user authentication information, any information that
// is available at the HTTP request that wouldn't cause side effects if it was collected (this object shouldn't
// read the body of the request).
type Extractor interface {
    Extract(r *http.Request) (string, error)
}

type httpHeaderExtractor struct {
    headers []string
}

// Extract extracts a collection of http headers and joins them to build the key that will be used for
// rate limiting. You should use headers that are guaranteed to be unique for a client.
func (h *httpHeaderExtractor) Extract(r *http.Request) (string, error) {
    values := make([]string, 0, len(h.headers))

    for _, key := range h.headers {
        // if we can't find a value for the headers, give up and return an error.
        if value := strings.TrimSpace(r.Header.Get(key)); value == "" {
            return "", fmt.Errorf("the header %v must have a value set", key)
        } else {
            values = append(values, value)
        }
    }

    return strings.Join(values, "-"), nil
}

// NewHTTPHeadersExtractor creates a new HTTP header extractor
func NewHTTPHeadersExtractor(headers ...string) Extractor {
    return &httpHeaderExtractor{headers: headers}
}

// RateLimiterConfig holds the basic config we need to create a middleware http.Handler object that
// performs rate limiting before offloading the request to an actual handler.
type RateLimiterConfig struct {
    Extractor   Extractor
    Strategy    Strategy
    Expiration  time.Duration
    MaxRequests uint64
}

// NewHTTPRateLimiterHandler wraps an existing http.Handler object performing rate limiting before
// sending the request to the wrapped handler. If any errors happen while trying to rate limit a request
// or if the request is denied, the rate limiting handler will send a response to the client and will not
// call the wrapped handler.
func NewHTTPRateLimiterHandler(originalHandler http.Handler, config *RateLimiterConfig) http.Handler {
    return &httpRateLimiterHandler{
        handler: originalHandler,
        config:  config,
    }
}

type httpRateLimiterHandler struct {
    handler http.Handler
    config  *RateLimiterConfig
}

func (h *httpRateLimiterHandler) writeRespone(writer http.ResponseWriter, status int, msg string, args ...interface{}) {
    writer.Header().Set("Content-Type", "text/plain")
    writer.WriteHeader(status)
    if _, err := writer.Write([]byte(fmt.Sprintf(msg, args...))); err != nil {
        fmt.Printf("failed to write body to HTTP request: %v", err)
    }
}

// ServeHTTP performs rate limiting with the configuration it was provided and if there were not errors
// and the request was allowed it is sent to the wrapped handler. It also adds rate limiting headers that will be
// sent to the client to make it aware of what state it is in terms of rate limiting.
func (h *httpRateLimiterHandler) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
    key, err := h.config.Extractor.Extract(request)
    if err != nil {
        h.writeRespone(writer, http.StatusBadRequest, "failed to collect rate limiting key from request: %v", err)
        return
    }

    result, err := h.config.Strategy.Run(request.Context(), &Request{
        Key:      key,
        Limit:    h.config.MaxRequests,
        Duration: h.config.Expiration,
    })

    if err != nil {
        h.writeRespone(writer, http.StatusInternalServerError, "failed to run rate limiting for request: %v", err)
        return
    }

    // set the rate limiting headers both on allow or deny results so the client knows what is going on
    writer.Header().Set(rateLimitingTotalRequests, strconv.FormatUint(result.TotalRequests, 10))
    writer.Header().Set(rateLimitingState, stateStrings[result.State])
    writer.Header().Set(rateLimitingExpiresAt, result.ExpiresAt.Format(time.RFC3339))

    // when the state is Deny, just return a 429 response to the client and stop the request handling flow
    if result.State == Deny {
        h.writeRespone(writer, http.StatusTooManyRequests, "you have sent too many requests to this service, slow down please")
        return
    }

    // if the request was not denied we assume it was allowed and call the wrapped handler.
    // by leaving this to the end we make sure the wrapped handler is only called once and doesn't have to worry
    // about any rate limiting at all (it doesn't even have to know there was rate limiting happening for this request)
    // as we have already set the headers, so when the handler flushes the response the headers above will be sent.
    h.handler.ServeHTTP(writer, request)
}

So what do we have here? We start with the Extractor interface that returns a string to be used as the key to a rate limiting check. You could have multiple implementations of this interface, checking HTTP headers or any other fields available on the HTTP request to identify the client, the best way to use this is to have a user/account ID that can be pulled from cookies or headers as IPs change with some frequency, so implement a solution that makes sense for your application.

Then we have the RateLimiterConfig struct that wraps all fields needed to perform rate limiting on requests, an Extractor, a Strategy (our counters), how many requests the client can make and for how long. With all this and an http.Handler to wrap we have a fully functional HTTP rate limiter middleware and you could build the same kind of middleware for any other request/response protocol using this same pattern, just changing how this handler is built.

What should I worry about?

First, as we mentioned before, bursty traffic. Clients that use up their whole limit too fast are still going to be an issue for your systems and while the rolling window limiter diminishes this a little bit it should not be the only thing you're doing to prevent thundering herds of traffic. Another possible solution is to use smaller durations, so instead of giving clients 10_000 requests every hour, give them 160 requests every minute, this way the worst they could do is 160 requests in a short period of time instead of 10_000.

On the same note, do not set limits for very long periods of time, like 100 requests every day, as that makes using the API very frustrating, you make 100 requests and you now have to wait a whole day for the limit to expire again, use smaller periods so people can spread out the load more evenly.

Dynamic deny lists that completely block traffic from specific clients are also a must have on a system like this. You could make this a configuration you'd use on a proxy that sits before the app that performs rate limiting (like a Nginx/Apache server)or at a CDN you're using (like Cloudflare) so you can quickly blackhole all traffic from known abusers.

The implementation we have build here is a fail closed solution, which means any error (talking to redis, running extractors)causes the request to be denied. This is mostly because a paranoid implementation is generally safer but might not be the best solution for your service, you might want to have a best effort solution that fails open, once it finds a failure somewhere during rate limiting it lets the request happen.

While this might be more user friendly it also adds to the risk that abuses could overload the rate limiter and then overload the systems behind it because they now have no protection with the rate limiter down. So before moving completely to a fail open solution make sure you have other mitigation measures if the rate limiter fails and there is monitoring on all pieces of the stack, both the app and the Redis servers, so you're notified quickly if any of them start failing and letting on traffic. You might even add an extra HTTP header to downstream clients to let them know that rate limiting failed but the request was still sent so they can make a decision if they want to accept that request or not.

Again, your main goal with rate limiting should be protecting your systems so that you're providing the best service possible for most users and preventing that a small number of misbehaving clients wreak havoc on your applications preventing good users from using them.