My favourite backend tools

I spend a lot of my time as a software developer writing backend services and infrastructure and wanted to share some of the tools I use most regularly.

I'm always in the market for new tools, so please feel free to add your favourite tools in the comments!

golang

awesome

Go is a fantastic language for web server development. It's standard library is very comprehensive, meaning you can do pretty much everything you'd want to do with just the standard library (although there are plenty of packages that have already solved a lot of common challenges).

Spinning up a web server is as simple as:

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/hello", hello)

    log.Fatal(http.ListenAndServe(":8080", nil))
}

func hello(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintln(w, "Hello, World!")
}

...although you should not use http.ListenAndServe when developing applications destined for production, as it does not configure sensible timeout defaults.

Go's tooling is as comprehensive as its standard library, meaning you can test, benchmark, profile, and generate coverage of your code just as easily as you wrote it.

crystal

awesome

Crystal is another great programming language for backend development. It's very concise, very declarative, and very fast.

Crystal allows a developer to write very terse code to achieve what would typically be much more verbose in other programming languages. For example, here's a web service handler using Crystal’s most well-known web framework, kemal. It accepts the id of a person from the URL path, queries a database for them using their is, and returns their first and last name as JSON:

get "/people/:id" do |env|
  id = env.params.url["id"]
  first, last = db.query_one "select first_name, last_name from person where id = $1 limit 1", id, as: { String, String }

  {"firstName": first, "lastName": last}.to_json
end

replit

Replit started out as an online IDE but thanks to the team's ridiculous efficiency in getting out new releases, it's become a completely game-changing platform.

With Replit, you can create websites, web services, games, and more, in any language you can think of (including esoteric ones). I use Replit to host long-running websites with custom domain names; all linked to GitHub. Everything's super easy and I think the world of online hosting is going to be disrupted by this awesome website.

benthos

Benthos is a declarative stream processor that makes mundane tasks fun and allows you to achieve ridiculous levels of productivity. It's under active development with new components (inputs, processors, and outputs etc.) being added regularly.

It can run anywhere and can be configured statically with YAML files or dynamically with Streams Mode. The following example consumes from 2 Kafka topics and writes messages from both into a Cassandra table:

input:
  kafka:
    addresses:
      - localhost:9092
    topics:
      - topic_one
      - topic_two
    consumer_group: my_benthos_app

output:
  cassandra:
    addresses:
      - localhost:9042
    query: INSERT INTO my_keyspace.my_table (id, event, created_at) VALUES (?, ?, ?)
    args:
      - ${! json("id") }
      - ${! json("event") }
      - ${! json("timestamp").format_timestamp() }

kubernetes

Kubernetes is a tool that's synonymous with the deployment of scalable and reliable application workloads. It's a huge tool that'll likely offer more than most people will need but it's very powerful, has a huge community, and allows you to quickly and reliably scale and serve your applications.

I use Kubernetes to run all sorts of application workloads and regularly make use of its ability to integrate with cloud providers to transparently create things like load balancers.

nomad

Nomad is an awesome alternative to Kubernetes. It's hugely scalable, allowing you to schedule many different types of workload including Docker and Podman containers, raw executables, and Java applications (to name a few).

Unlike Kubernetes, applications can (and should) be scheduled onto machines that are not part of the orchestration cluster nodes. This allows Nomad to scale to millions of running applications, whereas Kubernetes best practices recommend a limit of 150,000 pods and 300,000 containers.

I use Nomad to orchestrate IoT workloads but it can be used for many more use cases.

wrk / hey

wrk and hey are amazing HTTP benchmarking CLIs. They allow you to write very concise commands to benchmark web services, offering higher speeds and concurrency levels than alternative tools.

wrk is the fastest available (by a large margin) and can be scripted with Lua. Although it doesn't run on Windows without the use of WSL (Windows Subsystem for Linux). I tend to use wrk when I'm on my Mac and hey when I'm on Windows machines.

k6

k6 is a fantastic Open Source load testing tool that can be scripted with JavaScript.

You can configure very complex load testing scenarios with large numbers of VUs (Virtual Users) to put your web services under realistic smoke, load, stress, and soak scenarios.

I use k6 for anything more complicated than simply hammering a single endpoint.

terraform

Terraform is a de-facto standard for deploying infrastructure with code. It has official support for many cloud providers and services and can be extended with custom providers depending on your needs.

I use Terraform for managing infrastructure. It's not without its issues but once you're aware of them, it's a very productive tool that frequently leaves me in awe.

prometheus / grafana

Prometheus is a tool for collecting metric data from applications. You expose an endpoint from your application (e.g. /metrics), point Prometheus at it, and simply write metrics; all using the Prometheus client libraries that are available for most major programming languages.

Grafana is a great tool for graphing metrics. It can read data from lots of different metric/time-based backends (including Prometheus and InfluxDB etc.) and can be configured to raise alerts.

ngrok

ngrok creates "secure introspection tunnels to localhost", which - put simply - allows you to expose your machine to the outside world, even if it's behind a firewall.

The following command will expose any application running on port 8080 on your machine to the outside world. It will generate 2 unique URLs (one for HTTP and one for HTTPS):

$ ngrok http 8080

I use ngrok for testing local services against external services like applications that publish web hooks.

postman

Postman is a tool for testing web servers. It's feature-rich and allows you to test HTTP and GraphQL services with minimal effort.

You can also use Postman as a documentation and code-generation tool (create a request, then generate code for lots of different tools and programming languages).

I use Postman for testing both HTTP and GraphQL services, saving collections of request for each service for reproducible tests.

k9s

k9s is an awesome CLI for interacting with Kubernetes.

With k9s you can view pods, jump into their shells, view their logs, port forward, and delete them. You also have a lot of options for managing deployments, stateful sets, services and many other Kubernetes constructs.

grpcurl / grpcui

grpcurl and grpcui are great tools for working with gRPC services. To start, simply create a gRPC service with reflection enabled and open either of the tools. You'll now be able to call the endpoints just like you would with tools like cURL or Postman.

table plus

TablePlus is a great tool for working with a lot of database engines including Postgres, CockroachDB, MySQL, SQLServer, Cassandra, MongoDB, Redis, and more.

Having the ability to work with so many data stores makes developing any kind of database-backed service very straightforward.

datagen

A tool from me

datagen is a tool for generating lots of random relational data quickly. You write a template file using a mixture of SQL and Go templates to create multi-row DML (Data Manipulation Language), or more simply, batches of INSERT statements.

Outputs from one datagen insert (e.g, the output from a RETURNING or a follow-up SELECT statement) can be used as the input for the next, allowing you to build up related data across multiple tables.

I use datagen for generating monstrous amounts of data for testing with while developing, as the best time to find issues with database performance and integration is during development!

setcfg

A tool from me

setcfg allows you to substitute values in one YAML file with values from another YAML file like so:

Input file:

region: ~region~

credentials:
    username: ~username~
    password: ~password~

brokers:
    - broker: ~broker-1~
    - broker: ~broker-2~

subnet_cidrs: ~subnet-cidrs~

Config file:

region: eu-west-1

username: admin
password: supersecret

broker-1:
    host: https://localhost
    port: 8001

broker-2:
    host: https://localhost
    port: 8002

subnet-cidrs:
- 1.2.3.0/25
- 1.2.3.128/25

Running setcfg with these two files (setcfg -i input.yaml -e config.yaml) will result in the following, which can be piped into a file or a kubectl command, depending on the type of input file you’re using:

brokers:
- broker:
    host: https://localhost
    port: 8001
- broker:
    host: https://localhost
    port: 8002
credentials:
  password: supersecret
  username: admin
region: eu-west-1
subnet_cidrs:
- 1.2.3.0/25
- 1.2.3.128/25

I use this for working with things like Kubernetes manifests and Benthos configuration files, although it'll happily work with any YAML file.

pa55

A tool from me

pa55 is a simple CLI for generating cryptographically secure passwords and copying them to the clipboard. It can generate passwords of any length and output them in ascii, base32, base64, and hexadecimal.

I use pa55 for generating passwords for session keys, and SaaS solutions.

see

A tool from me

see is a simple CLI that does what the Linux watch command does but for any platform.

Conclusion

A great tool can not only help you to become more efficient but can also help to make mundane tasks fun. If you find a tool that you love, make sure to support it with a star on GitHub and possibly a contribution!