DEV Community: Dhwaneet Bhatt

LocalStack S3: Persistent local development

Dhwaneet Bhatt — Wed, 22 Apr 2026 13:59:13 +0000

Building with AWS S3? You don't want to hit real AWS while developing. LocalStack gives you a full S3 instance running locally in Docker — the catch is that the community edition doesn't persist data across restarts. This post walks through a setup that does: Docker Compose + a simple backup script + an init hook to restore on startup. No paid plan needed.

Getting started

You'll need:

Docker and Docker Compose
AWS CLI (brew install awscli on macOS)

Add LocalStack to your docker-compose.yaml:

services:
  localstack:
    image: localstack/localstack:4.4.0
    container_name: localstack
    restart: unless-stopped
    environment:
      - SERVICES=s3
      - AWS_DEFAULT_REGION=us-east-1
      - LOCALSTACK_HOST=localstack
      - LOCALSTACK_SKIP_SSL_CERT_DOWNLOAD=1
    ports:
      - "4566:4566"
    volumes:
      - localstack_data:/var/lib/localstack
      - /var/run/docker.sock:/var/run/docker.sock
      - ./localstack/init/ready.d:/etc/localstack/init/ready.d
      - ./localstack-backup/s3:/localstack-backup/s3

volumes:
  localstack_data:
    driver: local

A few notes on this config:

Pin the image version — never use latest. If LocalStack ships a breaking change, latest will break your dev setup.
LOCALSTACK_SKIP_SSL_CERT_DOWNLOAD=1 — LocalStack tries to fetch a certificate from its API on startup. If your container has no internet access (or the download fails), this gets noisy. The flag skips it and uses a self-signed cert instead.
SERVICES=s3 — only start S3, not the whole suite. Faster startup.

Bring it up:

docker compose up -d localstack

Create a bucket and test:

aws --endpoint-url=http://localhost:4566 s3 mb s3://my-bucket --region us-east-1
aws --endpoint-url=http://localhost:4566 s3 ls

The persistence problem

Community LocalStack doesn't persist data across restarts. PERSISTENCE=1 and snapshots are Pro-only. Restart the container and your buckets are gone.

The fix: sync your S3 data to the host with a backup script, and restore it on startup with a LocalStack init hook.

The backup script

Create scripts/backup-s3.sh:

#!/usr/bin/env bash
set -euo pipefail

AWS="/opt/homebrew/bin/aws"   # adjust to your `which aws` path
ENDPOINT="http://localhost:4566"
BACKUP_DIR="$(cd "$(dirname "$0")/.." && pwd)/localstack-backup/s3"

# Check if LocalStack is reachable
if ! "$AWS" --endpoint-url="$ENDPOINT" s3 ls &>/dev/null; then
    echo "$(date): LocalStack not reachable, skipping backup"
    exit 0
fi

BUCKETS=$("$AWS" --endpoint-url="$ENDPOINT" s3 ls 2>/dev/null | awk '{print $3}')

if [ -z "$BUCKETS" ]; then
    echo "$(date): No buckets found, nothing to back up"
    exit 0
fi

for BUCKET in $BUCKETS; do
    mkdir -p "$BACKUP_DIR/$BUCKET"
    "$AWS" --endpoint-url="$ENDPOINT" s3 sync "s3://$BUCKET" "$BACKUP_DIR/$BUCKET" --delete --quiet
    echo "$(date): Backed up s3://$BUCKET -> $BACKUP_DIR/$BUCKET"
done

Make it executable:

chmod +x scripts/backup-s3.sh

Run it manually to verify:

./scripts/backup-s3.sh

Automate backups with cron

Add a cron job to back up every 5 minutes:

(crontab -l 2>/dev/null; echo "*/5 * * * * /path/to/local-infra/scripts/backup-s3.sh >> /path/to/local-infra/localstack-backup/backup.log 2>&1") | crontab -

Use absolute paths — cron has a minimal environment and won't expand relative paths. Logs go to localstack-backup/backup.log so you can see what's happening.

The init hook

LocalStack runs scripts in /etc/localstack/init/ready.d/ after the service is ready. We'll use this to restore the backup on startup.

Create localstack/init/ready.d/restore-s3.sh:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_DIR="/localstack-backup/s3"

if [ ! -d "$BACKUP_DIR" ] || [ -z "$(ls -A "$BACKUP_DIR" 2>/dev/null)" ]; then
    echo "No S3 backup found, skipping restore"
    exit 0
fi

for BUCKET_DIR in "$BACKUP_DIR"/*/; do
    [ -d "$BUCKET_DIR" ] || continue
    BUCKET=$(basename "$BUCKET_DIR")
    awslocal s3 mb "s3://$BUCKET" 2>/dev/null || true
    awslocal s3 sync "$BUCKET_DIR" "s3://$BUCKET" --quiet
    echo "Restored s3://$BUCKET"
done

echo "S3 restore complete"

chmod +x localstack/init/ready.d/restore-s3.sh

The script uses awslocal (built into LocalStack) which points to localhost:4566 automatically — no endpoint flag needed. The || true on s3 mb keeps the script from failing if the bucket already exists.

Putting it together

local-infra/
├── docker-compose.yaml
├── scripts/
│   └── backup-s3.sh          # runs via cron every 5 min on the host
├── localstack/
│   └── init/
│       └── ready.d/
│           └── restore-s3.sh # runs inside container on every startup
└── localstack-backup/
    ├── backup.log
    └── s3/
        └── my-bucket/        # backed-up objects live here
            └── ...

The two volume mounts in docker-compose.yaml are what connect everything:

- ./localstack/init/ready.d:/etc/localstack/init/ready.d   # init hook
- ./localstack-backup/s3:/localstack-backup/s3             # shared backup dir

How it works

# Start everything
docker compose up -d localstack

# → LocalStack starts → restore-s3.sh runs → your buckets and objects are back

# Work with S3 normally
aws --endpoint-url=http://localhost:4566 s3 cp file.txt s3://my-bucket/

# Backup runs every 5 min automatically (or manually)
./scripts/backup-s3.sh

# Safe to restart — data comes back
docker compose restart localstack

Trade-offs

Data loss window — if LocalStack crashes mid-write before the next cron run, you lose up to 5 minutes of work. Fine for dev.
Large files — big S3 objects slow down sync. You can increase the cron interval or use --exclude for test fixtures.
When to upgrade — if you need real persistence, fine-grained IAM, or more AWS services, LocalStack Pro is worth it.

That's it

~50 lines of bash, two volume mounts, and you have persistent local S3. Not production-grade, but solid enough for development.

strapd: CLI/webapp for everyday micro tasks

Dhwaneet Bhatt — Tue, 23 Dec 2025 07:24:59 +0000

Developers often run into small, repetitive tasks that interrupt flow—transforming strings, generating identifiers, formatting data, encoding or decoding values, or converting timestamps. These tasks are usually too small to justify writing a script, but annoying enough to slow you down when done manually.

strapd is a CLI (and web app) designed to handle these kinds of micro-tasks in one place. The CLI is intentionally forgiving: it supports multiple aliases and interchangeable subcommands, so you don’t have to remember a single “correct” syntax. The goal is to make common operations feel natural in the terminal, with minimal cognitive overhead.

All of the following commands do the same thing:

strapd text upper "abc"
strapd str upper "abc"
strapd string uc "abc"

The same flexibility applies to piped workflows:

cat input.json | strapd json format --sort
cat input.json | strapd json sort --format

This approach keeps the focus on speed and flow rather than memorization, and works well alongside existing shell habits and aliases.

Instead of reaching for ad-hoc scripts, online tools, or custom shell aliases, strapd provides a consistent interface for common operations developers encounter in day-to-day work.

What strapd does

String tools: case conversion, trim, slugify, reverse, replace, analysis
Identifiers: UUID (v4, v7), ULID
Encoding / decoding: Base64, URL, Hex
Data formatting: JSON, YAML, XML, SQL (beautify, minify, sort)
Format conversion: YAML ⇄ JSON
Security utilities: hashing (MD5, SHA-1, SHA-256, SHA-512), HMAC
Random generators: numbers, strings
Date/time: timestamps
Clipboard utilities: copy & paste (CLI only)

The focus is on tasks that are small, frequent, and easy to forget the exact syntax for—but useful to have immediately available.

Getting started

GitHub (open source):
https://github.com/dhwaneetbhatt/strapd

Web app:
https://dhwaneetbhatt.com/strapd

CLI docs:
https://dhwaneetbhatt.com/strapd/#/cli

Quick CLI install

# Unix/Linux/macOS
curl -fsSL https://raw.githubusercontent.com/dhwaneetbhatt/strapd/main/scripts/install.sh | bash

# Windows
Invoke-RestMethod -Uri "https://raw.githubusercontent.com/dhwaneetbhatt/strapd/main/scripts/install.ps1" | Invoke-Expression

History of CRLF vs LF for Line Endings

Dhwaneet Bhatt — Wed, 23 Aug 2023 08:59:33 +0000

Programmers often grapple with varying line endings across diverse operating systems. I've encountered this frustration multiple times when opening a file created on Unix-like systems (such as Linux and macOS) in Windows, only to find one long line without any breaks.

This issue arises because Windows uses CRLF (\r\n) to denote the newline character, while Unix-like systems represent it using LF (\n).

The origins of this discrepancy can be traced back to the 1960s, when many early computer systems adopted TeleType machines as their console devices. These machines featured a physical print head that moved to print characters on tape. To transition to the next line, the print head had to return from the far right to the left and then move the paper upward to continue printing from the left. Consequently, the instruction for moving to the next line comprised a Carriage Return (return to the initial position) and a Line Feed (add an extra line).

Subsequent operating systems, including CP/M, which was designed for Intel 8080/85-based computers, adhered to this convention. MS-DOS inherited it from CP/M during its development, and the practice endured.

The Multics operating system recognized this as wasteful and integrated logic into device drivers to identify the LF character and translate it as necessary for physical devices. Unix followed suit, and as a result, all Unix-like systems - comprising various Linux distributions and macOS - use LF as the line ending character.

It's intriguing that a decision made in the era of printing tape persists and continues to challenge developers today. Consequently, many programs dealing with files, such as text editors and version control systems, must accommodate both types of line endings. Some decisions, it seems, are indeed irreversible, but they certainly possess a captivating historical context.

Benchmarking Node.js Worker Threads

Dhwaneet Bhatt — Wed, 23 Mar 2022 09:06:46 +0000

NodeJS official documentation states that there is no real benefit of using worker threads for I/O, but wanted to benchmark it to understand the difference.

Setup

We used Benchmark.js for benchmarking and piscina as a pool for worker threads. benchmark.js was used to run the same code in 2 scenarios - one using a single thread and one using the piscina pool. The degree of parallelism was passed to the program via an environment variable. The test code is present in worker.js in both the cases.

These tests were run on a Macbook Pro (13-inch, 2020, Intel CPU) with 2.3 GHz Quad-Core Intel Core i7 (8 CPU cores) and 16GB of memory. The tests were run from an embedded terminal in VSCode. No other foreground processes were running.

Httpbin was used for testing I/O. This has the disadvantage of being slow vs a locally hosted mock server but reduces noise as I didn't want a competing server process sharing the same resources.

// benchmark.js
const path = require("path"),
  Benchmark = require("benchmark"),
  suite = new Benchmark.Suite(),
  Piscina = require("piscina"),
  pool = new Piscina({
    idleTimeout: 5000,
    filename: path.resolve(__dirname, "./worker.js"),
  }),
  method = require("./worker");

const parallelism = parseInt(process.env.P);

suite
  .add("single thread", {
    defer: true,
    fn: async function (deferred) {
      const promises = [];
      for (let i = 0; i < parallelism; i++) {
        promises.push(method());
      }
      await Promise.all(promises);
      deferred.resolve();
    },
  })
  .add("worker threads", {
    defer: true,
    fn: async function (deferred) {
      const promises = [];
      for (let i = 0; i < parallelism; i++) {
        promises.push(pool.run());
      }
      await Promise.all(promises);
      deferred.resolve();
    },
  })
  .on("cycle", function (event) {
    console.log(String(event.target));
  })
  .on("complete", function () {
    console.log("Done");
  })
  .run();

Default options for piscina.

{
    "filename": "worker.js",
    "name": "default",
    "minThreads": 4,
    "maxThreads": 12,
    "idleTimeout": 5000,
    "maxQueue": null,
    "concurrentTasksPerWorker": 1,
    "useAtomics": true,
    "taskQueue": {
        "tasks": []
    },
    "niceIncrement": 0,
    "trackUnmanagedFds": true
}

Only I/O

Send an HTTP request to an endpoint.

// worker.js
const request = require('request-promise');

module.exports = () => {
  return request('https://httpbin.org/get');
};

Parallelism	Single Thread	Worker Threads
1	1.15 ops/sec ±15.95% (11 runs sampled)	1.30 ops/sec ±15.04% (12 runs sampled)
2	1.20 ops/sec ±13.77% (11 runs sampled)	1.32 ops/sec ±12.93% (11 runs sampled)
4	1.29 ops/sec ±19.01% (11 runs sampled)	1.32 ops/sec ±10.32% (11 runs sampled)
8	1.09 ops/sec ±33.97% (10 runs sampled)	1.16 ops/sec ±22.55% (12 runs sampled)
16	1.09 ops/sec ±17.84% (10 runs sampled)	0.62 ops/sec ±28.86% (8 runs sampled)
32	1.09 ops/sec ±20.92% (10 runs sampled)	0.41 ops/sec ±38.40% (7 runs sampled)
64	0.72 ops/sec ±20.05% (8 runs sampled)	0.23 ops/sec ±26.54% (6 runs sampled)
128	0.64 ops/sec ±39.99% (8 runs sampled)	0.13 ops/sec ±14.95% (5 runs sampled)

Observations

No significant performance benefits over using a single thread.
Multi-threaded performance starts to degrade as parallelism increases beyond maxThreads.
Single thread performance also takes a hit as parallelism increases, but owing to large standard deviation, this could be because of server performance too.

CPU and I/O

Send an HTTP request to an endpoint after calculating fibbonacci recursively.

// worker.js
const request = require('request-promise');

function fibonacci(n) {
    if (n < 2)
      return 1;
    else
      return fibonacci(n - 2) + fibonacci(n - 1);
}

module.exports = async () => {
    fibonacci(20);
    await request('https://httpbin.org/get');
};

Parallelism	Single Thread	Worker Threads
1	1.04 ops/sec ±20.11% (10 runs sampled)	1.41 ops/sec ±7.75% (12 runs sampled)
2	1.38 ops/sec ±14.02% (12 runs sampled)	1.46 ops/sec ±6.33% (12 runs sampled)
4	1.10 ops/sec ±18.55% (10 runs sampled)	1.36 ops/sec ±11.84% (11 runs sampled)
8	1.04 ops/sec ±13.21% (10 runs sampled)	1.08 ops/sec ±23.24% (11 runs sampled)
16	1.10 ops/sec ±14.28% (11 runs sampled)	0.93 ops/sec ±59.30% (11 runs sampled)
32	1.04 ops/sec ±15.95% (10 runs sampled)	0.68 ops/sec ±84.99% (10 runs sampled)
64	0.69 ops/sec ±33.10% (9 runs sampled)	0.29 ops/sec ±110.97% (7 runs sampled)
128	0.72 ops/sec ±20.01% (8 runs sampled)	0.20 ops/sec ±146.04% (9 runs sampled)

Observations

I/O trumps CPU work, maybe a larger fibbonacci number could've provided different results.
Using worker threads is slightly better when parallelism is less than maxThreads but beyond that no advantage.

Only CPU

Calculate fibbonacci recursively.

// worker.js
function fibonacci(n) {
    if (n < 2)
      return 1;
    else
      return fibonacci(n - 2) + fibonacci(n - 1);
}

module.exports = () => {
    fibonacci(20);
};

Parallelism	Single Thread	Worker Threads
1	9,359 ops/sec ±1.05% (81 runs sampled)	7,048 ops/sec ±1.35% (83 runs sampled)
2	4,484 ops/sec ±1.94% (81 runs sampled)	6,678 ops/sec ±3.26% (83 runs sampled)
4	2,363 ops/sec ±0.83% (86 runs sampled)	5,390 ops/sec ±2.11% (84 runs sampled)
8	1,180 ops/sec ±0.85% (87 runs sampled)	1,632 ops/sec ±20.82% (68 runs sampled)
16	581 ops/sec ±0.78% (85 runs sampled)	726 ops/sec ±28.02% (68 runs sampled)
32	293 ops/sec ±0.86% (84 runs sampled)	493 ops/sec ±16.54% (66 runs sampled)
64	145 ops/sec ±1.02% (82 runs sampled)	266 ops/sec ±15.86% (69 runs sampled)
128	68.47 ops/sec ±1.62% (80 runs sampled)	106 ops/sec ±35.60% (63 runs sampled)

Observations

For CPU intensive work, use worker threads.

Conclusion

Worker threads for pure I/O based work do not provide any significant performance improvements. At higher parallelism, it performs worse than a single thread.
Worker threads provide significant performance benefits for CPU intensive work.
For mixed workloads, YMMV. There could be a minor performance bump as the CPU intensive work is offloaded to threads, but it depends on the time spent in CPU vs I/O.
Worker threads work well when parallelism is less than the number of CPU cores on the machine. Beyond that, performance starts to dip as the pool starts to queue work.

Run Docker without Docker Desktop on macOS

Dhwaneet Bhatt — Mon, 06 Sep 2021 05:47:08 +0000

Docker Inc. recently announced that Docker Desktop would no longer remain a free product for large organizations. It will remain free for personal and open-source projects and for organizations smaller than a certain size. Usually this is not a cause for concern as a company with revenue greater than $10 million would be able to afford Docker Desktop's $5 per user/month starter pricing. This post does in no way discourages organizations in paying and I believe that Docker Inc. has all the rights to monetize their product.

With the disclaimer out of the way, let us deep dive into explaining a little around what is free, what is paid and how exactly can we continue to use "containers", the core technology, without having to pay for Docker Desktop.

This digression explains terminology around Docker. Experts who know Docker can skip this part.

Docker can mean a lot of things. I'll try to break down and explain each term. This is by no means a full conceptual architectural explanation of how Docker works and I would recommend exploring other resources on the web for that purpose. This digression is just meant to ease the rest of the discussion.

Docker Inc - It is a USA based company that produces some open-source and not-open source software that makes it easier to develop, test and run applications in containers.
Docker Engine - The core technology behind Docker. It is an open source software that runs on linux as a daemon that makes it possible to run containers on top of Linux kernel. It is responsible for the container lifecycle and isolation of physical resources (compute, memory, storage) that containers can access. The engine can run on a physical or a virtual machine, but it can only run on top of a Linux kernel i.e. any OS that is flavour of Linux. This is important to understand. Docker engine only runs on Linux.
Docker CLI - This is the CLI that developers usually use to interact with the docker engine. This consists both of docker and docker-compose commands. Again, this is open source software.
Docker Desktop - Since Docker Engine only runs on Linux, developers who use Windows and macOS for software development cannot run the engine until they spin up a virtual machine (VM) that runs linux. That is where Docker Desktop comes in. Docker Desktop is a closed-source software that allows developers working on Windows/macOS to use container technology seamlessly on their development environment without needing to manage the complexity of operating a VM and all the nitty-gritty that comes along with it (networking, virtualization, knowledge of linux etc.). Docker Desktop is meant to be used during software development, it does not play a part in containers that run on production-like environments, where only Docker Engine is mostly involved.

Docker Desktop is not the core technology that runs containers, it only aims to make it easier to develop software on Windows/macOS that runs in containers. Thus Docker Inc. is only trying to get large companies to pay for the convenience that Docker Desktop offers when developing applications. So, I completely sympathise with the move for trying to earn revenue from a product that their software developers have worked so hard to develop. Company's other revenue comes from Docker Hub.

The way to continue to run and build applications for containers on macOS would be run Docker Engine on a Linux VM. I discussed two approaches that I've tried on my development environment (Macbook Pro 13" 2020 Intel Chip). This list is not exhaustive and they maybe more ways to do this. This post does assume some working knowledge of Docker.

Before doing this, uninstall Docker Desktop by removing /Applications/Docker.app. Sometimes this is not enough and it leaves certain things so I recommend searching for "uninstall docker desktop on macos" and follow a guide for full cleanup.

minikube

So far, minikube has emerged the easiest drop-in replacement for Docker Desktop. minikube is used to run a Kubernetes cluster on local environment. But it also runs a docker daemon that can be used to run containers. On macOS, minikube runs on a lot of virtualization technologies, but hyperkit is the easiest to use.

# Install hyperkit and minikube
brew install hyperkit
brew install minikube

# Install Docker CLI
brew install docker
brew install docker-compose

# Start minikube
minikube start

# Tell Docker CLI to talk to minikube's VM
eval $(minikube docker-env)

# Save IP to a hostname
echo "`minikube ip` docker.local" | sudo tee -a /etc/hosts > /dev/null

# Test
docker run hello-world

minikube Cheatsheet

minikube stop - stop the VM and k8s cluster. This does not delete any data. Just run minikube start to spin up the cluster.

minikube delete - This deletes the cluster with all the data. All mapped volumes will be lost. Know what you're doing before running this. If you just want to stop the cluster use minikube stop.

minikube ip - IP address of the VM where the cluster and docker engine run.

minikube runs a k8s setup and thus runs a lot of containers that are not required if not using k8s. We can run minikube pause to pause k8s related containers so they do not end up consuming system resources.

Manually managing the VM

If you already use a Linux VM locally for some other purposes or want more control over the setup, then this can be a good option. For this purpose we'll use VirtualBox to run the Linux VM and use Vagrant to make provisioning the VM easy and codified. We will use Ubuntu 20.04 LTS as the base OS for the VM.

# Install VirtualBox
brew install --cask virtualbox
brew install --cask virtualbox-extension-pack

# Install Vagrant and the vbguest plugin to manage VirtualBox Guest Additions on the VM
brew install vagrant
vagrant plugin install vagrant-vbguest

# Install Docker CLI
brew install docker
brew install docker-compose

# Create a Vagrantfile and a provisioning script
mkdir vagrant-docker-engine
echo \
"Vagrant.configure('2') do |config|
  config.vm.box = 'ubuntu/focal64'
  config.vm.hostname = 'docker.local'
    config.vm.network 'private_network', ip: '192.168.66.4'
  config.vm.network 'forwarded_port', guest: 2375, host: 2375, id: 'dockerd'
  config.vm.provider 'virtualbox' do |vb|
    vb.name = 'ubuntu-docker'
    vb.memory = '2048'
    vb.cpus = '2'
  end
  config.vm.provision 'shell', path: 'provision.sh'

  # Configuration for Port Forwarding
  # Uncomment or add new ones here as required
  # config.vm.network 'forwarded_port', guest: 6379, host: 6379, id: 'redis'
  # config.vm.network 'forwarded_port', guest: 3306, host: 3306, id: 'mysql'
end" | tee Vagrantfile > /dev/null
echo \
"# Install Docker
apt-get remove docker docker-engine docker.io containerd runc
apt-get update
apt-get install -y apt-transport-https ca-certificates curl gnupg lsb-release net-tools software-properties-common
curl -fsSL <https://download.docker.com/linux/ubuntu/gpg> | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] <https://download.docker.com/linux/ubuntu> focal stable' | tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io

# Configure Docker to listen on a TCP socket
mkdir /etc/systemd/system/docker.service.d
echo \\
'[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --containerd=/run/containerd/containerd.sock' | tee /etc/systemd/system/docker.service.d/docker.conf > /dev/null
echo \\
'{
  \"hosts\": [\"fd://\", \"tcp://0.0.0.0:2375\"]
}' | tee /etc/docker/daemon.json > /dev/null
systemctl daemon-reload
systemctl restart docker.service" | tee provision.sh > /dev/null
chmod +x provision.sh

# Spin up the machine
vagrant up

# Save IP to a hostname
echo "192.168.66.4 docker.local" | sudo tee -a /etc/hosts > /dev/null

# Tell Docker CLI to talk to the VM
export DOCKER_HOST=http://docker.local:2375

# Optionally add it to your shell so don't need to repeat everytime
# echo "export DOCKER_HOST=http://docker.local:2375" | tee -a ~/.zshrc > /dev/null

# Test
docker run hello-world

Vagrant Cheatsheet

vagrant suspend - stop the VM for saving system resources. This does not delete any data. Just run vagrant up to spin up the VM.

vagrant reload - for reloading the VM for any changes made to the config e.g. adding a new port mapping.

vagrant delete - This deletes the VM with all the data. All mapped volumes will be lost. Know what you're doing before running this. If you just want to stop the VM use vagrant suspend.

For every port that we want to natively access on macOS host, we need to modify the Vagrantfile to add port forwarding. Use vagrant reload after changing the file. Some examples have already been provided in the Vagrantfile for reference.

Caveats

Accessing Ports

Docker Desktop makes it very convenient to access services/apps running on containers by making everything available on localhost. How they exactly do it is unknown, although it must involve some port mapping via hyperkit, but we must do this manually.

Both in the minikube and virtualbox guides above, we make the IP address of the VM available under docker.local hostname so to access any services we must use that hostname instead of using localhost.

Using Vagrant we can actually do double port mapping (between container ↔ linux and linux ↔ host) to access stuff on localhost. That can be done my adding forwarded_port entries in Vagrantfile as mentioned above.

Volumes and Data Persistence

Since the Docker Engine is running on the VM, any volumes created or mapped will be present on the Linux VM, not on the macOS (host). This is very important to remember. This means that by destroying the VM we will lose access to the data in the volumes. Volumes are generally used for persistent databases like MySQL, PostgreSQL etc.

There are ways in which vagrant allows you to map folders to the VM, that would again be 3 layer-mapping like ports mentioned above, but becomes very complicated because or permission issues in the way Docker works. I tried a lot and could not make it work for MySQL. I could if I put in some extra hours, but the point is that it is cumbersome.

My advice would be to backup volumes to a location on the VM and pull that backup to the host either via scp if you're using minikube or vagrant has a default drive mapped at /vagrant which can be used for backup. Thankfully, backing up volumes is easy and it can be put in a cronjob on the VM/host if needed.

Performance

If you're using minikube, performance more or less remains the same because same underlying virtualization technology (hyperkit) is being employed. But for some strange reasons, I saw a huge jump in performance in read/write performance on MySQL when using vagrant + virtualbox combination. I have not delved into the reasons of it, but that has made to stick to it. I am yet to try minikube + virtualbox combination.

minikube actually runs a kubernetes cluster on the VM so if that is not needed, doing a minikube pause will make sure k8s cluster related containers are suspended so they do not consume any system resources.

Extract the query fields in graphql-go

Dhwaneet Bhatt — Tue, 19 May 2020 09:02:24 +0000

I have been using graphql-go for one of my projects at work, and I was looking for a way to extract the fields requested in the query to optimize the response.

The graphql API I am working on is actually a pass-through layer for another legacy API. Some of the fields are very heavy to query in the underlying API so we query for them only if the user is interested in those fields.

Sounds like the perfect use-case for GraphQL right? The request has a set of fields that the user is interested in. Except, it is not easy to extract the fields during query/mutation that the user is interested in.

I had to debug and inspect the code before I could write code to extract the fields that user wants to query from the response.

The set of fields are hidden behind what is known as an Abstract Syntax Tree, or AST for short. I do not claim to understand AST (or GraphQL internals fully), but here is an excellent post that explains the internals: https://medium.com/@cjoudrey/life-of-a-graphql-query-lexing-parsing-ca7c5045fad8.

For the purposes of the demo, consider the following schema:

type Person {
  id: ID!
  name: String!
  age: Int!
}

type Query {
  person(id: ID!): Person
}

Suppose that it takes an insane amount of time to calculate the age field, so we want to query it only if the user wants it and mentions in the field.

Here is the code for query in graphql-go, for querying the above schema:

package main

import (
    "encoding/json"
    "fmt"
    "net/http"

    "github.com/graphql-go/graphql"
    "github.com/graphql-go/graphql/language/ast"
)

type Person struct {
    id   string
    name string
    age  int
}

var personFields = graphql.Fields{
    "id": &graphql.Field{
        Type: graphql.NewNonNull(graphql.String),
        Resolve: func(p graphql.ResolveParams) (interface{}, error) {
            if person, ok := p.Source.(Person); ok {
                return person.id, nil
            }
            return nil, fmt.Errorf("expected Person")
        },
    },
    "name": &graphql.Field{
        Type: graphql.NewNonNull(graphql.String),
        Resolve: func(p graphql.ResolveParams) (interface{}, error) {
            if person, ok := p.Source.(Person); ok {
                return person.name, nil
            }
            return nil, fmt.Errorf("expected Person")
        },
    },
    "age": &graphql.Field{
        Type: graphql.NewNonNull(graphql.Int),
        Resolve: func(p graphql.ResolveParams) (interface{}, error) {
            if person, ok := p.Source.(Person); ok {
                return person.age, nil
            }
            return nil, fmt.Errorf("expected Person")
        },
    },
}

var personQuery = &graphql.Field{
    Type: graphql.NewObject(graphql.ObjectConfig{
        Name:   "Person",
        Fields: personFields,
    }),
    Args: graphql.FieldConfigArgument{
        "id": &graphql.ArgumentConfig{
            Type: graphql.NewNonNull(graphql.String),
        },
    },
    Resolve: func(p graphql.ResolveParams) (interface{}, error) {
        id, ok := p.Args["id"].(string)
        if !ok {
            return nil, fmt.Errorf("expected string")
        }
        queriedFields := getQueriedFields(&p)
        if queriedFields["age"] {
            // rest of the code here
            // query the database with age, otherwise skip it
        }
        return Person{id: id}, nil
    },
}

func main() {
    schema, _ := graphql.NewSchema(graphql.SchemaConfig{
        Query: graphql.NewObject(graphql.ObjectConfig{
            Name: "Query",
            Fields: graphql.Fields{
                "person": personQuery,
            },
        }),
    })
    handler := func(w http.ResponseWriter, r *http.Request) {
        result := executeQuery(r.URL.Query().Get("query"), schema)
        json.NewEncoder(w).Encode(result)
    }
    http.HandleFunc("/graphql", handler)
    http.ListenAndServe(":8080", nil)
}

func executeQuery(query string, schema graphql.Schema) *graphql.Result {
    result := graphql.Do(graphql.Params{
        Schema:        schema,
        RequestString: query,
    })
    if len(result.Errors) > 0 {
        fmt.Printf("unexpected errors: %v", result.Errors)
    }
    return result
}

The missing method which fetches the queried fields from the graphql.Params object:

// getQueriedFields returns the fields requested in the query/mutation
func getQueriedFields(p *graphql.ResolveParams) map[string]bool {
    fields := map[string]bool{}
    if p.Info.FieldASTs == nil {
        return fields
    }
    for _, fieldAST := range p.Info.FieldASTs {
        if fieldAST.SelectionSet == nil {
            return fields
        }
        for _, sel := range fieldAST.SelectionSet.Selections {
            field, ok := sel.(*ast.Field)
            if !ok {
                return fields
            }
            fields[field.Name.Value] = true
        }
    }
    return fields
}

So by calling getQueriedFields we can get the fields that the user has requested in the query, and then pass those fields list to the backend, either to only select fields for the SQL query, or something else.

This can also technically be applied to a mutation, in fetching the record after the mutation completes, but makes more sense in a query.

Using AWS Parameters Store for Storing Secrets and Configuration

Dhwaneet Bhatt — Tue, 19 May 2020 09:00:02 +0000

AWS Parameters Store is a very good candidate for storing secrets and configuration. It supports encryption of parameters at rest.

AWS also has a Secrets Manager which is a more evolved product, and I think it has evolved out of Parameters Store itself. The only benefit it has over Parameters Store is the automatic rotation of keys with other products like RDS. Also, it is not free. It costs $0.40 per secret and $0.05 per 10,000 API calls.

Key rotation can be achieved manually in Parameters Store by writing a lambda, which is a content for a future post. For now, will focus on Parameters Store.

Storing Secrets in Parameters Store

AWS Console

Using Parameters Store via AWS Console is very simple. It is located under Systems Manager panel.

Parameters Store supports creating String, StringList and SecureString types. When creating SecureString parameter, we can choose the KMS key that is required to encrypt the parameters.

Using AWS CLI

To create a parameter:

aws ssm put-parameter --name "<name>" --value "<value>" --type SecureString --key-id "<kms-key-id>"

If SecureString is not required, --key-id parameter can be omitted. If you want to overwrite an existing parameters, pass the flag --overwrite to the command.

To test whether the parameter was created:

aws ssm get-parameter --name "<name>" --with-decryption

If it is not a type SecureString, --with-decryption is not required. Note that the decryption happens on the server side, the secret is transferred in plaintext (of course over HTTPS).

Hierarchical Storage

To me, this is hands-down one of the best features of Parameters Store. It supports organizing secrets in a hierarchical format, allowing you to issue fetch requests for a top-level key and retrieve all parameters under it.

There is a complete article on AWS about it. I'll cover a little here.

In Parameters Store, you can define you keys in a hierarchy separated by /, for example:

/Dev/Database/Username
/Dev/Database/Password
/Dev/Database/Host

These are called "paths" in the AWS nomenclature. Then, you can issue a single request to fetch all parameters under a path:

aws ssm get-parameters-by-path --path "/Dev/Database" --with-decryption

This will give out all 3 properties. The response uses pagination and the maximum limit of records is 10.

One can use hierarchical storage to group Development and Production properties separate, or in case of microservices, group properties by service.

For example, following various applications, here is how you can define various properties for different environments:

/Dev/App1/Param1
/Dev/App1/Param2

/Dev/App2/Param1
/Dev/App2/Param2

/Prod/App1/Param1
/Prod/App1/Param2

/Prod/App2/Param1
/Prod/App2/Param2

You can then issue a request for /Dev/App1 or /Prod/App1 to fetch all properties of the particular app/service.

Integrating in applications

The easiest and obvious way is to integrate AWS SDK into your application and fetch parameters when needed. There are advantages and disadvantages of this approach.

The only advantage I see is it is intuitive and direct. Using AWS SDK in your app tells that this app needs some functionality of AWS to run. And AWS SDK is available in a lot of languages.

But, if you have a lot of apps, you end up putting the SDK in all of them (or maybe in a common library) which ends up bloating all the applications. If you want to ever change the way config and secrets are injected into your apps, code change is required in all the applications.

12-factor says that configuration should be injected into the program externally. Most preferred way is to use environment variables. (Note: When running apps in containers, the most intuitive way would be to pass environment variables to the container using -e. But anyone can see the environment variables passed to the container using docker inspect, so I don't believe it to be that secure.)

Thus, the best way is to create configuration at runtime before the actual application starts. That is, run a program that fetches data from Parameters Store and prepares the it in a format the way the app requires it. It could be JSON, YAML or simple environment variables.

The benefit of the approach is that the app simply dictates that it requires the configuration in a particular format, and the way the app is presented with the configuration is the concern of some other program. This works well with switching providers, or using some other source to store secrets.

Open Source Applications

There are very few open source applications which work with Parameters Store and export variables in a desired format. Before I started working on this problem, I just found one such open source app - chamber (It's name is inspired from Harry Potter's 2nd book, Chamber of Secrets).

It is really well maintained, and it does cover a lot of use cases. I evaluated use of chamber for use at my current organization, but it did not fit the requirements, because it does not work with hierarchical configuration in Parameters Store. It supports only one level of hierarchy, where the first level is a service, and the rest all the secrets are stored below it.

As discussed above, we needed to organize our parameters into multiple nested hierarchies, by environment, app etc. So we wrote an open source CLI for it - configurator. It allows exporting the hierarchical parameters in the store into a nested format like JSON, which is what we required in order to minimize changes, because our apps ingested configuration in JSON config files.

It also has the support for printing export name=value on stdout, if someday we move out from JSON files to environment variables. But for now, it served our purpose well and allowed us to move to Parameters Store without a lot of effort or code changes in apps.

Contributions and feedback to configurator are welcome.

Time based log file rotation with zap

Dhwaneet Bhatt — Tue, 19 May 2020 05:48:24 +0000

zap is one of the nicely maintained, well-written and nicely performing open source logging libraries in Go. Unfortunately, not all apps in our organization are 12-factor apps and thus we still log to physical files. So, I needed a way to rotate log files while using zap. I come from Java background, and log4j has a very vast array of log rotation options. But I found it lacking in zap, clearly because in the world of containers, logging to stdout (treating logs as event streams - 12-factor logs) is very common.

To be specific, I was looking to do time based rotation (one log file per hour, helps in debugging) and not size based. On zap's FAQ, they've shown integration with lumberjack, which only supports size based rotation.

I found another library file-rotatelogs which does support time based rotation. So I used it along with zap.

The good thing about zap is that it takes any io.Writer interface as a WriteSyncer and file-rotatelogs returns *RotateLogs which implements the io.Writer interface.

Here the rotatelogs.WithRotationTime(time.Hour)) indicates hourly rotation. rotatelogs.WithMaxAge(60*24*time.Hour) indicates the files are cleaned up after 60 days.

package main

import (
    "encoding/json"
    "time"

    rotatelogs "github.com/lestrrat-go/file-rotatelogs"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func main() {
    // initialize the rotator
    logFile := "/var/log/app-%Y-%m-%d-%H.log"
    rotator, err := rotatelogs.New(
        logFile,
        rotatelogs.WithMaxAge(60*24*time.Hour),
        rotatelogs.WithRotationTime(time.Hour))
    if err != nil {
        panic(err)
    }

    // initialize the JSON encoding config
    encoderConfig := map[string]string{
        "levelEncoder": "capital",
        "timeKey":      "date",
        "timeEncoder":  "iso8601",
    }
    data, _ := json.Marshal(encoderConfig)
    var encCfg zapcore.EncoderConfig
    if err := json.Unmarshal(data, &encCfg); err != nil {
        panic(err)
    }

    // add the encoder config and rotator to create a new zap logger
    w := zapcore.AddSync(rotator)
    core := zapcore.NewCore(
        zapcore.NewJSONEncoder(encCfg),
        w,
        zap.InfoLevel)
    logger := zap.New(core)

    logger.Info("Now logging in a rotated file")
}

This will create the file /var/log/app-2020-05-18-15.log, which is exactly how we wanted it. It will automatically start logging to the next file during the start of the next hour.