DEV Community: Eugene Zimin

Removing Microsoft's Bloatware

Eugene Zimin — Sun, 25 Oct 2026 11:16:46 +0000

The updater that outlived its apps

About a year ago I deleted Office from my Mac the way every Mac user is taught to delete anything, by dragging it to the Trash, and two weeks later Microsoft AutoUpdate opened a window to offer me updates for the applications I no longer had. The moment was funny, but it pointed at something real, because the Trash only works as an uninstaller when an app keeps all of itself inside its own bundle, and Microsoft's software does not. What looks like a handful of applications in /Applications is the visible top of a platform that installs its own updater, registers background jobs with the system, and parks a helper tool that runs as root, and none of those pieces live inside the apps you can see.

You can check this on your own machine with two commands, and Figure 1 shows what they printed on mine. The first lists Microsoft's entries in the three system folders where software asks macOS to run something on its behalf, and the second opens one of those entries to show when it runs.

Figure 1. The AutoUpdate agent, its daemon and its root helper sit in /Library, outside every app bundle, and the agent is set to start at login and again every 7,200 seconds.

Each of the three files does a different job. com.microsoft.update.agent.plist is a launch agent, a small property list that tells launchd, the process that starts and supervises background work on macOS, to run Microsoft Update Assistant whenever someone logs in (RunAtLoad) and every two hours after that (a StartInterval of 7,200 seconds). The plist in LaunchDaemons describes com.microsoft.autoupdate.helper, and because launch daemons run in the system domain rather than inside a user's session, the binary it points to in PrivilegedHelperTools runs as root and offers services with names like install, clone and cleanup. That arrangement is how AutoUpdate replaces apps in /Applications without asking for your password each time, and it is also why deleting Word does nothing to it. The helper was installed once, by a package you approved, it has no window and no menu, and nothing in the Finder will ever show it to you.

I ended up writing a script, ms-uninstall.sh, to find and remove all of this properly. The deleting turned out to be the easy part, since it comes down to mv or rm, and almost all of the work went into two other questions, namely which files belong to which product, and how to avoid deleting anything that a product you decided to keep still needs. The rest of this article follows the script in the order in which it answers them.

Finding Microsoft's leftovers

The obvious first move is to search /Applications for anything whose name starts with "Microsoft", and it fails almost at once. Remote Desktop is now called Windows App, Teams has shipped as both "Microsoft Teams classic" and plain "Microsoft Teams", and OneDrive has never carried the company's name in its title. Names belong to marketing, and marketing renames things.

What does not change is the bundle identifier, a reverse-DNS string such as com.microsoft.Word that every app declares in its Info.plist under the key CFBundleIdentifier. macOS keys nearly everything about an app on that string, from the location of its sandbox to the name of its preferences file, which makes it the one name worth matching on. Figure 2 sets the two kinds of name side by side for the apps that make the point most clearly.

Figure 2. Finder names drift with every rebrand, while the bundle identifier that macOS stores data under stays put, so matching on com.microsoft.* finds apps that a search by name would miss.

Reading the identifier takes a single command in principle, and the script wraps it in a small function that tries two different readers in turn.

bundle_id_of() {
  local plist="$1/Contents/Info.plist"
  [[ -f "$plist" ]] || return 0
  defaults read "$1/Contents/Info" CFBundleIdentifier 2>/dev/null || \
  plutil -extract CFBundleIdentifier raw -o - "$plist" 2>/dev/null || true
}

defaults reads most property lists without complaint, plutil serves as the fallback for the ones it cannot parse, and the closing || true makes an unreadable bundle return an empty string instead of halting a script that runs under set -euo pipefail. With that function in hand, the scan walks /Applications, /Applications/Utilities and ~/Applications, keeps every bundle whose identifier begins with com.microsoft. or com.skype., and adds one last entry for Microsoft data that no installed app claims. Figure 3 is that first screen on my Mac, which currently has Teams and Windows App installed.

Figure 3. The discovery screen lists each product with its bundle identifier, its total size and the number of paths it owns, followed by leftovers that no installed app claims.

The sizes are the first surprise. The Teams bundle itself weighs 1.1 GB, yet the product comes to 3.4 GB, because its container in ~/Library holds another 2.3 GB that dragging the app to the Trash would have left exactly where it was. The "12 items" beside it raises the obvious next question, which is where those twelve paths come from.

Where each app actually lives

A modern Mac app keeps its state in a set of well-known folders, each named after its bundle identifier, so that once you know the identifier you can predict nearly every place the app has written to. Figure 4 maps those places, using Teams as the example and grouping the folders by what they hold rather than by where they sit.

Figure 4. One bundle identifier spreads across about two dozen locations in ~/Library and /Library, and the red column, the part that keeps running, is the part people forget.

The green column holds the app's real data. Containers/$id is the sandbox, a private home directory that macOS gives every sandboxed app, complete with its own Data/Documents and Data/Library, and it is usually the largest piece by far. The yellow column holds what the app can rebuild on its own, such as caches, web storage, cookies and logs. Those first two columns are the ones people remember when they clean up by hand, and they are also the least harmful to leave behind, since a stale cache takes up space but never does anything.

The red column is different in kind. A launch agent in ~/Library/LaunchAgents runs as you when you log in, an agent in /Library/LaunchAgents runs in the session of every user on the machine, and a launch daemon in /Library/LaunchDaemons runs as root from boot, usually pointing at a binary in PrivilegedHelperTools. Each of these files is an instruction to launchd, and launchd carries it out whether or not the app that wrote it still exists, which is the entire story behind Figure 1.

In the script, all of this lives in one function that turns an identifier into candidate paths and prints the ones that exist. The excerpt below keeps the pattern and trims the middle of the list.

paths_for_id() {
  local id="$1" p
  shopt -s nullglob
  local cands=(
    "$HOME_DIR/Library/Containers/$id"
    "$HOME_DIR"/Library/Containers/"$id".*
    "$HOME_DIR/Library/Application Scripts/$id"
    "$HOME_DIR"/Library/Application\ Scripts/"$id".*
    "$HOME_DIR/Library/Preferences/$id.plist"
    # ... caches, cookies, saved state, logs, launch agents ...
    "/Library/LaunchDaemons/$id.plist"
    "/Library/PrivilegedHelperTools/$id"
  )
  shopt -u nullglob
  for p in "${cands[@]+"${cands[@]}"}"; do
    [[ -e "$p" || -L "$p" ]] && printf '%s\n' "$p"
  done
}

The "$id".* patterns are there because apps ship extensions alongside themselves, and macOS sandboxes each extension separately, so Teams owns com.microsoft.teams2 together with com.microsoft.teams2.agent, .notificationcenter, .respawn and .widgetextension, every one of them with a container and an Application Scripts folder of its own. Orphan discovery leans on the same naming rule from the other side, skipping any folder whose name begins with an installed product's identifier, so that a widget extension is never mistaken for a leftover of an app that is gone. The nullglob setting, meanwhile, stops a pattern that matches nothing from expanding to itself, since without it bash would hand the literal string .../com.microsoft.teams2.* to the loop as though it were a path. The existence test at the bottom would catch that string here, but literal patterns are the most dangerous thing a removal script can produce, and section 8 comes back to them.

One product under many faces

Prefix matching handles extensions, but it cannot help when a product's helpers carry identifiers that share no prefix with the app. OneDrive is the clearest case. The app you see is com.microsoft.OneDrive, while its updater, its standalone updater, the daemon behind that standalone updater, its launcher and its SharePoint integration each have identifiers of their own, and several of them install launch items. Figure 5 shows the whole family.

Figure 5. A single visible OneDrive.app expands into seven bundle identifiers, and each one is searched separately for preferences, caches and launch items.

Nothing inside the app bundle mentions these relatives, so the only option is to look them up once and write them down. The script keeps them in a case statement that is about as glamorous as a phone book.

sibling_ids() {
  case "$1" in
    com.microsoft.OneDrive*)
      printf '%s\n' com.microsoft.OneDrive com.microsoft.OneDrive-mac \
        com.microsoft.OneDriveUpdater com.microsoft.OneDriveStandaloneUpdater \
        com.microsoft.OneDriveLauncher com.microsoft.SharePoint-mac \
        com.microsoft.OneDriveStandaloneUpdaterDaemon ;;
    com.microsoft.wdav*)
      printf '%s\n' com.microsoft.wdav com.microsoft.wdav.tray com.microsoft.wdav.epsext \
        com.microsoft.wdav.netext com.microsoft.dlp com.microsoft.dlp.daemon ;;
    # ... Teams, Edge, Outlook, AutoUpdate, Remote Desktop ...
    *) printf '%s\n' "$1" ;;
  esac
}

Defender, which identifies itself as com.microsoft.wdav, is the other large family, bringing a menu bar app, a network extension, an endpoint security extension and two data loss prevention components along with it. Every discovered app passes through sibling_ids first, and every identifier that comes out passes through paths_for_id, so the list that finally reaches you covers the whole product rather than only the part with an icon. A table like this goes stale whenever Microsoft ships a new helper, which is the price of knowledge that cannot be derived from the system, and the orphan entry at the bottom of the menu exists partly to catch whatever the table misses.

Shared parts and the keeper's veto

Removing everything is straightforward. The hard case, and the common one, is removing only some things, Teams but not Word, say, or Office but not the Defender agent your employer requires, because Microsoft's shared components really are shared, and deleting one on behalf of an app you are removing can quietly break an app you are keeping.

The group container is the best example. Apps signed by the same developer can share a folder in ~/Library/Group Containers named after that developer's Apple team ID, and Microsoft's team ID is UBF8T346G9. Word and Excel keep settings and autosave state in those folders, and Outlook keeps its entire local mail store under UBF8T346G9.Office, so deleting that folder while Outlook is still installed uninstalls nothing at all and simply empties someone's mailbox. AutoUpdate, which serves every Microsoft product at once, and the fonts in /Library/Fonts/Microsoft, which anything that opens a .docx relies on, are in the same position.

The script's answer is to make every dependency explicit. Each shared component is registered with the bundle identifier prefixes of the apps that still need it, and each product you decide to keep becomes a keeper that holds a veto over the components it relies on. Here is the registration for the group containers, exactly as it appears in the script.

shared_add "Office group containers (documents, settings, Outlook mail store)" \
  "com.microsoft.Word com.microsoft.Excel com.microsoft.Powerpoint com.microsoft.Outlook com.microsoft.onenote.mac com.microsoft.OneDrive com.microsoft.teams com.microsoft.rdc com.microsoft.wdav com.microsoft.CompanyPortalMac com.skype." \
  "$HOME_DIR"/Library/Group\ Containers/UBF8T346G9.* \
  "$HOME_DIR"/Library/Application\ Scripts/UBF8T346G9.*

The second argument is the list of apps with a claim on the component. Once you have made your selection, the script gathers the identifiers of every product you did not select, and any shared component whose list matches one of them is marked as kept and taken off the menu. Figure 6 traces that logic for a typical case, in which all of Office goes and Teams stays.

Figure 6. Teams still depends on AutoUpdate and on the group containers, so both stay, while the Office helper data and the fonts, which only Office apps need, are offered for removal.

The dashed lines matter as much as the solid ones. Word, Excel and the others depend on the group containers too, but they are leaving, so their claims no longer count, and the only thing standing between those folders and the Trash is Teams.

A vetoed component is still printed, together with the reason, because hiding it would leave you wondering why a 214 MB folder survived, whereas a line naming the app that holds it back tells you exactly which answer to change if that was not what you meant. Figure 7 is a real dry run on my Mac in which I chose Teams and kept Windows App.

Figure 7. With Windows App kept, every shared component is marked KEPT and labeled with the app that needs it, so removing Teams touches only Teams' own files.

Read the plan! Before, not After!

Everything described so far only reads the disk. Nothing changes until the script has shown you the complete plan, and with --dry-run it stops at exactly that point, which makes a dry run the right first command on any Mac. Figure 8 shows the plan for the most aggressive choice available, every product and every shared component, trimmed where the lists repeat themselves.

Figure 8. The removal plan groups every path by product with its size, warns about files inside containers that are about to go, and ends with a total and the removal method.

The note at the top exists because some apps keep real documents inside their sandbox, in Data/Documents, and the script lists them before the plan so that you can copy them out first. Here the only file is a README.txt from Windows App's health checks, but on a Mac that has run Office for years the same note can reveal a folder of files you assumed were somewhere else. Lower down, the [Keychain] block appears only when no Office app is being kept, and it removes the cached sign-in and license entries that Office stores in the login keychain, at the cost of signing in again if you ever reinstall. The closing lines state the method, and the default is a move to ~/.Trash.

Without --dry-run, the script asks you to type the word REMOVE exactly, treats any other answer as a cancellation that exits with code 3, and refuses to continue at all when standard input is not a terminal, so that a confirmation can never be piped in by accident. For automation there are --select and --shared, which accept the same answers you would type at the two prompts, such as 1,3 or all, but even then the final REMOVE has to come from a person at the keyboard.

One preparation is easy to miss. macOS shields the data in app containers from other processes, and a terminal without Full Disk Access receives "Operation not permitted" when it tries to read ~/Library/Group Containers/UBF8T346G9.*. On my Mac that made the group containers appear as 0 MB until I granted Full Disk Access to my terminal in System Settings, under Privacy & Security, and a real removal would have failed on those folders halfway through. Grant the permission before you run the script, and revoke it afterward if you prefer to keep your terminal unprivileged.

Run it!

Once you confirm, the removal proceeds in four steps, and their order is anything but arbitrary, since each step exists to prevent a specific failure that the next one would otherwise cause. Figure 9 pairs every step with the failure it prevents.

Figure 9. The script quits the apps, unloads their launch jobs, moves their files and restarts cfprefsd, in that order, and each box below shows what goes wrong when the step above it is skipped.

The apps go first. The script asks each running app to quit through AppleScript, which gives it the chance to save its work or to ask you to, then checks with pgrep two seconds later and aborts the whole run if the app is still alive, because a save dialog left open is precisely the situation in which deleting files would destroy work. The launch jobs come next, unloaded with launchctl bootout while their plists still exist, since launchd keeps a loaded job in memory after its plist disappears, and a deleted agent with an interval like AutoUpdate's would go on trying to fire every two hours until the next reboot. Only after both of those steps does any file move.

The final step looks like an afterthought.

killall cfprefsd >/dev/null 2>&1 || true

cfprefsd is the daemon through which every app's preferences pass, and it caches the contents of preference files in memory, so if it stays running after the plists are gone it can write the cached values back to disk, which is the usual explanation for a deleted preference that reappears a few minutes later. Killing it is safe, because launchd restarts it at once and the fresh copy reads only what is actually on disk.

Guard rails

A script that runs rm -rf under sudo deserves protections beyond a confirmation prompt, however much its author trusts it, and this one has two.

The first is the Trash default. Every path moves to ~/.Trash unless you pass --hard-delete, and when two items share a name, as the three separate folders called Microsoft do, the later arrivals receive a timestamped suffix instead of overwriting the earlier ones. Paths that needed sudo to move are handed back to your user afterward, so that you can empty the Trash, or drag something back out of it, without a password. Uninstalling is the one operation where you tend to discover an hour later that some container held the only copy of something, and the Trash turns that discovery from a restore from backup into a drag and drop.

The second is a denylist inside remove_path, checked against every path immediately before it is touched.

case "$path" in
  /|/Applications|/Applications/Utilities|/Library|/Library/Fonts|/Library/Caches|\
  /Library/Preferences|/Library/LaunchAgents|/Library/LaunchDaemons|\
  /Library/PrivilegedHelperTools|/Library/Application\ Support|\
  "$HOME_DIR"|"$HOME_DIR"/Library|"$HOME_DIR"/Library/Containers|\
  "$HOME_DIR"/Library/Preferences|"$HOME_DIR"/Library/Caches|\
  "$HOME_DIR"/Library/Group\ Containers|"$HOME_DIR"/Library/LaunchAgents|\
  "$HOME_DIR"/Library/Application\ Support)
    die "internal: refusing to delete $path" ;;
esac

None of these paths should ever reach this point, and that is exactly why the check is there. Removal scripts classically go wrong in one of two ways, either through a variable that turns out to be empty, so that a path built as "$dir/$id" collapses to /, or through a glob that matches nothing and survives as a literal, and in both cases a precise delete becomes the deletion of a parent directory. The script already defends against each of those separately, with set -u for unset variables, nullglob for patterns, and a refusal to run as root at all, so the denylist serves as the final check, cheap enough to keep even though it should never fire.

What's left, and what to leave

A completed run ends with a verification sweep that lists any Microsoft items still on disk and any Microsoft processes still running, and on a Mac where you kept something, some of both are expected. A few things fall outside the script's reach even after a full removal, and Figure 10 shows two of them on my machine.

Figure 10. Package receipts outlive the files they describe, and one of them points to a Teams audio driver in /Library/Audio/Plug-Ins/HAL that the script does not yet look for.

Package receipts come first. macOS records every installer package in a database of its own, so pkgutil --pkgs keeps listing Microsoft packages after their files are gone, which is harmless bookkeeping that sudo pkgutil --forget <package-id> will clear if it bothers you. The receipts earn their keep in another way, because you can ask any of them what it installed, and that is how I found MSTeamsAudioDevice.driver, a virtual audio device that Teams places in /Library/Audio/Plug-Ins/HAL so that it can share your computer's sound during a meeting. It belongs to none of the bundle identifiers the script searches, so it survives a Teams removal, and until the script learns about it, two commands remove it by hand.

sudo rm -rf /Library/Audio/Plug-Ins/HAL/MSTeamsAudioDevice.driver
sudo killall coreaudiod

The second command restarts Core Audio, so the device drops out of the sound settings without a reboot.

Edge escapes for a related reason, since it keeps its profile in ~/Library/Application Support/Microsoft Edge, a folder named after the product rather than its identifier, which the per-identifier search never reaches and the shared Microsoft folder does not include. Visual Studio Code genuinely is com.microsoft.VSCode and appears in the list like everything else, which is one more reason to read the list before typing all. If your employer manages your Mac, leave Defender, Company Portal and the Intune agent alone entirely, because device management will put them back at its next check-in, and the removal is likely to raise a compliance alert along the way.

The gentlest way to begin is ./ms-uninstall.sh --dry-run, which runs the discovery, prints the plan and changes nothing. That output is worth seeing even if you never remove a thing, because the honest answer to the question of what Microsoft is running on your Mac tends to be longer than you would guess. On mine it came to a 3.4 GB chat app, an agent that wakes every two hours, a helper that runs as root, and an audio driver I had never heard of, and once you can read a list like that, the Trash looks less like an uninstaller and more like the first of several steps.

From Source Code to a Running Pod

Eugene Zimin — Sun, 25 Oct 2026 09:38:02 +0000

The road from a source file to a Service

At the end of Part 2, the cluster in our VM could run any image that somebody had already built and published, as it did with the nginx web server. The step that remains is the one every enterprise developer takes many times a day, which is turning code they have just written into something the cluster can run, and that step is less a single leap than a chain of small transformations. Source code becomes a JAR file, the JAR becomes a container image, the image is pushed to a registry, the cluster pulls it to start pods, and a Service routes requests to those pods. Figure 1 draws the chain.

Figure 1. The path from source code to a running Service, in which Maven turns the code into a JAR, Docker turns the JAR into an image and pushes it to the registry, and Kubernetes starts two pods from the image behind a NodePort Service, with a check available after each link.

Each link in that chain can be checked on its own, as the line beneath the figure reminds us. A JAR can be started with java -jar, an image can be started with docker run, and a Service can be called with curl. That property is what makes the chain practical, because when the final request fails, a developer who understands the links can test them one at a time and find the broken one in minutes, whereas a developer who knows only the first and last steps can do nothing but start again from the beginning. This article therefore walks the chain from left to right, stopping at each link to make sure it holds before moving on.

The toolchain

Java and Maven

Our service is written in Java, so the VM needs a Java Development Kit, the compiler and runtime together, and Maven, the build tool that downloads libraries and packages the compiled code into a JAR. The commands below install OpenJDK 25, the current long-term-support release of Java, together with Maven and curl, a small command-line HTTP client that the default Ubuntu 26.04 desktop does not include, and then print the versions of all three Java tools.

sudo apt-get install -y openjdk-25-jdk maven curl
java -version && javac -version && mvn -v

The -y option answers yes to the installer's confirmation question in advance. If every command prints a version, as in Figure 2, the Java toolchain is ready. The screenshot pipes the Maven output through head -1 to keep it to a single line, and your own output of mvn -v will be a few lines longer.

Figure 2. The version check after installation, in which java and javac report OpenJDK 25.0.4.1 and mvn reports Apache Maven 3.9.12.

Docker

Building container images requires Docker, which we used in the containerization module. Ubuntu packages the Docker Engine in its own repositories under the name docker.io, and the separate docker-buildx package adds BuildKit, the modern build component that Docker uses to execute a Dockerfile. After installing both, add your user to the docker group, so that Docker commands work without sudo.

sudo apt-get install -y docker.io docker-buildx
sudo usermod -aG docker $USER     # then log out and log in again
docker run --rm hello-world

As with the microk8s group in Part 2, group membership takes effect only in a new login session, so log out and back in before running the last command. If you forget, the command fails with a message containing "permission denied" and docker.sock, which means only that the current session does not yet belong to the group. In the new session, the hello-world image downloads, runs, prints its greeting, and is removed again because of the --rm option, as Figures 3 and 4 show.

Figure 3. Installing the Docker Engine and BuildKit from the Ubuntu repositories.

Figure 4. The hello-world container confirming that Docker can download and run images.

Anyone who meet the "permission denied" message sometimes find advice online to make the Docker socket writable by everyone, with a command such as chmod 666 /var/run/docker.sock. Never follow it. The socket is the door through which programs give orders to the Docker daemon, and the daemon runs as root, so anyone who can write to the socket can start a container that mounts the host's entire file system and do anything they like with it. Membership in the docker group grants the same power, which is why that group is treated as equivalent to administrator rights, but it grants it only to the users you deliberately add.

The IDE and the project

Installing IntelliJ IDEA

Any IDE you are comfortable with will do for this course, whether IntelliJ IDEA, Visual Studio Code with the Java extensions, Eclipse, or NetBeans, and this article uses IntelliJ IDEA. Since 2025 IntelliJ IDEA has been a single product, free for non-commercial use, with its Ultimate features available through a trial or through a free educational license, for which you can apply at jetbrains.com/community/education with your college e-mail address. Like MicroK8s, it is distributed as a snap, so one command installs it, as Figure 5 shows.

sudo snap install intellij-idea --classic

Figure 5. Installing IntelliJ IDEA from the snap store.

Generating the project with Spring Initializr

A Spring Boot project needs a build file that lists the right libraries in compatible versions, a main class, a configuration file, and a Maven wrapper, and writing all of that by hand is both tedious and error-prone. Spring Initializr, a web service at start.spring.io, generates it for us. Open it in Firefox inside the VM and make the choices shown in Figure 6, which means Maven as the build tool, Java as the language, Spring Boot 4.1.x, the group com.cst8277 and the artifact demo, Jar packaging, Java 25, and two dependencies, Spring Web for writing REST controllers and Spring Boot Actuator for the health endpoints that Kubernetes will use later.

Figure 6. Spring Initializr configured for a Maven project with Spring Boot 4.1, Java 25, the group com.cst8277, the artifact demo, and the Spring Web and Spring Boot Actuator dependencies.

Click Generate. The browser downloads a file named demo.zip, which the commands below unpack into a projects folder in your home directory, after which the last command opens the new project in IntelliJ IDEA, the trailing & letting the IDE run in the background so that the terminal stays free.

mkdir -p ~/projects && cd ~/projects
unzip ~/Downloads/demo.zip
intellij-idea ~/projects/demo &

Opening the project and choosing the JDK

On its first start, IntelliJ IDEA asks you to accept its user agreement and to confirm that you trust the project, which matters because opening a Maven project runs code from its build file. It then loads the project and downloads the libraries listed in pom.xml, which takes a minute or two the first time.

The IDE does not always notice the JDK installed by itself, in which case the editor shows the warning Project JDK is not defined, as in Figure 7. Click Setup SDK and select the detected Java 25 installation, as in Figure 8, and the warning disappears.

Figure 7. The editor warning that no project JDK is defined, with the Setup SDK link that resolves it.

Figure 8. Selecting the OpenJDK 25 installation that IntelliJ IDEA detected on the VM.

Earlier versions of this course added Lombok at this point, a library that generates getters, constructors, and similar boilerplate during compilation. It is no longer needed, because Java records give the same benefit with plain Java and without an extra plugin in the IDE.

Hello World as a REST service

A record for the response

Our service will answer an HTTP request with a small JSON document containing a greeting and a counter. The shape of that document is described by a data transfer object, or DTO, a class whose only job is to carry data across a boundary, and a record is the natural way to write one. Create a package named dto inside com.cst8277.demo and, in it, the following file.

package com.cst8277.demo.dto;

public record Greeting(long id, String content) { }

That single line declares an immutable class with two fields, a constructor that sets both, accessor methods named id() and content(), and sensible equals, hashCode, and toString methods. When Spring later converts a Greeting to JSON, it uses the accessors, so the response will contain exactly the keys id and content.

The controller

The controller is the class that receives HTTP requests and decides what to return. Create a second package, controllers, and in it the class below, which answers GET requests to the root URL with a greeting and counts the requests it has served.

package com.cst8277.demo.controllers;

import com.cst8277.demo.dto.Greeting;
import java.util.concurrent.atomic.AtomicLong;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class GreetingController {

    private static final String TEMPLATE = "Hello, %s!";
    private final AtomicLong counter = new AtomicLong();

    @GetMapping("/")
    public Greeting greet(
            @RequestParam(defaultValue = "World") String name) {
        return new Greeting(counter.incrementAndGet(),
                            TEMPLATE.formatted(name));
    }
}

The annotation @RestController tells Spring that this class handles web requests and that whatever its methods return should be written into the response body, converted to JSON, rather than interpreted as the name of an HTML page. The annotation @GetMapping("/") maps GET requests for the root URL to the greet method, and @RequestParam reads the query parameter name from the URL, substituting World when the parameter is absent. The counter is an AtomicLong rather than a plain long because Spring serves requests on many threads at once, and an atomic counter guarantees that two simultaneous requests never receive the same number. Remember that counter, because later it will tell us something important about how Kubernetes distributes requests.

Health probes

The last piece of code is a line of configuration. Spring Boot Actuator can report two kinds of health that Kubernetes understands, liveness, which says whether the application is still working at all, and readiness, which says whether it is ready to receive traffic. Spring Boot switches these probes on automatically when it detects that it is running inside Kubernetes, and the line below switches them on everywhere, so that we can also see them when the application runs in the IDE. Open src/main/resources/application.properties, which Spring Initializr has already created with the application's name, and make it read as follows.

spring.application.name=demo
management.endpoint.health.probes.enabled=true

Running it

Figure 9 shows the finished project in IntelliJ IDEA, with the two new packages beside the generated DemoApplication class. The figure also shows a Dockerfile and a file named demo-app.yaml in the project root, which we write a bit later.

Figure 9. The project in IntelliJ IDEA, with the controllers and dto packages under com.cst8277.demo and GreetingController open in the editor.

Run the application by clicking the green Run button next to DemoApplication. The Run panel at the bottom of the window shows Spring Boot's log, in which the application reports that its embedded web server is listening on port 8080, and, for Spring Boot projects, tabs that list the application's beans, its health, its request mappings, and its environment, as Figure 10 shows.

Figure 10. The application running in the IDE, with the Spring Boot log in the Console tab and the Beans, Health, Mappings, and Environment tabs beside it in the Run panel.

Now open http://localhost:8080/?name=CST8277 in Firefox. The browser displays the JSON in Figure 11, whose content is "Hello, CST8277!" and whose id is 1. Reload the page, and the id grows by one each time, because the same controller object, and therefore the same counter, serves every request. Remove the parameter from the URL, and the greeting becomes "Hello, World!", the default that @RequestParam supplied.

Figure 11. The service's JSON response in Firefox, containing the request counter and the greeting built from the name parameter.

Before continuing, stop the application in the IDE with the red Stop button. That is necessary as the next section starts the same application in a container that also listens on port 8080, and only one program can listen on a port at a time, so an application left running in the IDE would make the container fail to start.

From JAR to image

A multi-stage Dockerfile

The second link in the chain turns the application into a container image, and the recipe for an image is a Dockerfile, a text file of instructions that Docker executes one after another. Create a file named Dockerfile in the project root, next to pom.xml, with the content below, which Figure 12 also shows in the editor.

# ---- build stage: compile and package with the full JDK ----
FROM eclipse-temurin:25-jdk AS build
WORKDIR /src
COPY .mvn .mvn
COPY mvnw pom.xml ./
RUN ./mvnw -q dependency:go-offline
COPY src src
RUN ./mvnw -q package -DskipTests

# ---- runtime stage: only the JRE and the application ----
FROM eclipse-temurin:25-jre
RUN useradd --system --uid 10001 app
WORKDIR /app
COPY --from=build /src/target/*.jar app.jar
USER 10001
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "app.jar"]

Figure 12. The multi-stage Dockerfile in the project root.

The file describes two images, as we saw in the containerization module, and only the second becomes the result. The first stage, named build, starts from eclipse-temurin:25-jdk, a full Java 25 development kit, and compiles the application with the Maven wrapper mvnw, a script that Spring Initializr placed in the project so that the build uses a known Maven version regardless of what is installed. The order of its instructions is deliberate. The build file is copied and the dependencies are downloaded before the source code is copied, because Docker caches each step and reuses it as long as its inputs have not changed, so that editing a Java file later reruns only the compilation and not the much slower download of every library.

The second stage starts afresh from eclipse-temurin:25-jre, which contains only the Java runtime, and copies in nothing from the first stage except the finished JAR. The compiler, Maven, the downloaded libraries, and the source code all stay behind in the discarded build stage, which makes the final image smaller and leaves an attacker who might break into the container with far fewer tools. The same reasoning explains the useradd and USER 10001 lines, which make the application run as an ordinary, unprivileged user instead of as root, so that a flaw in the application cannot easily become control of the container. The last two instructions document the port the application listens on and the command that starts it.

Building, trying, and pushing the image

The commands below build the image, start a container from it to check that it works, remove that container, and finally push the image to the registry that Part 2 created.

cd ~/projects/demo
docker build -t localhost:32000/cst8277/demo-app:1.0 .
# try the container locally
docker run -d --name demo -p 8080:8080 \
    localhost:32000/cst8277/demo-app:1.0
curl 'http://localhost:8080/?name=Docker'
docker rm -f demo
# publish it to the cluster's registry
docker push localhost:32000/cst8277/demo-app:1.0

The name given with -t deserves a close look, because it does two jobs at once. The part before the first slash, localhost:32000, tells Docker which registry the image belongs to, so that docker push later knows where to send it without being told again. The part after it, cst8277/demo-app, names the image, and the tag after the colon, 1.0, names this particular version. The final dot tells Docker to use the current folder, with its Dockerfile and source code, as the build context. Figure 13 shows the build, in which the build stage runs Maven and the runtime stage copies in the JAR.

Figure 13. Building the image, in which the build stage runs Maven inside the JDK image and the runtime stage copies the resulting JAR into the JRE image.

The docker run command starts the image in the background, because of -d, under the name demo, and connects port 8080 of the VM to port 8080 of the container, because of -p 8080:8080. Spring Boot needs a few seconds to start, so if curl reports that the connection was reset or that the server returned an empty reply, wait a moment and try again. Figure 14 shows the image and the running container, together with the answer "Hello, Docker!".

Figure 14. The image listed with a content size of 135 MB and a disk usage of 534 MB, the container running with port 8080 published, and the service answering with "Hello, Docker!".

The two sizes in the image listing measure different things. The content size of 135 MB is roughly what has to travel over the network when the image is pushed or pulled, since image layers are stored and transferred in compressed form, while the disk usage of 534 MB is the space the unpacked layers occupy on the VM. Either figure is far below what a single-stage image containing the full JDK and Maven would need, and that difference is what the second stage of the Dockerfile buys.

Once the test succeeds, docker rm -f demo stops and removes the container, freeing port 8080 again, and docker push uploads the image to the registry, as Figure 15 shows. From this moment the image exists in a place where Kubernetes can find it.

Figure 15. Pushing the image to the MicroK8s registry at localhost:32000.

We are now halfway along the chain in Figure 1, with source code, JAR, image, and registry behind us, and only the pods and the Service still to come.

From image to Service

Describing the desired state

In Part 2 we started nginx with imperative commands. This time we write the desired state into a file, so that it can be read, reviewed, kept in version control, and applied again on any cluster. Create demo-app.yaml in the project root with the content below. It holds two objects separated by the line ---, a Deployment that runs two replicas of our image and a Service that makes them reachable on port 30080 of the node.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  labels:
    app: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: demo-app
          image: localhost:32000/cst8277/demo-app:1.0
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }
---
apiVersion: v1
kind: Service
metadata:
  name: demo
  labels:
    app: demo
spec:
  type: NodePort
  selector:
    app: demo
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 30080

The file is long, but every part of it answers a question that Kubernetes needs answered. The Deployment asks for replicas: 2, two identical pods, and its template describes what each pod looks like, including the label app: demo that every pod receives. The selector tells the Deployment which pods it is responsible for, namely those carrying that same label, and the Service at the bottom uses the same label to decide which pods should receive its traffic. Labels and selectors are therefore the glue of the whole file, and a mismatch between them is a classic reason why a Service answers nothing.

The container section names the image we pushed, down to its registry and tag, and declares the two probes. Kubernetes calls the readiness endpoint and sends traffic to a pod only after it answers successfully, so that a pod still starting up never receives requests it cannot serve. It calls the liveness endpoint throughout the pod's life and restarts the container if that endpoint stops answering, which is how a hung application gets a fresh start without anyone noticing. The resources block tells the scheduler that each pod needs at least a tenth of a processor core, written 100m, and 256 MiB of memory, and it caps the memory at 512 MiB, beyond which the container is stopped rather than allowed to starve its neighbors. Finally, the Service is of type NodePort, as in Part 2, but this time it fixes the node port at 30080 instead of letting Kubernetes choose one.

Applying it and calling the service

Apply the file, wait for the rollout to complete, and list what was created. Then call the service four times, and finally delete both pods and look at what remains.

kubectl apply -f demo-app.yaml
kubectl rollout status deployment/demo
kubectl get pods,svc -l app=demo -o wide
# call the service several times
for i in 1 2 3 4; do
  curl -s 'http://localhost:30080/?name=Kubernetes'; echo
done
# self-healing: delete the pods and watch them come back
kubectl delete pod -l app=demo --wait=false
kubectl get pods -l app=demo

Figure 16 shows the first three commands. The rollout reports that first zero, then one, then both of the two replicas became available, the pod listing shows two pods with their own cluster addresses on the node cst8277-vm, and the Service listing shows the mapping 8080:30080/TCP together with the selector app=demo.

Figure 16. The deployment rolled out, with two running pods, each with its own address on the node cst8277-vm, and a NodePort Service mapping port 30080 to port 8080 of the pods selected by app=demo.

What the counter reveals

Figure 17 shows the rest, and it rewards a careful look.

Figure 17. Four requests answered with the ids 1, 1, 2, and 3, which shows that two pods with separate counters shared the requests, followed by the deletion of the pods and their replacement by new pods, one of them not yet ready.

When the application ran in the IDE, the id rose by exactly one with every request. Here the four answers carry the ids 1, 1, 2, and 3, which can only mean that the requests were served by two different pods, each holding its own counter in its own memory, with the Service distributing them between the two. We are watching horizontal scaling at work, and we are also watching its most important consequence. A counter kept in memory belongs to one replica and is invisible to the others, so any state that must be shared, whether a counter, a shopping cart, or a login session, has to live outside the pods, in a database or a cache, and the pods themselves should be stateless.

The second half of the figure repeats the experiment from Part 2 with our own code. After the pods are deleted, the Deployment immediately creates new ones, and the listing catches one of them still starting, shown as 0/1 in the READY column, because its readiness probe has not yet succeeded. Until it does, the Service sends that pod no traffic at all, so a client calling the service during the restart is answered by the pod that is already ready.

Watching it in Headlamp

The same events can be followed in Headlamp, which Part 2 installed. If the port forward from Part 2 is no longer running, for example because the VM has been restarted since, start it again, create a fresh token if the old one has expired, and open http://localhost:8888 in Firefox.

kubectl -n kube-system create token headlamp-admin
kubectl -n kube-system port-forward svc/headlamp 8888:80

The cluster overview in Figure 18 summarizes the resources of the cluster, which in the screenshot are four processor units, about 7 GB of memory, eight pods, and one ready node, and lists recent warning events. Among them you will probably find readiness probes that failed with "connection refused" while the demo pods were starting, which is the same moment the 0/1 in Figure 17 captured, recorded this time as an event rather than as a line of command output.

Figure 18. The Headlamp cluster overview, with one ready node and eight pods, and four warning events reporting readiness probes that were refused while the demo pods were starting.

The Workloads page in Figure 19 counts the controllers in the cluster by kind. It shows eight healthy pods, six Deployments with their six ReplicaSets, which are the system components, the registry, Headlamp, and our own demo, and the single DaemonSet that runs Calico on the node, and a ring beside each count shows what share of them is running. Below the summary it lists the individual workloads. The Pods page in Figure 20 lists every pod in every namespace, including our two demo pods beside the system pods, the registry, and Headlamp itself.

Figure 19. The Workloads page, with eight healthy pods, six Deployments, six ReplicaSets, and one DaemonSet, all of them running.

Figure 20. The Pods page, listing every pod in the cluster, among them the two demo pods.

Finally, the Map in Figure 21 draws the cluster as a picture, grouped by namespace. The system components, from CoreDNS and Calico to Headlamp and the storage provisioner, sit together in kube-system, the registry has a namespace of its own, and our demo Deployment sits in default beside the kubernetes Service that leads to the API server. Each card can be expanded to show the objects that belong to it, which makes the Map a good place to return to whenever a listing on the command line becomes hard to read.

Figure 21. The Map view grouped by namespace, with the system components in kube-system, the registry in container-registry, and the demo Deployment in default.

When it breaks

Sooner or later a deployment fails, and the first command to run is always kubectl get pods, because the status of the pods says which link of the chain has broken. Figure 22 turns that observation into a decision guide.

Figure 22. A decision guide for troubleshooting, which starts from the pod status reported by kubectl get pods and leads to the likely cause and the next command for each status.

A pod in the state ImagePullBackOff could not obtain its image, which nearly always means that the image name or tag in the YAML file does not match what was pushed, or that the push itself failed, so the remedy is to compare the two names character by character and to repeat the docker push. A pod in CrashLoopBackOff has an image that starts but exits almost at once, and Kubernetes keeps restarting it, waiting longer each time, exactly as Part 2 warned. The reason is in the application's own output, which kubectl logs followed by the pod's name displays. A pod that is Running but shows 0/1 in the READY column has a failing readiness probe, and kubectl describe pod shows the probe's recent results, which usually point to a wrong path or port. A pod stuck in Pending is waiting for memory or processor time that the node cannot provide, which calls for lower resource requests or, as Part 1 explained, a VM with more resources. Finally, a pod that is Running and 1/1 ready while curl on port 30080 is refused points away from the pod and toward the Service, whose selector and ports deserve a second look.

Two problems from earlier in this article belong on the same list, even though they occur before any pod exists. A "permission denied" message that mentions docker.sock means that the current login session is not yet in the docker group, which a log-out and log-in cures. A container or application that cannot start because port 8080 is already in use means that something else, usually the application still running in the IDE or a forgotten test container, holds the port, and stopping it solves the problem.

What you have built

The VM that began Part 1 as an empty disk now holds a complete, modern development environment, with Ubuntu 26.04 LTS running OpenJDK 25 and Maven, Docker, IntelliJ IDEA, and a MicroK8s 1.36 cluster with its own image registry and Headlamp. More important than the list of tools is the path we walked with them. We generated a project, wrote a REST controller with a record as its response, ran it in the IDE, packaged it with a multi-stage Dockerfile that leaves the compiler behind and runs as an unprivileged user, published the image to a registry, and deployed it declaratively to Kubernetes, where we watched requests spread across two replicas, pods replaced without anyone asking, and readiness probes holding traffic back from a pod that was not yet ready.

A Kubernetes Cluster of Your Own

Eugene Zimin — Sun, 25 Oct 2026 09:37:46 +0000

From running a container to keeping it running

In the module on containerization we ran containers with Docker, one command at a time, and watched each one start, answer a request, and stop. That way of working is perfect for a developer's laptop and hopeless for a production system, because it depends on a person being present at every moment something changes. If a container crashes at three in the morning, nobody restarts it, and if traffic doubles at noon, nobody starts a second copy, so the service is down, or slow, until someone notices.

Kubernetes removes the person from that loop by changing what we tell the system. Instead of issuing commands such as "start this container", we describe the state we want, for example "two copies of this image should always be running and reachable on this port", and Kubernetes compares that description with what is actually running and acts whenever the two differ. A thermostat works on the same principle, since it does not switch the heater on at a particular hour but compares the room's temperature with the one you asked for and acts on the difference, again and again, for as long as it is switched on. Kubernetes runs many such control loops at once, one for each kind of object it manages, and most of what it does can be understood as a loop of this kind noticing a difference and closing it.

The analogy has a limit worth stating now, because students meet it in Part 3. A thermostat cannot repair a broken heater, and Kubernetes cannot repair a broken program either. If an application crashes as soon as it starts, Kubernetes faithfully restarts it, watches it crash again, and keeps restarting it, more slowly each time, which is a symptom to diagnose rather than a fault in Kubernetes.

This article installs a complete Kubernetes cluster inside the virtual machine from Part 1, takes a first look at what is running in it, starts and deliberately damages a small workload to watch the control loop at work, and then adds the two tools that Part 3 needs, an image registry to deliver our own images to the cluster and a graphical interface to watch the cluster from a browser.

Choosing a Kubernetes that fits on one machine

A production cluster spreads across many machines, called nodes, so that the loss of any one of them does not take the service down. For learning, a single node is enough, and several projects package a full Kubernetes so that it runs on one computer. Minikube, the oldest of them, starts a cluster inside a VM or a container of its own, while kind, whose name abbreviates Kubernetes in Docker, runs each node as a Docker container and is popular in automated test pipelines. K3s is a lightweight distribution packaged as a single binary, aimed at small devices and edge computing, and MicroK8s, from Canonical, the company behind Ubuntu, installs a complete single-node cluster as one snap package.

All four would serve the course, and the choice is made here for convenience. Because our VM already runs Ubuntu, and because the snap system that Ubuntu uses for self-contained packages installs, updates, and removes MicroK8s with one command each, MicroK8s is the shortest path from an empty VM to a running cluster. It also ships optional add-ons, such as the image registry we need later, that are enabled with a single command rather than assembled by hand.

Installing MicroK8s

Open a terminal in the VM, either on its desktop with Ctrl+Alt+T or through the SSH connection described in Part 1, and run the commands below. The first installs MicroK8s, the second and third give your user account the right to control it without sudo, and the last waits until the cluster has started.

sudo snap install microk8s --classic --channel=1.36/stable
# let your user run microk8s without sudo, then log out and in
sudo usermod -a -G microk8s $USER
mkdir -p ~/.kube && sudo chown -R $USER ~/.kube
microk8s status --wait-ready

Each option in the first command has a purpose. The --classic flag gives the snap classic confinement, which means full access to the system rather than the sandbox most snaps run in, and Kubernetes needs that access because it manages networking, storage, and processes for everything that runs on the node. The --channel=1.36/stable option pins the installation to the stable releases of Kubernetes 1.36, so that automatic snap updates bring security fixes within that version but never move the cluster to a new version in the middle of the term. Figure 1 shows the installation completing.

Figure 1. MicroK8s 1.36 installed from the snap store.

The usermod command adds your user to the microk8s group, the group whose members may control the cluster, and the ~/.kube directory is where Kubernetes tools keep their configuration, which is why it must belong to you rather than to the administrator account. Group membership, however, is read only when you log in, so the change made by usermod does not yet apply to the terminal in which you typed it. Running microk8s status in that same terminal fails with the message "Insufficient permissions to access MicroK8s", which looks like the sign of a broken installation but only means that the old login session does not know about the new group.

The cure is to log out of Ubuntu completely and log in again. Older guides suggest the command newgrp microk8s as a shortcut, but it is no longer available on Ubuntu 26.04 Desktop, so the full log-out is the reliable way. In the new session, microk8s status --wait-ready returns once the cluster is up and prints its state, as Figure 2 shows.

Figure 2. The output of microk8s status, reporting that MicroK8s is running on a single node, with the dns, ha-cluster, helm, and helm3 add-ons enabled and a long list of others, including registry, available but disabled.

The output is worth reading line by line. The words "high-availability: no" say what we already know, that a single node cannot survive its own failure. The list of enabled add-ons includes dns, which runs CoreDNS, the service that lets programs in the cluster find one another by name, and the long list of disabled add-ons is a catalog of optional features, among them the registry add-on which we need to enable later.

Meeting the cluster

One command for the API

Every interaction with a Kubernetes cluster goes through its API server, and the standard client for that API is a command-line program called kubectl. MicroK8s bundles its own copy, invoked as microk8s kubectl, which guarantees that client and cluster have matching versions. Since every guide, book, and error message in the Kubernetes world writes plain kubectl, it is convenient to define an alias, so that the shorter name works too, and then to ask the cluster what it is running.

echo "alias kubectl='microk8s kubectl'" >> ~/.bash_aliases
source ~/.bash_aliases
kubectl get all --all-namespaces

The first line appends the alias to ~/.bash_aliases, a file that the Ubuntu shell reads each time a terminal opens, so the alias survives restarts. The second loads it into the terminal that is already open, and the third lists the main objects in every namespace of the cluster, producing the output in Figure 3.

Figure 3. The objects in a freshly installed cluster, where the kube-system namespace holds the Calico networking components and CoreDNS, each running as a pod, and the default namespace holds only the kubernetes service that leads to the API server.

Reading the first listing

Although we have not deployed anything yet, the cluster is far from empty, and its contents introduce most of the vocabulary that the rest of the series uses. The first column names a namespace, a label that divides a cluster into separate areas so that system components do not mix with applications. Everything Kubernetes needs to run itself lives in kube-system, and our own work goes into default.

The smallest objects in the listing are pods. A pod is one or more containers that Kubernetes schedules together, and in practice it is usually a single container wrapped in the information Kubernetes needs to manage it. Three pods are running here. Two of them, calico-node and calico-kube-controllers, belong to Calico, the networking layer that gives each pod an address of its own and lets pods reach one another, and the third, coredns, is the name service mentioned in section above.

Below the pods, the listing shows the objects that keep them alive, and here the control loop becomes visible. A Deployment, such as deployment.apps/coredns, records the desired state, which here is one ready copy of the CoreDNS pod. It does not create pods directly but through a ReplicaSet, whose only job is to keep the right number of identical pods running, replacing any that disappear. The calico-node pod is managed differently, by a DaemonSet, a controller that runs exactly one copy of a pod on every node, because each node needs its own networking agent. Finally, a Service, such as service/kube-dns, gives a stable address to a group of pods, so that clients do not need to know which pods currently exist or where they run.

A first workload

Starting it

The best way to see the control loop at work is to give it something to do. The commands in this section are not part of the permanent environment, and the section ends by removing everything they create, but they show in a few minutes what Kubernetes does with a workload, using the public nginx web server image so that no code of our own is involved yet.

These commands were not captured in the module's screenshots, so the output you see may differ slightly in names and numbers from the descriptions below. [TODO: verify on VM]

The first command creates a Deployment named hello that runs one pod with the nginx image, which Kubernetes downloads from Docker Hub. The second waits until that pod is ready, and the third creates a Service that makes the pod reachable from outside the cluster.

kubectl create deployment hello --image=nginx
kubectl rollout status deployment/hello
kubectl expose deployment hello --type=NodePort --port=80
kubectl get pods,svc -l app=hello

The type NodePort asks Kubernetes to open a port in the range 30000 to 32767 on every node and to forward traffic arriving there to the pods behind the Service. Because we did not choose a port, Kubernetes picks one at random, and the last command shows which. Its listing contains a line for the Service whose PORT(S) column reads something like 80:31234/TCP, where the number after the colon is the node port. The option -l app=hello filters the listing by label, a key and value that kubectl create deployment attached to the pods and the Service for us, and labels are also how the Service knows which pods belong to it.

5.2 Calling it, breaking it, and watching it heal

With the node port in hand, call the web server, replacing 31234 with the number your own cluster chose.

curl -s http://localhost:31234 | grep title

The answer should be the line <title>Welcome to nginx!</title>, taken from the server's default page, which shows that a request from the VM reached a container inside the cluster through the Service. Now comes the instructive part. Delete the pod, as if it had crashed, and list the pods again immediately.

kubectl delete pod -l app=hello
kubectl get pods -l app=hello

A pod is still listed, but its name ends in a different random suffix and its age is a few seconds, because the ReplicaSet noticed that the number of running pods had fallen from one to zero and created a replacement at once. Nobody told it to do so. The desired state still said one pod, the actual state said none, and the controller closed the gap. Changing the desired state works the same way in the other direction, as the next command shows by asking for three copies instead of one.

kubectl scale deployment hello --replicas=3
kubectl get pods -l app=hello

Within seconds three pods are listed, and the Service, which selects pods by label rather than by name, spreads requests across all of them without any further change.

The commands so far have been imperative, each one telling Kubernetes to perform an action, which is convenient for experiments but leaves no record of the intended state other than the cluster itself. In Part 3 we write the desired state down in a YAML file instead, the declarative style that real projects use, because a file can be reviewed, versioned, and applied again on any cluster. For now, remove the experiment so that the cluster is clean.

kubectl delete deployment,service hello

A registry for our own images

The nginx image came from Docker Hub, a public registry, which is a server that stores container images so that Docker and Kubernetes can pull them by name. Our own application, which we build in Part 3, has no place there, and publishing student code to a public registry would be neither necessary nor wise. A cluster can only run images it can pull from some registry, however, so we need a registry of our own, and MicroK8s provides one as an add-on.

microk8s enable registry      # image registry at localhost:32000

Enabling the add-on asks for your password, because it changes system settings, and then reports what it creates, as Figure 4 shows. The registry needs somewhere to keep its images, so MicroK8s first enables the hostpath-storage add-on, which stores data in a directory on the node, and warns that this kind of storage is not suitable for production, a warning that is true and harmless for a lab. It then creates a namespace called container-registry, a 20 GiB storage claim, and the registry itself, reachable on port 32000.

Figure 4. Enabling the registry add-on, which asks for the sudo password, enables hostpath storage, and creates a 20 GiB registry in the container-registry namespace.

From now on, an image whose name begins with localhost:32000/ refers to this registry. Docker, when we install it in Part 3, will push images there, and Kubernetes will pull them from the same address. Neither program asks for the TLS certificate that a public registry would need, because Docker treats any registry on localhost as trusted by default and MicroK8s configures its own container runtime to trust this one address, an arrangement that is acceptable inside a private VM and would not be anywhere else.

Seeing the cluster with Headlamp

Why a graphical view

The command line shows the cluster one object type at a time, and that is how experienced engineers usually work, but a beginner learns faster when the relationships between objects are drawn on a screen. Earlier versions of this course used the Kubernetes Dashboard for that purpose. The Dashboard project was archived in 2026, and the Kubernetes project now points users to Headlamp, an open-source, extensible web interface that is its official successor (Kubernetes, 2026).

Installing and opening Headlamp

Headlamp runs inside the cluster as an ordinary Deployment, so installing it means applying its manifest, the YAML file that describes its objects, straight from the project's repository. Logging in requires an identity, and Kubernetes gives programs and tools an identity called a service account. The commands below create one, grant it full rights over the cluster, issue a login token for it, and finally forward a port of the VM to Headlamp so that a browser can reach it.

kubectl apply -f https://raw.githubusercontent.com/\
kubernetes-sigs/headlamp/main/kubernetes-headlamp.yaml
# a service account to log in with (lab use only!)
kubectl -n kube-system create serviceaccount headlamp-admin
kubectl create clusterrolebinding headlamp-admin \
    --clusterrole=cluster-admin \
    --serviceaccount=kube-system:headlamp-admin
kubectl -n kube-system create token headlamp-admin
kubectl -n kube-system port-forward svc/headlamp 8888:80

A backslash at the end of a line tells the shell that the command continues on the next line, which is how the long URL and the long clusterrolebinding command fit on the page. The option -n kube-system places the service account in the same namespace as Headlamp, and the cluster role binding attaches the predefined role cluster-admin, which permits every action on every object, to that account. The create token command prints a long string, the token, which is valid only for a limited time, so when Headlamp later refuses it you run the command again for a new one. The last command keeps running and occupies its terminal for as long as the forward is needed, carrying connections from port 8888 of the VM to port 80 of the Headlamp Service inside the cluster.

Figure 5. Deploying Headlamp and creating its service account and cluster role binding, captured later in the series, which is why the prompt shows the project folder from Part 3 and the IDE appears in the dock, although the commands work the same from any folder.

Figure 6. A variant of the token and port-forward commands that saves the token to a file and runs the port forward in the background, so that the same terminal stays free for further commands.

Figure 6 shows a variation that some students find more convenient. It writes the token to a file, prints only its first 60 characters as a check, and starts the port forward in the background with &, so that it does not occupy the terminal. Either form works, and the plain commands above are easier to read when you meet them for the first time.

With the port forward running, open Firefox in the VM, go to http://localhost:8888, and paste the token into the login page shown in Figure 7.

Figure 7. The Headlamp login page, which accepts the service-account token printed by kubectl create token.

After logging in you see the cluster overview, with the health of the single node and of the pods listed from the command line, and a navigation bar for workloads, network objects, storage, and the cluster map. At this stage the picture is quiet, with only the system pods, the registry, and Headlamp itself, and Part 3 returns to it once our own application is running, so the screenshots of those pages appear there.

A warning about the cluster-admin binding

The binding created above gives anyone holding the token complete control of the cluster, including the ability to read every secret and delete every object. On a personal lab cluster inside a VM that nobody else can reach, that convenience is reasonable. On a shared or production cluster it would be a serious security hole, and such clusters grant each person and tool only the permissions their work requires.

When the cluster misbehaves

Two problems account for most of the trouble at this stage. The first, the message "Insufficient permissions to access MicroK8s", appears in every terminal opened before you logged out and back in after the usermod command, and it disappears in a new session. The second is pods that remain in the Pending state for a long time, which means that Kubernetes cannot find a node with enough free memory or processor time to run them. On a single-node cluster that node is the VM itself, so the remedy is the one given in Part 1, to power off the VM, give it more memory or processors, and start it again, or, when an application is involved, to run fewer replicas of it.

A third problem looks alarming but is entirely expected. When the VM restarts, the port forward to Headlamp ends with it, and the browser reports that the connection to localhost:8888 was refused. Headlamp is still installed and running in the cluster, so the remedy is to run the port-forward command again and, if the token has expired, to create a new one.

What comes next

The VM from Part 1 now holds a working Kubernetes cluster. It runs a network layer and a name service, can host workloads that heal themselves and scale on request, keeps a private registry ready to receive images, and can be inspected through a browser as easily as from the command line. Every image the cluster has run so far, however, was written by somebody else. Part 3 closes that gap by installing a Java toolchain, Docker, and an IDE, writing a small Spring Boot service, packaging it as an image, pushing that image to the registry we have just created, and deploying it with a YAML file, so that the self-healing we watched with nginx happens to code of our own.

A Computer Inside Your Computer

Eugene Zimin — Sun, 25 Oct 2026 09:37:31 +0000

Why we begin with an empty machine

In the second or third week of almost every programming course that asks students to run real infrastructure, the room divides in two. Half the class has a working setup and has already moved on to the exercise, while the other half is staring at an error that mentions a port already in use, a library of the wrong version, or a permission that some earlier, half-remembered installation quietly changed. The code on both sides of the room is usually identical, which means the difference lies underneath it, in the layers of software that each laptop has accumulated over the years and that nobody in the room can fully describe.

The remedy this series adopts is to stop installing course software on your own computer at all and to build, instead, a second computer that lives inside the first. That second computer, a virtual machine, starts completely empty, receives exactly the tools the course needs and nothing else, and behaves the same way on a Windows laptop, a MacBook, and a Linux desktop. If it is ever damaged beyond repair, it can be deleted and rebuilt in under an hour without touching a single file of your own, a freedom that makes it much easier to experiment, to break things on purpose, and to learn from what breaks.

Figure 1 shows the whole environment that the three articles of this series assemble. The outer frame is your own computer, which from now on we call the host. Inside it runs VirtualBox, and inside VirtualBox runs a virtual machine with Ubuntu Linux, which will in turn hold a Java toolchain, Docker, an IDE, a browser, and a small Kubernetes cluster that runs our application. This first article builds the two outer layers, the hypervisor and the operating system, and leaves the rest of the picture for the parts that follow.

Figure 1. The complete lab environment, in which VirtualBox on your own computer hosts a single Ubuntu virtual machine that contains every tool and the Kubernetes cluster used for the rest of the course.

The work is divided into eight steps, which Figure 2 lays out in order. This article covers the first two, installing VirtualBox and creating the Ubuntu virtual machine, Part 2 covers the third, and Part 3 covers the remaining five. Each step ends with a small check, such as a version number, a login screen, or an HTTP response, so that a problem is caught in the step that caused it rather than three steps later, when its cause has become much harder to find.

Figure 2. The eight steps of the series, of which this article covers the first two.

The environment is not a throwaway exercise. Every later module of the course, from security to debugging, continuous delivery, and observability, reuses this machine, so the time spent building it carefully now is repaid many times over.

What a virtual machine is

A computer simulated in software

A virtual machine, usually shortened to VM, is a computer whose processor, memory, disk, network card, and screen are provided by a program rather than by physical hardware. The operating system installed inside it, which is called the guest, believes it is running on an ordinary computer, boots from what it takes to be a DVD, writes to what it takes to be a hard disk, and talks to what it takes to be a network. All of those devices are in fact files and processes managed by a program on the host, and that program is called a hypervisor.

Hypervisors come in two broad kinds. A type 1 hypervisor runs directly on the hardware in place of an ordinary operating system, as happens in the data centers of cloud providers, whereas a type 2 hypervisor runs as an application on top of the operating system you already use, next to your browser and your editor. VirtualBox, a free and open-source hypervisor maintained by Oracle, belongs to the second kind, which is exactly what a student laptop needs, because it can be installed and removed like any other program.

The image of a computer inside a computer is accurate in one important respect and misleading in another. It is accurate because the guest is genuinely isolated, so that nothing you install or delete inside Ubuntu can damage the files of your host. It is misleading if it suggests that the guest has resources of its own, because every virtual processor and every gigabyte of virtual memory is borrowed from the host for as long as the VM runs. A VM given too much starves the host, and a VM given too little crawls, which is why the next section is about choosing those numbers carefully.

How much of your computer to lend

A good rule is to give the VM about half of your computer's resources. The setup used throughout this series assigns 4 processor cores, 8 GB of memory, and 60 GB of disk space, which leaves enough room for Kubernetes, Docker, and an IDE to run at the same time without long pauses. The environment also works, though noticeably more slowly, with 2 cores, 6 GB of memory, and 40 GB of disk, which is the practical minimum. The disk figure is less frightening than it looks, because, as we shall see, the virtual disk occupies only as much real space as the guest has actually written.

Two kinds of processor, two editions of Ubuntu

The last decision to make before downloading anything concerns the processor in your computer. VirtualBox 7.2 supports both the Intel and AMD processors found in most Windows and Linux computers and the Apple silicon chips found in recent Macs, but these two families speak different machine languages, and a guest operating system must be built for the language of the host. On an Apple silicon Mac you therefore need the ARM64 edition of Ubuntu, while on every other computer you need the AMD64 edition. The screenshots in this series were taken on an Apple silicon Mac and so show ARM64 file names, but every command and every screen of the installer is identical on AMD64.

Creating the virtual machine

Two downloads

Two files are needed before VirtualBox can create anything. The first is VirtualBox itself, which you download for your operating system from virtualbox.org and install with the default options, since it is free for personal and educational use. The second is the Ubuntu 26.04 LTS desktop installer, distributed as an ISO image, a single file that contains a complete copy of an installation DVD, which you download from ubuntu.com/download/desktop. LTS stands for Long Term Support, a promise that Canonical, the company behind Ubuntu, will publish security updates for this release for at least five years, which is why LTS releases are the usual choice for servers and for a course that should not change under its students' feet.

VirtualBox is not the only possible host for the VM. VMware Workstation Pro and VMware Fusion Pro are also free for personal use, and Parallels Desktop is a common choice on Macs, so if anyone who already uses one of them can keep it, since the Ubuntu installation that follows is the same in all of them. The screenshots and settings in this article, however, are those of VirtualBox.

Naming the machine and choosing the image

With both files downloaded, start VirtualBox and click New. The dialog that opens, shown in Figure 3, asks first for a name, which we set to Ubuntu, and then for the ISO image, which you select from your downloads folder. VirtualBox reads the image and fills in the operating system fields by itself. Recent VirtualBox versions may not list Ubuntu 26.04 yet, in which case they show the closest version they know, here Ubuntu 25.04, which is harmless because the field only selects sensible default settings.

One checkbox on this page needs your attention. VirtualBox offers to Proceed with Unattended Installation, which would install Ubuntu silently with settings of its own choosing, and the box is checked by default, as it still is in the screenshot. Clear it, so that you follow the ordinary Ubuntu installer yourself, because the choices that installer asks you to make are exactly the ones you need to understand when something goes wrong later.

Figure 3. The first page of the New Virtual Machine dialog, where the VM is named Ubuntu and the ARM64 ISO is selected, and where the Proceed with Unattended Installation box, still checked here, must be cleared before continuing.

Processors, memory, and disk

Open the section Specify virtual hardware and enter the resources chosen earlier, which means 8192 MB of base memory and 4 CPUs, as in Figure 4. Keep Use EFI enabled. EFI is the modern firmware that a computer runs before its operating system starts, the successor of the older BIOS, and Ubuntu on ARM64 in particular expects to find it.

Figure 4. The virtual hardware section, with 8192 MB of memory, 4 processors, and EFI enabled.

Next, open Specify virtual hard disk and set the size to 60 GB, as in Figure 5. The disk is dynamically allocated, which means that VirtualBox creates a file on your computer that starts small and grows only as Ubuntu actually writes data, up to the 60 GB limit, so a freshly installed VM occupies far less of your real disk than its nominal size suggests.

Figure 5. The virtual hard disk section, where a dynamically allocated 60 GB disk is created for the VM.

Click Finish, and the new machine appears in the VirtualBox Manager, as Figure 6 shows. The details pane is worth reading once, because it confirms each decision made so far, with 8192 MB of memory, four processors, EFI enabled, a 60 GB disk file named Ubuntu.vdi, the Ubuntu ISO inserted into a virtual optical drive, and a single network adapter attached to NAT, a setting we return later on.

Figure 6. The new Ubuntu VM in the VirtualBox Manager, powered off, with its memory, processors, disk, installation ISO, and NAT network adapter listed in the details pane.

Select the VM and click Start. VirtualBox opens a window that plays the role of the VM's screen, and the machine boots from the ISO image just as a physical computer would boot from an installation DVD.

Installing Ubuntu 26.04 LTS

From the boot menu to the installer

The first thing the VM displays is the GRUB menu of the installation image, shown in Figure 7. GRUB is the boot loader that Ubuntu uses to choose what to start, and here it offers a few ways to start the installation medium. Choose Try or Install Ubuntu and press Enter.

Figure 7. The GRUB boot menu of the Ubuntu installation image, where Try or Install Ubuntu starts the installer.

After a short wait the graphical installer appears and guides you through a series of pages. The first four ask for little thought, and Figures 8 to 11 show them in order. You choose the language of the installer and the keyboard layout that matches your physical keyboard, and then the installer reports a wired network connection. That connection is not a cable at all, but the virtual network card VirtualBox provides, which the guest experiences as an Ethernet port with internet access through the host. On the fourth page, choose Install Ubuntu rather than Try Ubuntu.

Figure 8. The language page of the installer.

Figure 9. The keyboard page, where the layout should match the keyboard you actually type on.

Figure 10. The network page, which reports a wired connection because the guest sees VirtualBox's virtual network card as an Ethernet port.

Figure 11. The page that chooses between trying Ubuntu from the image and installing it, where Install Ubuntu is the correct choice.

What to install

The next three pages decide what goes onto the disk. Choose Interactive installation, shown in Figure 12, which walks through the remaining questions one by one, and then the Default selection of applications in Figure 13, which installs the desktop, a browser, and basic utilities without the office tools of the Extended selection. The page in Figure 14, titled Optimise your computer, offers third-party graphics and Wi-Fi drivers and support for additional media formats. Leave both boxes empty, because the VM has no physical graphics card or Wi-Fi adapter of its own, so such drivers would have nothing to drive.

Figure 12. The installation type page, where Interactive installation lets you make each remaining choice yourself.

Figure 13. The applications page, where the Default selection installs a lean desktop.

Figure 14. The page offering proprietary drivers and media codecs, both left unchecked because a VM has no physical graphics or Wi-Fi hardware.

The disk page and why it is safe

The page in Figure 15 is the one that makes students hesitate, because it offers to Erase disk and install Ubuntu, and on a physical computer that sentence would mean losing everything on it. Inside a VM the meaning is different. The only disk the guest can see is the empty 60 GB virtual disk, which is a file on your host, so erasing it erases only that empty file and cannot reach a single byte of your own data. Choose this option with a clear conscience.

Figure 15. The disk setup page, where Erase disk and install Ubuntu refers only to the empty virtual disk created for this VM.

The installer then offers to encrypt the disk, as Figure 16 shows. Encryption protects a laptop that might be stolen, but a lab VM holds no personal data and would ask for an extra passphrase on every boot, so choose no encryption.

Figure 16. The encryption page, where no encryption is the sensible choice for a lab VM.

Account, time zone, and review

On the page in Figure 17, create your user account. The examples in this series use the user name student and the computer name cst8277-vm, and you will see both in every terminal prompt of the screenshots that follow, in the form student@cst8277-vm. You may choose other names, but using the same ones makes it easier to compare your screen with the figures. Choose a password you will remember, since Ubuntu will ask for it whenever a command needs administrator rights.

Figure 17. The account page, where the user student and the computer name cst8277-vm are created.

Choose your time zone next, as in Figure 18, where Toronto has been selected. The installer then summarizes every choice on a review page, shown in Figure 19, which is the last moment at which anything can be changed without starting over. Read it, and click Install.

Figure 18. The time zone page, with Toronto and the America/Toronto zone selected.

Figure 19. The review page, which lists every choice made so far before the installation begins.

The installation takes about ten minutes, during which the installer shows a progress page like the one in Figure 20. When it finishes, it announces that the installation is complete and offers Restart now, as in Figure 21.

Figure 20. The installation in progress.

Figure 21. The installer reporting that the installation is complete, with the Restart now button.

The first boot

When you click Restart now, the VM shuts down the installer and, before starting again, asks you to remove the installation medium and press Enter, as Figure 22 shows. On a physical computer you would take the DVD out of the drive, which is necessary because otherwise the machine would boot into the installer once more. In VirtualBox there is nothing to remove by hand, because pressing Enter is enough and VirtualBox ejects the virtual DVD by itself.

Figure 22. The request to remove the installation medium, which in VirtualBox only requires pressing Enter.

Ubuntu now boots from its own virtual disk for the first time and shows the login screen in Figure 23. Log in with the password you chose on the account page.

Figure 23. The Ubuntu login screen with the student account.

A welcome assistant greets you after the first login, as in Figure 24, and walks you through a few optional settings, which you may accept or skip as you like. In the screenshot a notification has already appeared that 74 updates are available, a reminder that an ISO image is a snapshot of the day it was built and that the packages it installed have been revised many times since.

Figure 24. The welcome assistant of Ubuntu 26.04.1 LTS, with a notification that software updates are already available.

Once the assistant closes, you have a working Ubuntu desktop, shown in Figure 25, with the dock of applications on the left and, at the bottom of the dock, the button that shows every installed application.

Figure 25. The Ubuntu 26.04 LTS desktop, ready for use.

The most important application on that desktop, for the rest of this series, is the terminal, which you open by pressing Ctrl+Alt+T. Its first job is to bring the system up to date. You can accept the updates offered by the Software Updater, or run the two commands below, which first refresh the list of available packages and then install the newer versions of everything already installed.

sudo apt-get update && sudo apt-get upgrade

The word sudo in front of each command runs it with administrator rights, which is why Ubuntu asks for your password the first time. The && between the two commands means that the second runs only if the first succeeds, a small habit that prevents a failed update from being followed by an upgrade based on stale information. If the upgrade replaces the kernel, Ubuntu will suggest a restart, and it is worth accepting before you continue.

Optional work from your own terminal

How the VM reaches the network

The VM window is a perfectly good place to work, but typing long commands into it can be tedious, and copying and pasting between the host and the guest does not always work smoothly. Many students therefore prefer to work from a terminal on their own computer and connect to the VM over SSH, the standard protocol for logging in to a remote Linux machine. Doing so requires one small change to how the VM is connected to the network, and understanding that change also explains how the VM reached the internet during the installation.

The network adapter we saw in Figure 6 is attached to NAT, which stands for Network Address Translation. In this mode VirtualBox acts like the router in a home network, giving the guest a private address of its own, usually 10.0.2.15, and forwarding the guest's outgoing connections to the internet under the host's address. That is why apt-get, the snap store, and Docker Hub can all be reached from inside the VM without any configuration. The same arrangement has a consequence that surprises many students, which is that connections cannot be opened in the other direction, from the host to the guest, unless we tell VirtualBox explicitly which port on the host should lead to which port in the guest. Figure 26 shows such a rule, called a port forward, for SSH.

Figure 26. VirtualBox NAT networking, in which the guest reaches the internet through the host, and a port-forwarding rule from 127.0.0.1:2222 on the host to port 22 in the guest makes SSH possible in the opposite direction.

Setting up the port forward

There are two equivalent ways to create the rule. In the VirtualBox Manager, open the VM's settings, go to Network, then to Adapter 1 (NAT), then to Port Forwarding, and add a rule from host address 127.0.0.1 and host port 2222 to guest port 22. Alternatively, while the VM is powered off, run the following command in a terminal on your host.

VBoxManage modifyvm Ubuntu --natpf1 "ssh,tcp,127.0.0.1,2222,,22"

The quoted argument lists, in order, a name for the rule, the protocol, the host address and port, an empty guest address that VirtualBox fills in itself, and the guest port. Binding the rule to 127.0.0.1 rather than to every address of the host means that only programs on your own computer can use the forward, and not other machines on the same Wi-Fi network.

The rule is useless until something inside the guest is listening on port 22, and the default Ubuntu desktop does not include an SSH server. Start the VM, install the server from its terminal, and then connect from a terminal on the host.

# inside the VM
sudo apt-get install -y openssh-server

# on your own computer
ssh -p 2222 student@127.0.0.1

The -p 2222 option tells the SSH client to use the forwarded port rather than the standard port 22, and VirtualBox carries the connection through to the guest. From here on, every command in this series can be typed either in the VM's own terminal or in this SSH session, whichever you find more comfortable, with the exception of graphical programs such as the browser and the IDE, which must run on the VM's desktop.

When the VM misbehaves

A VM fails in ways a physical computer does not, and three problems account for most of the trouble students report at this stage. The most common is a VM that is very slow, which almost always means it has too little memory or too few processors for what it is being asked to do. Power the VM off, since VirtualBox refuses to change these settings while it runs, raise the values toward the 4 cores and 8 GB, and start it again. Later in the series, when Kubernetes runs inside the VM, the same shortage shows up as application pods that stay in the Pending state, and the remedy is the same.

The second problem is a VirtualBox window that freezes or a VM that crashes, which is most often caused by an outdated version of VirtualBox on a recently updated host. Updating VirtualBox usually cures it. If the window itself is what freezes, saving the VM's state, or starting the VM headless from the VirtualBox Manager and working entirely over the SSH connection sidesteps the problem.

The third problem is the hardest to diagnose, because the VM fails to boot from the installation image, without any message that names the cause. The cause is nearly always an ISO of the wrong edition, AMD64 on an Apple silicon Mac or ARM64 on an Intel or AMD computer, and the remedy is to download the edition that matches the processor and to select it in the VM's storage settings.

What you have now

You now have a complete Ubuntu 26.04 LTS computer that runs inside your own, isolated from your files, sized for the work ahead, and reachable both from its own desktop and, if you set up the port forward, from your own terminal. Before moving on, it is worth confirming two things from a terminal in the VM. The first command below prints the name and version of the operating system, which should report Ubuntu 26.04 LTS, and the second shows the size and free space of the root file system, which should be close to the 60 GB you assigned, minus the space the installation already uses.

lsb_release -a
df -h /

The machine is still, in a sense, empty, since it runs a desktop and a browser and nothing that resembles an enterprise application platform. The earlier modules of the course described such a platform in theory, with containers grouped into pods, pods managed by an orchestrator that restarts them when they fail, and services that route traffic to whichever pods happen to be alive. Part 2 installs that orchestrator, Kubernetes, inside the VM we have just built, so that from then on the theory can be watched at work.

A Clean and Fast macOS Terminal with Ghostty, Starship, and Antidote

Eugene Zimin — Sat, 24 Oct 2026 20:10:36 +0000

My terminal is where I spend most of my working day, yet for years my setup was one long .zshrc, a full framework I only used a fraction of, and a home folder full of hidden files I couldn't explain. In the last few weeks, I sat down and rebuilt it from the ground up around three small tools: Ghostty, starship, and antidote.

So, below, you will find a short walkthrough of that setup, along with a few suggestions for keeping your home folder clean without breaking anything. All config files are in my development-utilities repository. In my experience, reading through a few small config files is still the best way to understand how a shell really starts up.

How Ghostty, zsh, antidote, and starship fit together

As a small side note, and since these names often get mixed up, here's how the pieces relate.

If we think of the setup as a car, zsh is the engine. antidote is the parts manager that bolts plugins onto the engine (completion, syntax highlighting, git aliases). starship is the dashboard that shows where you are and what state your repo is in. And Ghostty is the body around it: the terminal emulator that draws everything quickly on the GPU.

Each of these can be swapped out on its own, which is the main reason I like this combination over an all-in-one framework.

Figure 1: The end result in a new Ghostty tab. A one-table system summary, followed by a two-line starship prompt.

Scope and prerequisites

This setup targets macOS (Apple Silicon or Intel) with Homebrew. Installing the tools takes one line each:

brew install --cask ghostty
brew install antidote starship

There's no need to install a patched font for the icons, since Ghostty bundles the Nerd Font symbols. (In iTerm2 or Terminal.app you would need a Nerd Font, otherwise the icons show up as boxes.)

1. Ghostty: the terminal

Ghostty is fast, native on macOS, and configured through a single plain-text file. Instead of the default ~/Library/Application Support/com.mitchellh.ghostty/ location, I keep that file at ~/.config/ghostty/config.ghostty, next to everything else. Here is a trimmed version of mine:

font-family = "Nordwand Mono"
font-size = 13
background-opacity = 0.92
background-blur-radius = 15
cursor-style = underline

keybind = global:cmd+grave_accent=toggle_quick_terminal

shell-integration = zsh
shell-integration-features = no-title,ssh-terminfo
copy-on-select = clipboard
mouse-hide-while-typing = true

Most of these lines are a matter of taste, but two of them changed how I work. The first is the keybind line, which turns Ghostty into a drop-down "quick terminal": wherever I am, whether in the browser, the editor, or a PDF, Cmd+Backtick slides a terminal down from the top of the screen, and the same key sends it away again. After a week, I stopped opening terminal windows for quick commands altogether.

The second is the shell integration line. I keep Ghostty's integration on, but switch off its tab titles with no-title, because I like my tabs to tell me a little more. A small zsh hook in conf.d/60-ghostty-title.zsh names each tab after the folder, the git branch, and the command that's running, so a tab reads, for example, acme-api - main - cargo. With five tabs open, that's the difference between finding the right one at a glance and clicking through all of them. The ssh-terminfo feature, finally, quietly teaches remote hosts about Ghostty's terminal type, so colours and keys keep working when I SSH into a server.

2. Keeping the home folder clean

By default, zsh reads ~/.zshrc, ~/.zprofile, ~/.zsh_history, and friends, all straight from your home folder. The trick is a single variable, ZDOTDIR, which tells zsh where to look instead.

The only zsh file left in ~ is a three-line ~/.zshenv:

# zsh reads this file before ZDOTDIR is known; everything else lives in ~/.config/zsh.
export ZDOTDIR="$HOME/.config/zsh"
[[ -r "$ZDOTDIR/.zshenv" ]] && source "$ZDOTDIR/.zshenv"

The nice part is that macOS plays along. /etc/zshrc sets HISTFILE=${ZDOTDIR:-$HOME}/.zsh_history, and Terminal's session files go to ${ZDOTDIR:-$HOME}/.zsh_sessions, so both follow you into ~/.config/zsh with no extra config.

Everything else now lives together in ~/.config/zsh: the .zshrc and .zprofile, the plugin list, and a conf.d/ folder with small numbered files for options, environment variables, aliases, and functions. Instead of scrolling through one long file, I open the one file that's about the thing I want to change. API keys and tokens get a file of their own, .zsh_secrets, readable only by me (chmod 600) and never copied into the repository, so I can share my dotfiles without worrying about leaking anything. Caches stay out of sight as well: starship and the greeting write to ~/.cache, and antidote keeps its plugin clones in ~/Library/Caches/antidote.

There's one catch I ran into myself. After the switch, the old ~/.zsh_history and ~/.zsh_sessions are still sitting in your home folder, but nothing reads them anymore, so your shell suddenly seems to have forgotten every command you ever typed. The fix is to move the history over before you open your first new shell (mv ~/.zsh_history ~/.config/zsh/), and then delete ~/.zsh_sessions. In the same spirit, keep in mind that an old ~/.zshrc is silently ignored once ZDOTDIR is set, which can be confusing if you forget it's there.

And if you'd like to go further, many other tools can be convinced to move out, too. Rust, for example, reads CARGO_HOME and RUSTUP_HOME, and npm reads NPM_CONFIG_CACHE. The trick is to set these in $ZDOTDIR/.zshenv before installing the tool. Otherwise, you'll need to move the existing folders yourself.

Figure 2: The home folder before (Oh My Zsh and Powerlevel10k, everything in ~) and after the ZDOTDIR switch, where a single .zshenv is left. The listings are demo values.

3. Antidote: plugins without a framework

I used Oh My Zsh for years, but I only needed a handful of its files. antidote reads a plain list of plugins, builds a static bundle from it, and only rebuilds that bundle when the list changes, so startup stays fast.

getantidote/use-omz                      # Oh My Zsh pieces, without the framework
ohmyzsh/ohmyzsh path:lib/history.zsh
ohmyzsh/ohmyzsh path:plugins/git
marlonrichert/zsh-autocomplete           # completion menu as you type
zsh-users/zsh-syntax-highlighting        # must stay last

The first line, use-omz, is what makes this work. It lets me pick individual Oh My Zsh files, like the history settings or the git aliases I've had in my fingers for years, without loading the framework around them.

The order of the list matters more than it looks, though. zsh-autocomplete sets up completion on its own, so it needs to come after the Oh My Zsh pieces, and zsh-syntax-highlighting wants to see every other plugin first, so it always goes last. I learned both of these the hard way, so I left myself comments in the file.

Day to day, I hardly think about antidote at all. When I add a line to the list, the next shell picks it up, and every few weeks I run antidote update to pull the latest version of everything.

4. Starship: the prompt

Finally, starship draws a two-line prompt modelled on Powerlevel10k's "lean" style. Line one shows the path and git state. Line two holds the ❯ input marker, with status and project information on the right.

Figure 3: The prompt in different repos: local changes, a deep subfolder, a detached tag, a rebase in progress, and a read-only folder.

The first line is where the git information lives, and it's the part I'd miss most. In the top row of the figure, ⇡2 +1 !2 ?1 *1 tells me, without typing git status, that I have two commits I haven't pushed, one staged file, two modified files, an untracked file, and something stashed. A clean repo shows nothing but the branch name, so when the line is short, I know I'm done.

The path took the most work to get right. Long paths are hard to read, but cutting them down too far loses track of which project I'm in. So parent folders are shortened to a single letter, while the repo root and the current folder stay whole: deep inside a project, the prompt reads ~/D/acme-api/s/h/auth. Starship can't do this on its own, so a small pure-zsh function in prompt.zsh works out the pieces before each prompt.

The right side stays empty most of the time and only fills in when there's something worth saying, like a failed command's exit code, a command that ran longer than three seconds, or the package and language versions in a project folder. If you ever wonder what a symbol means or why the prompt feels slow, starship explain and starship timings will tell you.

Final words

You don't have to adopt all of this at once. In fact, I'd suggest starting with the ZDOTDIR change, since it's three lines and immediately tidies up your home folder. Then add antidote and starship one at a time and open a new tab after each step.

If you'd rather not copy files by hand, the installer in the same repo can restore just this part: run ./install-all-macos.sh --dry-run first, and select only the shell step.

Thanks for reading, and I hope you enjoy your new terminal!

Building Pace Maker or how I did load testing

Eugene Zimin — Mon, 19 Oct 2026 18:49:07 +0000

Most of the load tests I have run over the years were not grand capacity-planning exercises but small and slightly anxious questions asked late in the afternoon, such as whether a freshly deployed endpoint would survive a few hundred requests per second or whether a caching change had actually made a service faster, and almost every time the tooling around those questions felt heavier than the questions themselves. Just imagine the JMeter - nothing wrong, but I'd rather use a curl with a loop, than spend time on setting it up, even though it's powerful. Writing a scenario file, pulling a container image, or keeping a browser tab open on a hosted dashboard is perfectly reasonable for a scheduled performance suite, yet it is a lot of ceremony when all you want is to point something at a URL, decide how hard to push, and watch what happens. Brrr, no. I stopped and decided to make something which is free and much comfortable for me.

That's how Pace Maker, a small native macOS app written in Swift and SwiftUI, was appeared.

Why?

Thinking about who would actually reach for such a tool shaped almost every later decision, because the typical user is a backend developer who already knows the endpoint, has a rough idea of the expected traffic, and mostly needs a fast and trustworthy feedback loop rather than a report generator, which means the window should be organized around the one thing that person keeps looking at while the test runs. And that's exactly what I needed.

The application is simple yet powerful - its live chart owns the main area of the window, the settings live in a native inspector that can be hidden once a test is configured, and the controls that change the state of a run, together with the elapsed time and the number of requests sent, sit in the toolbar and the window subtitle, where they stay visible without competing with the data. Name it "focus mode", but that's definitely not a cockpit.

Separating pacing from concurrency

The most important architectural idea in the app is that the rate at which requests are released and the number of requests allowed to be in flight are two different concerns, and conflating them is the reason so many simple load generators quietly misreport what they measured. If a tool sends the next request only after a previous one returns, a slow server automatically slows the tool down, and the chart ends up showing whatever pace the server could tolerate instead of the pace you asked for, which hides precisely the degradation you were trying to observe.

Pace Maker therefore releases requests on a schedule derived from the target rate alone, and those requests enter a bounded backlog that is served by a fixed pool of workers, so nothing is created per request thus memory stays flat regardless of how long a test runs or how high the rate goes. At the same time between the backlog and the workers sits an adaptive connection gate, which decides how many requests may be in flight at any moment by reasoning roughly along the lines of Little's law, since the concurrency needed to sustain a rate is approximately that rate multiplied by the current latency, and which backs off when errors climb while throughput stalls, because that combination usually signals a saturated server rather than one that merely needs more connections. When the backlog fills up, the excess is shed and counted instead of being buffered without limit, and this choice is what keeps the backpressure honest, because the user sees the achieved rate fall below the target exactly when the endpoint can no longer keep up, which is the very signal a load test exists to produce.

Measuring as if someone will check

A chart is only as useful as the arithmetic behind it, and several of the more instructive bugs in the project came from numbers that looked plausible while being subtly wrong, such as failures being counted per sampling window while successes were normalized per second, which quietly halved the apparent error rate whenever the sampling interval was shorter than a second. Every series is now normalized to requests per second, the error rate is defined as the share of finished requests that failed rather than the share of requests sent, and the final partial window of a run is included in the totals but deliberately left out of the chart, because a rate computed over a few milliseconds is noise that would otherwise draw a dramatic spike at the very end of an otherwise calm test. The same concern for honesty extends to pausing, since resuming a test continues with whatever time and request budget remain, so a sixty-second test stays a sixty-second test no matter how often it is interrupted.

Composing real requests

Read endpoints are only half of most APIs, so the app sends every common HTTP method, from GET and HEAD through POST, PUT, PATCH, and DELETE to OPTIONS, and the methods that carry a body get a dedicated Load tab in which the body is written, typed as JSON, form data, plain text, or XML, and highlighted as you type.

Building that editor surfaced a small lesson that is easy to miss, because the standard text editor on macOS follows the system setting for smart quotes and silently turns straight quotation marks into typographic ones, which produces JSON that looks correct on screen and fails on the server, so the body editor disables every automatic substitution and shows a live check of the body size and of whether the JSON is valid. The Texts tab then closes the loop by showing the request exactly as it is sent, together with the most recent distinct responses, which makes it easy to confirm that the server answered with the expected status and payload before trusting anything the chart says.

Architecture

Underneath the interface the structure is deliberately plain, with SwiftUI views observing a single session model, the model talking to an engine facade that owns the lifecycle of a run, and the facade delegating the actual traffic to a transport adapter, so that the views never know how requests are sent and the adapters never know how anything is drawn.

Because the model is observable at the level of individual properties, a tick from the engine redraws only the views that display tick data, and editing a setting no longer redraws the chart, which mattered more than expected once tests ran long enough to accumulate thousands of points. The adapter boundary is also where the roadmap lives, since HTTP is the only real transport today while gRPC, WebSocket, and raw TCP are planned as further adapters behind the same contract, and keeping the pure parts of the system, such as the pacing arithmetic, the chart windowing, and the response bookkeeping, free of interface code is what allows most of the behavior to be covered by fast unit tests that run on every pull request.

Where it goes from here

Pace Maker is still a small tool, and that is very much the point, because the value of a quick load test lies in how little stands between the question and the answer. The next steps are the remaining transports and a few refinements to how the connection gate tells random errors apart from genuine saturation, and if you would like to try the current release, the signed builds and a full feature overview are available in the public repository.

The Concurrency Cost in Different Programming Languages

Eugene Zimin — Sun, 21 Jun 2026 21:09:40 +0000

Many of the developers and engineers, especially younger generation used to use concurrency in their projects as is - just throw a thread, join it at the end, then destroy. More experiences guys use threadpool, which doesn't kill threads and doesn't waste time on their re-initializing. Seniors and staff guys probably knows that it's not that obvious under the hood, and that's what we would like to examine to reveal for making more pragmatic choice while using multithreading, concurrency and parallelism in general.

If we take a look around, most of the benchmarks use prepared workflows which usually are well enough to be parallelized, meaning that data is isolated, algorithms are never changed and there is no direct impact on datasets in different threads. This is what is called ideal conditions - we all know them and definitely use threading as we can see there real impact on our efficiency and it is positive. Under such circumstances the problem is almost invisible and negligible.

However the data is not always prepared and in many variations it became so small that spawning the thread, queuing a job, waking a worker, cost of making application working with threading becomes notable enough to be considered and surprisingly may lead to worse performance with enabled concurrency than without it.

I prepared few test cases to find out all of this machinery and measure its impact.

My goal is to measure and quantify the pure overhead of different concurrency approaches - raw thread creation and join, a simple thread pool with work-stealing and lock-based queues, as well as some higher-level abstractions like async. In this case, by isolating the costs of thread startup, context switches, cache bouncing and memory barriers with the cost of synchronizing primitives, I can draw a simple decision boundary - at which point concurrency stops giving benefits and brings more problems. Results should help to decide whether to stay sequential, go async, use a thread pool, or invest in more sophisticated technics.

The First Workload - CPU only

The work itself is always the same and it's very CPU intense (for the first round - later I'll make other ops tested, like IO, mixed load, etc.). I have ten thousand terms of the Leibniz series for π, which is a perfect pure computation with a tight floating-point loop. Usually it finishes in a couple of microseconds, but that's the point - it’s small enough to understand whether it worths to start a job sequentially or throw it concurrently, because the compiler have to do some extra job for that. This is exactly the regime where all the dispatch machinery suddenly becomes visible.

However don't read this as a CPU benchmark. The π loop is only a good known fixed payload which has minimal size of the side effects. I'm measuring the only cost of spawning a thread, pushing a job onto a queue, then getting work from another core thus waking another worker, and to make it more clean I need to minimize any size effects which may give unpredictable and unreproducible results. In this picture the arithmetic becomes just the minimal work without any side effects and nothing more - pure calculations without going back and forth with IO, networking or something else.

Getting such a workload, small enough, run in microseconds, but repeated multiple times one after another by pulling each one from the queue, I set the goal to check - how fast overall queue would be cleaned up from those jobs (to calculate π).

Each measurement fires a hundred of these tasks through one concurrency primitive and waits until all hundred are done. That full fan-out counts as one sample. Every variant gets a single warm-up fan-out first - to warm the instruction cache, the branch predictors, and, when a pool is involved, to make sure the workers are actually created before the clock starts - and then a thousand timed samples.

How to Read the Numbers

A mean value would be definitely wrong here. Dispatch cost isn't a single value, it's a distribution with a tail, and the tail is actually where I can catch some interesting failures. There might be a run that hit a scheduler hiccup, or another one which caught a collector mid-sweep, or the one that was thrown onto a slow core. Every variant is reported as three points on its distribution: p50 (the median - the typical run), p99(the bad-but-not-rare run), and p999 (the worst run in roughly a thousand). With a thousand samples, p999 gets landing on rank ~998 - but it's a genuine one-in-a-thousand observation, not a synonym for "the slowest sample." When p999 pulls far away from p50 then something intermittent fell into the measurement and naming that something - whether it was a GC pause, or a clock transition, or a cold pool - is most of the work of the later articles.

One methodological note that matters. A loop whose result is never used, is a loop which a good optimizer is entitled to delete, and with a flag --release Rust will absolutely do that as part of its optimization, as leaving you a workload for just an empty fan-out is a glorious nonsense. Thus each of the variant here wraps each task's result into a single global accumulator (simple XOR of the raw float bits) at the very end. At the same time that accumulator does a second job as it keeps every print out of the timed region to minimize side effects on IO operations to console - I don't want to make a benchmark of stdout.

What I Can't Control For

My working machine is Apple M5 Pro, and Apple silicon don't use a CPU governor, so I can't pin the clocks, I can't disable boost, and it's not possible to stop the scheduler from migrating a task between performance cores and super cores. That's another reason for p99 and p999 measurements - as a task that lands on an performance core, shows up as a p99 or p999 tail that came from the machine, not from the runtime.

Unfortunately I don't have a Linux box under my desk, so I can't run those tests on Linux and the same is true for Windows (I happily breathed out when I switched on Mac in 2008 - sorry Bill), but I share all the code in my GitHub repository so anyone may examine it, repeat tests and make his own measurement. Pure experiment, promising to become exciting and brings the same exciting results.

Test Environment

All measurements were taken on a single machine, cold-start: one warm-up run, then 1000 timed fan-outs per variant. No warm/cold matrix, no cache manipulation.

Keys	Values
Machine	MacBook Pro (Mac17,9), Apple M5 Pro
Cores	15 (5 Super and 10 Performance)
Memory	24 GB
OS	macOS 26.6.2 (25G83) (Darwin 25.6.0 kernel)

Runtimes (exact, pinned)

Language	Version	Build / flags
Rust	rustc 1.98.1 (48a229cea 2026-09-01)	`--release`
Go	go version go1.27.1 darwin/arm64	default toolchain
Node.js	v24.20.0 (npm v11.19.0)	worker_threads
Python	Python 3.14.6 free-threading [Clang 22.1.3 ] Python 3.14.6 [Clang 22.1.3 ]	3.14t 3.14
Java	Oracle JDK 25.0.3 (25.0.3+9-LTS-195)	G1 GC, `-Xms2g -Xmx2g`

Next Step

Rust comes first because it gives me the cleanest baseline (and that's what I know better than C). It has such features like ahead-of-time compiled, no garbage collector, no interpreter loop - there's nothing between the code and the scheduler. The numbers I measure here are purely dispatch cost, but nothing else. All of these makes Rust the best starting point - once I know the cost a hundred tiny tasks through raw threads, through rayon, through a thread pool, and through an async runtime, I can do the same and compare with Go's scheduler, Java's JIT and its GC tail, Python's interpreter, and Node's runtime.

A native macOS load tester app — and backpressure made it honest

Eugene Zimin — Sat, 06 Jun 2026 22:16:45 +0000

Why I built Requester, a real-time HTTP load testing app for macOS, and what Swift structured concurrency taught me about telling the truth under load.

I wanted to hammer an HTTP endpoint and see what happened. Not read a summary report three minutes later — watch it, live, the way you watch a profiler.

The existing options are great but they all live in the terminal: wrk, hey, k6. I love them, but I kept wishing for a native window with a chart that moved. So I built one for macOS, in Swift and SwiftUI, and called it Requester.

This post is less "here are the features" and more "here are the three things I made building it." The most interesting one: making the tool honest about backpressure turned out to be a design decision, not an accident.

The model: RPS per channel × channels

Before any code, I needed a mental model the user could reason about. I landed on two knobs:

Channels — how many requests can be in flight at once to emulate multiuser load (even though from a single host).
RPS per channel — the rate I want each channel to sustain.

So the total target rate is just channels × rpsPerChannel. No hidden division, no surprises. The dispatcher converts that into a fixed firing interval:

let interval = Duration.seconds(
    1.0 / Double(session.throughputPerSec * session.concurrency))

The win here is conceptual, not technical: the number you type is the number you get (when the endpoint can keep up). Which brings me to the part where it can't.

A load tester should lie down when the server does

Here's the temptation. You set a target of 500 RPS. The endpoint can only serve 200. What do you do with the other 300 requests every second?

The naive answer is "buffer them." But then your queue grows without bound, memory balloons, and — worse — your tool reports that it's happily firing 500 RPS while reality is nowhere close. That's a lie, and it's the most dangerous kind because it looks like success.

So Requester does the opposite. The dispatcher fires requests as independent child tasks and never awaits a response. A single actor gates how many can be in flight:

private actor InFlightCounter {
    private let limit: Int
    private let maxQueueDepth: Int
    private var current = 0
    private var waiters: [CheckedContinuation<Void, Never>] = []

    func acquireSlot() async -> Bool {
        // Queue is full — shed this request, don't buffer forever.
        guard waiters.count < maxQueueDepth else { return false }

        if current < limit {
            current += 1
            return true
        }
        // At the limit: suspend until a response frees a slot.
        await withCheckedContinuation { waiters.append($0) }
        current += 1
        return true
    }

    func release() {
        current = max(0, current - 1)
        if !waiters.isEmpty { waiters.removeFirst().resume() }
    }
}

Two behaviors fall out of this:

The dispatcher suspends when every channel is busy. It can't fire faster than responses come back. So the achieved RPS visibly drops below your target — and that dip is exactly the signal you came for. The chart tells you the truth.
Past a configurable queue depth, requests are shed, not buffered. Bounded memory, bounded lag, no fantasy throughput.

When the green "Received" line peels away from the blue "Sent" line on the chart, that's the endpoint saying uncle. I didn't have to build a special "the server is struggling" indicator. The honest plumbing surfaces it for free.

Structured concurrency made the whole thing boring (in the best way)

The entire engine is async/await, TaskGroup, and actors — no GCD queues, no completion-handler pyramids, no manual locks.

The dispatcher and a snapshot timer run as siblings in one TaskGroup. The dispatcher records per-request outcomes into a RunCounter actor; the timer reads a delta from that same actor once per sampling interval and emits a .tick event. Because the counter is actor-isolated, the N writers and the one reader never race — and I never wrote a lock.

actor RunCounter {
    private(set) var totalCompleted = 0
    private(set) var totalFailed = 0

    func record(_ outcome: RequestOutcome) {
        switch outcome {
        case .success: totalCompleted += 1
        case .failure: totalFailed += 1
        }
    }

    /// Delta since the last read, then advance the baseline.
    func snapshot() -> (completedDelta: Int, /* … */ totalCompleted: Int) {
        let cd = totalCompleted - lastCompleted
        lastCompleted = totalCompleted
        return (cd, /* … */ totalCompleted)
    }
}

Events flow up through an AsyncThrowingStream from the transport adapter, into a facade, and finally onto @Publishedproperties in a @MainActor view model. SwiftUI redraws. The chart moves. Each layer has exactly one job and they communicate through Sendable values crossing isolation boundaries safely.

The payoff: the scary part of a load tester — concurrent state — became the calmest part of the codebase.

URLSession will silently eat your headers

This one cost me an embarrassing amount of time, so you get it for free.

I set a custom User-Agent on my URLRequest like any reasonable person:

request.setValue(session.userAgent, forHTTPHeaderField: "User-Agent") // 😶 silently dropped

It never showed up on the wire. Turns out URLSession treats User-Agent (and other reserved headers) as off-limits at the request level and quietly strips them. The fix is to set them on the configuration instead:

let config = URLSessionConfiguration.ephemeral
config.httpAdditionalHeaders = ["User-Agent": session.userAgent] // ✅ actually sent

No error, no warning — it just works when you move it. If you've ever sworn your custom User-Agent "isn't applying," this is probably why. Requester ships with realistic browser presets (Safari, Chrome, Firefox, mobile variants, curl, Googlebot) precisely so you can test how an endpoint responds to different clients.

Seeing the actual bytes

A throughput chart tells you how much; sometimes you need what. Requester has a request/response inspector that shows the exact request being fired and the live responses coming back — status line, headers, body — with a sliding-window history and an optional truncation toggle for noisy bodies.

What's next

HTTP is what ships today. The transport layer is already abstracted behind a TransportAdapter protocol (which also made testing trivial — every transport has a mock), so the architecture is ready for more:

gRPC endpoint testing
WebSocket load
Raw TCP load

Requester is a native macOS app built entirely in Swift and SwiftUI, targeting macOS 14+.

Built by Eugene Zimin and available on — GitHub.

If you've built load-testing tooling, I'd love to hear how you handled the buffer-vs-shed question — drop a comment.

Drawing the Blueprint: Flowchart, Functional Diagram, and Sequence Diagram

Eugene Zimin — Sat, 23 May 2026 20:45:57 +0000

Lesson 3 of Build a Twitter Clone - A Practical Guide to Software Modelling

In Lesson 2, we defined what Bird must remember: two databases, tables within, every field traced back to a use case. Now it's time to draw what Bird must do. Using "Post a message" as our single worked example, we construct all three diagrams in sequence - first a flowchart that traces the user action step by step, then a functional diagram that reads the required components directly off that flowchart, then a sequence diagram that shows exactly how those components talk. By the end, the use case that started as a sentence on a sticky note has a complete, traceable blueprint.

Introduction - The Foundation Is Ready

Lesson 2 ended with a promise, and it's worth quoting precisely:

"That's where Lesson 3 picks up: with the data model as the foundation, we draw all three diagrams."

That moment has arrived. But it's worth pausing for a second to notice what's different now compared to where we started.

In Lesson 1, we introduced the idea of three diagrams - a flowchart, a functional diagram, and a sequence diagram - as three lenses on a single system. We explained the mental model, argued for the order, and left it there. No diagrams were actually drawn, and that was deliberate: a diagram drawn before the project is bounded and the data understood is a diagram you'll redraw.

Figure 1. The Foundation Is Ready

In Lesson 2, we did the bounding. We read Bird's three use cases for data clues, named two entities (User and Message), defined five tables across two databases (ums and twitter), and established the relationships that tie them together. The schema is real now: a messages row has a specific shape - an id, a content field capped at 280 characters, an author_idpointing to a row in ums.users, and a created_at timestamp. The users row has a shape too. Every "save the message" step in every diagram we draw will refer to something concrete.

That concreteness is what changes everything. A diagram drawn on top of a real data model isn't a sketch - it's a specification. The boxes mean something. The arrows carry named fields. When a step says "validate the message," we already know what the validation is checking (which is called validation criteria): a non-empty content string, no longer than 280 characters, attached to an authenticated author_id. Nothing is left as a placeholder.

This lesson draws all three diagrams. We'll work in order - behaviour first, then the structure that behaviour implies, then the detailed conversation inside that structure - for the same reason we always work in that order: each diagram is the input to the next, and the input has to exist before the output can be correct.

A Quick Reminder: The Three-Lens Progression

Before we pick up the pen, one short bridge back to Lesson 1 - not a full re-teach, just enough to make sure the scaffold is standing.

The core idea is this: any software system can be understood from three distinct angles, and each angle demands a different kind of diagram.

Lens	The question it answers	The diagram	What you see
Behaviour	What should the app do?	Flowchart	Steps, decisions, and outcomes - the system as a process unfolding in time
Structure	What parts must exist to make that behaviour happen?	Functional diagram	Components and their connections - the system as a thing made of pieces
Interaction	How do those parts talk to each other at runtime?	Sequence diagram	Messages exchanged between components - the system as a conversation

None of these is more correct than the others. A flowchart can't tell you what components to build. A functional diagram can't tell you what happens when a user submits an empty message. A sequence diagram can't tell you whether you've correctly scoped the feature. They are complementary, not competing - and you need all three because no single one is complete.

The critical habit is the order. You move behaviour → structure → interaction, and each step is derived from the one before:

Figure 2. The three-lens progression - each diagram derives from the one before

The reason the order is fixed: you can't know what components to build until you know what the system must do. And you can't describe the conversation between components until you know which components exist. Behaviour is the requirement; structure serves it; interaction lives inside structure.

An analogy for the progression. Imagine designing a kitchen. First you decide what the kitchen must do- prepare meals, store food, clean dishes. That's behaviour. Then you ask what parts must exist to support each of those tasks: a cooktop, a refrigerator, a sink. That's structure, falling out of behaviour. Finally, you trace the specific workflow - the chef moves from the refrigerator to the prep surface to the cooktop, in that order, passing ingredients along. That's interaction, living inside the structure. You couldn't have designed the workflow without knowing the layout. You couldn't have designed the layout without knowing the tasks. The order is the method.

Lens 1: Behaviour - Drawing the Flowchart

A flowchart answers one question: what does the system do, step by step? Not what parts it has. Not which component calls which. Just the sequence of actions and decisions that carry a user from "I want to post a message" to "it's done" - or to an error, if something goes wrong along the way.

That makes the flowchart the right place to start. It captures the requirement in its purest form, before any implementation choices have been made.

Building it step by step

Step 1 - Name the start and end points.

The flow begins the moment the user presses "Post" - that is, when the system receives a submission. It ends either with a new row in twitter.messages and a confirmation to the user, or with an error message and no row written. Name those endpoints first, before filling in anything in between.

Step 2 - List the actions in order.

Walk the happy path first - what happens when everything goes right? For "Post a message," that sequence is:

Submit a message
Pass the authentication check - is the user logged in?
Pass the permissions check - is the user allowed to post?
Pass content validation - is the message valid?
Save the message to the database
Confirm success to the user

Six steps. Two are actions, three are decision gates, one is a confirmation. That's a healthy breakdown - the logic is explicit without becoming an implementation manual.

Step 3 - Add the decision points.

Three decision diamonds gate the happy path:

Is the user logged in? If no → redirect to the login page. After a successful login the user is returned to the submission step - a loop, not a dead end.
Is the user allowed to post? Not every account has posting rights - a read-only or suspended role is blocked here. If no → shared Error node → END.
Is the message valid? (Non-empty? Within 280 characters?) If invalid → shared Error node → END.

Notice that the two error branches converge on a single Error node before reaching END. This is an intentional design choice in the diagram: the system's response to a permissions failure and a content failure is the same kind of thing - an error returned to the user. Merging them keeps the diagram clean and makes that equivalence visible.

Step 4 - Connect and label.

Arrange the steps top to bottom, connect them with arrows, and label every arrow leaving a decision diamond. The labels on the branches ("yes / no", or more descriptively "valid / invalid") are what make the diagram readable at a glance.

The flowchart

Figure 3. Flowchart for "Post a message"

What this diagram tells us - and what it doesn't

The flowchart is intentionally silent on how each step is implemented. It doesn't say which component checks authentication, or what library validates the string length, or whether the database write is synchronous. Those details don't belong here. What the flowchart does - and does well - is make the logic visible: the decisions, the branches, the outcomes. That's the entire job of Lens 1.

Notice that the data model already sharpens the language. The "Save message" step doesn't just say "save the message"- it names the fields: author_id, content, created_at, writing into twitter.messages. That specificity comes directly from Lesson 2. The flowchart is still high-level, but it's no longer vague.

With the behaviour fully drawn, the next question asks itself: what parts must exist to make each of these steps happen? That's Lens 2 - and the answer comes directly from reading this flowchart.

Lens 2: Structure - Reading the Functional Diagram off the Flowchart

The flowchart is done. We know what "Post a message" must do - the steps, the decisions, the branches. Now a different question takes over: what parts must exist inside the system to make each of those steps happen?

This is where most beginners get into trouble. They open a new diagram and start inventing components - a generic AuthService, an APIGateway, a DatabaseLayer - based on instinct or prior experience. The result is a diagram that reflects what the developer already knew, not what the use case actually requires.

There is a better method. The functional diagram doesn't need to be invented. It can be read directly off the flowchart.

The method: scan every step, ask one question

For each step in the flowchart, ask: what component must own this responsibility? The answer names a block. Do that for every step, merge the ones that belong together, and the structure falls out of the behaviour.

Let's walk through the flowchart step by step.

Flowchart step	Responsibility	Component
User submits message	Sends the HTTP request	Frontend Client
Is the user logged in?	Identifies who is making the request	Twitter Service
Redirect to login	Returns the login page	Frontend Client
Is the user allowed to post?	Fetches the user record and checks for the `PRODUCER`role	Twitter Service → UMS Service
Is the message valid?	Enforces content rules (non-empty, ≤ 280 chars)	Twitter Service
Save message	Writes the row to `twitter.messages`	Twitter Service → MySQL Database
Confirm success	Returns `201` with the created message	Twitter Service → Frontend Client
Return error	Returns `403 Forbidden` to the client	Twitter Service → Frontend Client

Two things stand out immediately. First, the Twitter Service carries the most weight: it receives the request, performs content validation, and orchestrates the database write. But it doesn't hold user data - that lives in UMS. So for the permission check, the Twitter Service calls the UMS Service via an internal HTTP request, retrieves the user's roles, and decides whether to proceed. This cross-service call is the architectural heart of the feature.

Second, there is no separate API Gateway or Auth Service in Bird. The Frontend Client talks directly to the Twitter Service on port 9001. Role checking is a responsibility of the Twitter Service itself, delegated to UMS on demand - not a standalone component sitting in between.

This gives us four components: the Frontend Client, the Twitter Service, the UMS Service, and MySQL Database.

The functional diagram

Figure 4. Functional diagram for Bird

The diagram reads as a set of named blocks connected by labelled arrows. The arrows carry the nature of the relationship - not the step-by-step logic (that's the flowchart's job) but the structural dependency: the Frontend Client sends requests to the Twitter Service; the Twitter Service calls UMS to validate the user; the Twitter Service reads and writes MySQL directly. Each arrow represents a real, named communication channel in the codebase - HTTP on :9001 for the UI-to-service call, WebClient for the service-to-service call, JDBC for the database connection.

What this diagram is - and isn't

A functional diagram is a map of components, not a map of steps. It answers "what pieces exist and how are they connected?" - not "what happens first?" The flowchart and the functional diagram are not redundant; they answer different questions and each is incomplete without the other.

Notice also what the functional diagram deliberately omits: there are no arrows labelled with field names, no decision branches, no error paths, no sequence numbers. Those details live in the sequence diagram. Here, the goal is structural clarity - a reader should be able to see at a glance what Bird is made of and how the pieces relate.

With the components named and connected, one question remains: how exactly do they talk to each other during "Post a message"? That's the sequence diagram - and now that we know which components exist, we can finally draw it precisely.

Lens 3: Interaction - The Sequence Diagram

The flowchart told us what "Post a message" does. The functional diagram told us what components exist to do it. The sequence diagram answers the last question: what do those components say to each other, in what order, when the feature actually runs?

This is where the previous two diagrams earn their investment. Every component named in the functional diagram becomes a lifeline. Every step in the flowchart becomes a message on one of those lifelines. Nothing is invented - it's assembled from what we already know.

From components to lifelines

A sequence diagram is built around lifelines - vertical columns, one per component, representing each participant in the interaction. Reading the functional diagram directly, our lifelines are:

Frontend Client - the browser, initiating the request
MessageController - the entry point inside the Twitter Service, receiving the HTTP call
MessagesService - the orchestrator: calls UMS, checks roles, triggers the database write
UMSConnector - the thin HTTP client that talks to UMS via WebClient
UMS Service - the external service that owns user identity and roles
JdbcMessageRepository - the DAO that executes the SQL insert
MySQL - the database that persists the row

Reading the flow

The messages between lifelines follow the exact path the codebase executes. Let's trace it:

The Frontend Client sends POST /messages/message with { author, content } to the MessageController.
MessageController deserializes the body into a Message DTO and delegates to MessagesService via createMessage(message).
MessagesService needs to verify the author - it doesn't hold user data, so it calls UMSConnector with retrieveUmsData(/users/user/{id}).
UMSConnector fires a non-blocking WebClient GET to the UMS Service.
UMS Service returns a UserDto containing the user's roles.
MessagesService unpacks the response via HttpResponseExtractor and inspects the Roles. If the user is not a PRODUCER - the flow terminates here with a 403 Forbidden back to the client.
If the role check passes, MessagesService calls createMessage(message) on JdbcMessageRepository.
JdbcMessageRepository executes INSERT INTO messages against MySQL (converting the UUID via DaoHelper).
MySQL returns ok.
The saved Message travels back up: JdbcMessageRepository → MessagesService → MessageController → Frontend Client as 201 { message }.

The sequence diagram

Figure 5. Sequence diagram for "Post a message"

What this diagram shows that the others couldn't

Look at the alt block in the middle of the diagram. That's the permission check - the same diamond in the flowchart, the same role-checking responsibility assigned to MessagesService in the functional diagram. But now it has an exact form: a comparison against the PRODUCER role, returning 403 Forbidden if it fails, with the response passing back through MessageController to the client.

This is traceability working in practice. The validation step in the flowchart → the MessagesService block in the functional diagram → the HttpResponseExtractor → User → Roles call chain in the sequence diagram. One idea, three levels of resolution.

Notice also what the sequence diagram adds that neither previous diagram could: the reactive chain. The UMSConnectorcall is non-blocking - MessagesService issues the WebClient request and the rest of the chain executes inside a .flatMap(). The sequence diagram makes this visible through the activation boxes: MS stays active while UC and UMS do their work, then resumes. That concurrency detail is invisible in both the flowchart and the functional diagram.

With all three diagrams drawn, one use case - "Post a message" - now has a complete, layered blueprint. The next section puts all three side by side and shows what traceability looks like end to end.

Let's pause before moving on and do something the three diagrams individually don't do: put them side by side and trace a single idea across all of them.

Take the permission check - the moment the system decides whether the author is allowed to post. Here is where it lives in each diagram:

Diagram	Where the permission check appears	What you can see
Flowchart	The diamond "Is user allowed to post?" branching to `Error`or continuing to validation	The logic - two outcomes, one decision
Functional diagram	The arrow from `MessagesService` labelled "check roles contains PRODUCER" pointing to `Roles`	The ownership - which component holds this responsibility
Sequence diagram	The `alt` block: `HttpResponseExtractor → User → Roles`, branching to `403 Forbidden` or `INSERT`	The mechanics - what data moves, in what order, producing what result

Same check. Three levels of resolution. You could point at the diamond in the flowchart, follow it to the MessagesServicearrow in the functional diagram, and land on the alt block in the sequence diagram - and at each step you'd learn something the previous diagram couldn't tell you.

That property is traceability: the ability to follow a single requirement from its highest-level expression down to its lowest-level implementation without losing the thread. It's what separates a set of diagrams that decorate a document from a set that actually functions as a blueprint.

Figure 6. One feature, three views - the permission check traced across all three diagrams

Notice also what traceability reveals in the other direction. The sequence diagram shows that the 403 Forbidden response travels back through MessageController before reaching the client - a detail invisible in both the flowchart (which just says "Error → END") and the functional diagram (which shows MessageController connected to GlobalExceptionHandler without explaining when). The diagrams don't just repeat each other. Each one completes the picture in a direction the others leave open.

Limitations and Open Questions

Three diagrams drawn, one use case traced. Before closing, it's worth being clear about what this process is - and what it isn't.

Diagrams go stale. Every diagram in this lesson reflects the codebase as it stands today. The moment a new field is added to messages, or the permission model changes, or UMSConnector is replaced by a shared auth library - the diagrams need updating. A diagram that isn't maintained becomes misleading faster than no diagram at all. The discipline of drawing diagrams is only half the work; the other half is keeping them honest.

This is a snapshot, not a specification. The diagrams here describe what Bird does, not what it must do. A specification would go further - stating invariants, error contracts, performance bounds, and edge cases. These diagrams are the foundation of a spec, not the spec itself. They answer "how does it work?" not "how must it always work?"

Over-modelling is a real risk. Drawing all three diagrams for every feature in a large system would consume more time than it saves. The three-lens approach is most valuable at decision points: when designing a new feature from scratch, when onboarding someone to an unfamiliar part of the system, or when debugging behaviour that contradicts the mental model. Applied indiscriminately, it becomes ceremony rather than engineering.

These diagrams are not UML. Readers familiar with the Unified Modelling Language (UML) will notice that the diagrams in this lesson borrow UML conventions - sequence diagram lifelines, flowchart decision diamonds - without following the full UML standard. That's deliberate. UML is precise and complete, and its precision comes at the cost of accessibility. However when you move to production systems with formal specification requirements, UML or C4 may be the right choice.

What's Next

Three lessons in, and Bird has a bounded scope, a real data model, and a complete three-lens blueprint for its core feature.

What it doesn't have yet is a working API. The diagrams show that a POST /messages/message endpoint should exist, that it must accept { author, content }, that it must call UMS before writing to the database - but none of that is running code yet.

Lesson 4 is where the diagrams meet the wire. We'll define the API contracts and endpoints that connect Bird's components - what each service exposes, what it expects, and what it returns. The sequence diagram already told us that POST /messages/message must exist, that it calls GET /users/user/{id} on UMS, and that both exchanges carry specific payloads. Lesson 4 makes those contracts explicit and formal: request shapes, response shapes, status codes, and the communication protocol between the Twitter Service and UMS.

The diagrams were never the destination. They were the clearest possible way to know what to build before building it.

The Blueprint Beneath the Blueprint: Designing Data Model and Choosing Its Database

Eugene Zimin — Sat, 23 May 2026 06:36:22 +0000

Lesson 2 of Build a Twitter Clone - A Practical Guide to Software Modelling

A diagram shows you what a system does; a data model tells you what it remembers. Before drawing a single flowchart, you need to know what information Bird must store - and how that information is shaped. In this lesson we read our three use cases for data clues, name the entities the system must track, define their fields and relationships, and translate all of it into a real MySQL schema split across two purposefully separated databases.

Introduction - Data Before Diagrams

Lesson 1 was closed with a promise: in Lesson 2, we draw the diagrams. Flowchart, functional diagram, sequence diagram - the works. At the moment we're going to defer it for a while. The reason is quite simple - because of something that becomes obvious the moment you try to draw the flowchart for "Post a message" without it: a diagram that doesn't know what data it's moving is vague in exactly the wrong places.

You need to consider the following - the flowchart for posting a message will eventually have a step called something like "save the message." Straightforward enough on paper. But the moment you try to build it, questions pile up fast:

Save what, exactly? The text? The author? A timestamp? All three?
Where does the author's identity come from - a name typed into a field, or a reference to a stored account?
What does "the author" even mean to the system - a row in a table somewhere, or just a string?
If the same author posts twice, how does the system know both messages belong to the same person?

Figure 1. What is the Message?

A diagram that leaves those questions open isn't a blueprint. It's a sketch - useful for thinking, but not yet useful for building. The answers live in the data model: the formal description of what the system stores, and how the pieces of stored information relate to one another.

Think of the data model as the system's long-term memory. The diagrams describe what the system does- its behaviour, its structure, its conversations. The data model describes what it remembers between those moments of action. Without memory, each request starts from nothing. With it, a message posted today is still there tomorrow, and the author who posted it can be found.

Getting the data model right before drawing the diagrams isn't pedantry. It's the thing that turns vague boxes into precise components. When you know that a message is a row with an id, a content field capped at 280 characters, and an author_id that points to a specific user - then the "save message" step in your flowchart stops being a placeholder and starts being an instruction. The functional block that does the saving has a clear contract. The sequence diagram can name what passes between components.

One System, Two Databases - and Why That's the Right Call

Before we write a single table definition, there's a structural question to answer: should all of Bird's data live in one database, or more than one?

The instinct is usually to start with one. It's simpler, it requires less setup, and for a small project it feels like the obvious default. We're going to make a different call - and the reasoning behind it is worth understanding clearly, because the same question will come up in every non-trivial system you ever build.

The problem with putting everything in one place

Start with the simpler option: one database called bird, with all tables inside it - users, messages, sessions, roles, everything. Will it work?

The answer is clear - it would work. In fact, it's how most beginner projects start, and there's nothing wrong with it as a starting point. But consider what happens as the system grows:

A security audit requires changes to how passwords are stored. To make that change safely, you need to understand every table that touches user data - except now it's tangled up with message tables, timeline queries, and subscription records. The scope of "change one thing" has quietly expanded.
Message volume spikes. You want to move the messages table to faster storage, or archive old records. But the table is in the same database as your user credentials, which have entirely different performance and retention requirements. You can't move one without the other.
A colleague is working on authentication while you're working on the timeline feature. Both of you are making schema changes in the same database, to tables that are conceptually unrelated. You're in each other's way.

Figure 2. All in one Database - Will it Work?

The deeper problem is this: a single database couples concerns that have no real reason to be coupled. They are in the same place not because they belong together, but because it was convenient to put them there. Convenience now, complexity later.

The principle: each domain owns its data

The solution is to give each distinct area of the problem its own data store - its own database that it, and only it, is responsible for. Nothing else writes directly to that data; anything that needs information from another domain has to go through the interface that domain exposes.

This is a well-established pattern in software design called Database per Service: each logical service or domain owns its storage, and the boundary between services is enforced at the data layer, not just at the code layer. The pattern makes the separation real and durable - you can't accidentally reach across a boundary you'd have to explicitly cross.

![[Database per Service Design Pattern.jpg]]Figure 3. Database per Service Design Pattern

For Bird, two domains emerge clearly once you ask the question:

Identity and access - who people are, how they prove it, what permissions they hold, which sessions are currently active. This is security-critical data, relatively stable, and completely self-contained. It has nothing to do with messages.
Content and social graph - the messages people post, and the follow relationships between them. This data is high-volume and fast-moving, with entirely different performance and retention characteristics.

These two concerns have different owners, different change rates, and different security requirements. They will be given separate databases: ums (User Management System) for identity and access, and twitter for content and social graph.

Database	Domain	What it owns
`ums`	Identity & access	Who people are, how they authenticate, what permissions they hold, which sessions are active
`twitter`	Content & social graph	The messages people post, and who follows whom

Separating them means each can be optimized, scaled, or secured independently - a schema change in one won't touch the other.

The idea behind the split: bounded contexts

The deeper principle at work here comes from Domain-Driven Design (DDD) - a way of thinking about how to organize software around the real-world problems it solves, rather than around technical convenience.

DDD would describe ums and twitter as separate bounded contexts: distinct areas of the problem, each with its own vocabulary, rules, and data. The word user in the identity context means something specific - an account with credentials, roles, and a session history. The word author in the messaging context means something different - a source of content, identified by an ID, whose full identity details live elsewhere. These concepts correspond to the same human being, but they are different models of that person, serving different purposes. Keeping them in separate databases makes that distinction visible and enforces it.

Figure 4. How DDD Helps and Works

An analogy. Think of a hospital. The billing department and the medical records department both deal with the same patients - but they maintain completely separate files. The billing system doesn't need to know a patient's diagnosis; the medical records system doesn't need to know their payment history. Each department owns its data. Information is shared only when explicitly requested, through defined channels. The separation isn't bureaucracy - it's what keeps sensitive data appropriately contained, and what lets each department evolve independently.

If you want to go deeper on Domain-Driven Design, this article covers the core ideas without requiring a computer science background.

Reading the Use Cases for Data Clues

A use case describes what a user accomplishes. But read carefully, and it also tells you what the system must remember in order to make that possible. Each of Bird's three use cases leaves a trail of data requirements - we just have to follow it.

Post a message

A user writes some text and publishes it. For this to work, the system must store the text itself, know who posted it, and record when it was posted so the timeline can be ordered. That's three pieces of information: content, author, timestamp.

Scope	Field	Type	Why it's needed
`messages`	`id`	UUID	A stable, unique identifier for this message
`messages`	`author_id`	UUID	The `id` of the user who posted this message - links back to `users.id`
`messages`	`content`	String (max 280 chars)	The text of the message, capped at 280 characters
`messages`	`created_at`	Timestamp	When the message was posted - used to order the timeline

View the timeline

This use case produces no new fields. It reads existing messages rows, ordered by created_at descending, and joins to users on author_id to display the author's name. Every field it depends on was already required by Post a message. Here we have to introduce what it is called - subscriptions and set a relation between author and and consumer of the message, i.e. - subscriber.

Scope	Field	Type	Why it's needed
`subscriptions`	`subscriber_id`	UUID	The user doing the following - logical FK to `users.id`
`subscriptions`	`producer_id`	UUID	The user being followed - logical FK to `users.id`
`subscriptions`	`created_at`	Timestamp	When the follow relationship was created

Register an account

A user creates an identity. The system must store a name to display, credentials to authenticate with (stored as a hashed password, never plain text), and again a timestamp. It also needs a way to distinguish one account from every other - a unique identifier that never changes even if the username does.

Scope	Field	Type	Why it's needed
`users`	`id`	UUID	A stable, unique identifier for this user - never changes, even if name or email does
`users`	`name`	String	The display name shown alongside every post
`users`	`email`	String	Login credential; unique across all users - no two accounts can share an address
`users`	`password_hash`	String	The user's password after a one-way hashing function - never the plain-text password
`users`	`created_at`	Timestamp	When the account was registered
`users`	`updated_at`	Timestamp	When any field on this record last changed; refreshed automatically on every write

Scope	Field	Type	Why it's needed
`roles`	`id`	UUID	Unique identifier for this role
`roles`	`name`	String	The role label (e.g. `admin`, `member`) - must be unique
`roles`	`description`	String	Optional human-readable explanation of what this role permits

Scope	Field	Type	Why it's needed
`sessions`	`id`	UUID	Unique identifier for this login session
`sessions`	`user_id`	UUID	References `users.id` - whose session this is
`sessions`	`logged_in_at`	Timestamp	When the session started
`sessions`	`logged_out_at`	Timestamp	When the session ended; `NULL` if the user is still logged in

Across all three use cases, two distinct categories of information emerge: things the system needs to know about people, and things it needs to know about messages. That observation is the foundation of the data model.

Naming the Entities

An entity is a category of information the system tracks as a distinct thing - something that has its own identity, its own set of properties, and its own lifetime. Naming entities is the first act of data modelling: you're deciding what the system considers a noun.

From the use cases, two entities name themselves:

User - a registered account. Every message is posted by a user; the timeline attributes each post to one. Users exist independently of any message they've posted, and they persist even if all their messages were deleted. They are a distinct thing in their own right.

Message - a piece of content posted by a user. Messages depend on users (a message with no author makes no sense), but they are not part of a user - they are their own thing, with their own timestamp and their own text.

Two entities. That matches the two databases we decided to create: ums owns User, and twitter owns Message. The database boundary and the entity boundary are the same line drawn twice.

You'll notice there's no Timeline entity, no Feed entity, no Notification. The timeline is not a thing the system stores - it's a query - give me all messages, ordered by created_at, descending. It exists at runtime, not at rest. This is a useful distinction to internalise: not everything the user sees needs to be stored.

In the next section we'll define the fields each entity carries - and introduce the mechanism that connects a Messageback to the User who wrote it.

Defining the Fields

An entity is just a name until you give it fields - the individual pieces of data it holds. This is where the model gets concrete. For each field, we'll state what it is, what type of value it holds, and why it exists, tracing every decision back to a use case or a design constraint.

A note on identifiers

Every entity needs a primary key - a field whose sole job is to uniquely identify one row among all others, forever. A common choice is an auto-incrementing integer (1, 2, 3...), but Bird uses something different: a UUID (Universally Unique Identifier), stored as 16 bytes (or 128 bits) of binary data.

A UUID looks like 550e8400-e29b-41d4-a716-446655440000 - a 128-bit value generated in a way that makes collisions statistically impossible, even across separate systems. The binary storage (BINARY(16)) keeps it compact and fast to index. The reason to prefer UUIDs over integers here is forward-looking: if Bird ever scales to multiple servers generating records simultaneously, each can produce its own IDs without coordinating with the others. Integers can't do that safely.

Every table uses this same pattern for its primary key: a BINARY(16) column called id, defaulting to a freshly generated UUID.

`users` - the identity record

The users table lives in the ums database. It is the authoritative record of everyone who has registered an account.

Field	Type	Purpose
`id`	`BINARY(16)`	Primary key - uniquely identifies this user across the entire system
`name`	`VARCHAR(100)`	The display name shown on posts
`email`	`VARCHAR(255)`	Login credential and contact address; must be unique across all users
`password_hash`	`VARCHAR(255)`	The result of running the user's password through a one-way hashing function - never the password itself
`created_at`	`DATETIME`	When the account was registered
`updated_at`	`DATETIME`	When any field on this record was last changed; updated automatically

Two fields deserve a word of explanation. password_hash stores a hashed password, not a plain-text one. A hash functionis a one-way transformation: you can turn a password into a hash, but you cannot reverse the process. When a user logs in, the system hashes what they typed and compares the result to the stored hash - the original password never needs to be stored or retrieved. This is standard practice; storing plain passwords is a serious security failure.

updated_at tracks the last modification time and refreshes itself automatically on every write. It's low-cost to store and invaluable for debugging, auditing, and cache invalidation later.

Supporting the `User`: roles and sessions

The use cases named registration - but a real identity system has two more concerns lurking just beneath the surface: what is this user allowed to do, and is this user currently logged in? These concerns get their own tables.

roles is a lookup table - a simple list of named permission levels (for example, admin, moderator, member). Roles don't come from the use cases directly; they come from the reality that not all users have the same permissions.

Field	Type	Purpose
`id`	`BINARY(16)`	Primary key — uniquely identifies this role
`name`	`VARCHAR(50)`	The role label (e.g. `admin`, `member`) — must be unique
`description`	`VARCHAR(255)`	Optional human-readable explanation of what this role permits

users_roles links users to roles. Because one user can hold multiple roles and one role can be held by many users, this is a many-to-many relationship - and the standard way to model that in a relational database is a join table: a table with two columns, each a reference to one side of the relationship.

sessions records active login sessions. When a user authenticates, a session row is created with a logged_in_at timestamp. When they log out, logged_out_at is filled in. This gives the system a full audit trail of who was logged in and when - and lets it invalidate specific sessions without forcing a global logout.

Field	Type	Purpose
`id`	`BINARY(16)`	Primary key — uniquely identifies this session
`user_id`	`BINARY(16)`	References `ums.users.id` — whose session this is
`logged_in_at`	`DATETIME`	When the session started
`logged_out_at`	`DATETIME`	When the session ended; `NULL` if the user is still logged in

None of these tables store messages or content. They belong entirely to the ums database and the identity domain.

`messages` - the content record

The messages table lives in the twitter database. It is the record of everything posted.

Field	Type	Purpose
`id`	`BINARY(16)`	Primary key - uniquely identifies this message
`author_id`	`BINARY(16)`	The `id` of the user who posted this message, from `ums.users`
`content`	`VARCHAR(280)`	The text of the message - capped at 280 characters
`created_at`	`DATETIME`	When the message was posted

content is capped at 280 characters - the same limit Twitter uses, and a deliberate product constraint, not a technical one. The database enforces it with VARCHAR(280).

author_id is the field that links a message to its author. It holds the id value of a row in ums.users. Because the two tables live in separate databases, MySQL cannot enforce this link with a formal foreign key constraint - but the relationship is real. The application layer is responsible for ensuring that no message is ever written with an author_id that doesn't correspond to a real user.

`subscriptions` - the social graph

There is one more table in the twitter database: subscriptions. It didn't appear in the original three use cases, but it's present in the schema for a good reason - it's the data that would power a personalised timeline.

A subscription is a directional relationship between two users: one subscriber who follows one producer. The table has just three fields:

Field	Type	Purpose
`subscriber_id`	`BINARY(16)`	The user doing the following
`producer_id`	`BINARY(16)`	The user being followed
`created_at`	`DATETIME`	When the follow happened

The combination of subscriber_id and producer_id is the primary key - you can only follow someone once. Both columns are logical foreign keys to ums.users.id, subject to the same cross-database constraint limitation as author_id in messages.

subscriptions is infrastructure for a feature - "Follow a user" - that isn't in scope for the current lesson's use cases but is correct to model now, because adding it later would require a migration. Modelling it upfront costs nothing; omitting it and adding it later costs a schema change and a deployment. This is the kind of forward-thinking that separates a considered data model from a reactive one.

How the Entities Relate

Individual entities are only half the picture. A data model also defines how entities connect to one another - their relationships. For Bird, there are two:

A User has many Messages (one-to-many). One user can post any number of messages; each message belongs to exactly one user. This relationship is expressed through author_id in the messages table - a field that holds the id of the user who owns that row. Following the author_id from a message leads you to its author.

A User can follow many Users, and be followed by many Users (many-to-many). This is the social graph, modelled through the subscriptions table. To find everyone a user follows, query subscriptions where subscriber_id matches. To find everyone following a user, query where producer_id matches.

Figure 5. Types of Relationships

The pattern to remember. A one-to-many relationship is expressed by putting the "one" side's id as a field on the "many" side - the foreign key lives in the child table. A many-to-many relationship requires its own table, with one column for each side. Both patterns appear in Bird, and together they cover the vast majority of real-world data relationships you'll encounter.

With the entities named, the fields defined, and the relationships mapped, the data model is complete. What remains is to choose a database engine and write the schema.

What the Diagrams Will Now Be Built On

Lesson 1 introduced three lenses for looking at a system: functional structure, behaviour, and component interaction. Lesson 1 also made a promise — that diagrams would follow.

They will. But notice what's changed since then.

Before the data model existed, a flowchart for "Post a message" would have had a step labelled something like "save the message" — a placeholder that gestures at an action without defining it. Now that step has a precise meaning: create a row in twitter.messages with a content value, an author_id pointing to the authenticated user in ums.users, and a created_attimestamp set to now.

The data model doesn't just inform the diagrams — it creates clear mechanics how they work. Every box that reads or writes data now has a contract: specific fields, specific tables, specific relationships. The sequence diagram will name what passes between components. The functional diagram will know what each block owns. The flowchart steps will correspond to real operations.

That's where Lesson 3 picks up: with the data model as the foundation, we draw all three diagrams.

Conclusion

A data model is the contract the rest of the system is written against.

We started with three use cases and asked a simple question: what must the system remember for each of these to work? That question led us to two entities — User and Message — and from there to five tables across two purposefully separated databases.

The separation itself was a design decision, not a convenience. Identity and access belong to one domain; content and social graph belong to another. Keeping them apart makes each easier to change, scale, and reason about independently. The trade-off — enforcing cross-database relationships in application code rather than at the database level — is a known and manageable cost.

Every field in the schema exists for a reason traceable back to a use case. Every relationship reflects a real dependency between things the system tracks. Nothing was added for completeness or anticipation — except subscriptions, which earns its place by being cheaper to model now than to migrate in later.

The diagrams come next. They'll have something real to say.

From Idea to Blueprint: Turning a Vague App Concept into Something You Can Actually Build

Eugene Zimin — Sun, 17 May 2026 22:03:47 +0000

Lesson 1 of Build a Twitter Clone — A Practical Guide to Software Modelling

"Build a Twitter clone" is a wish, not a plan — you can't write code from a single sentence. This lesson shows you how to think about turning a fuzzy app idea into a concrete blueprint: we scope a deliberately small messaging app called Bird, then introduce the three-lens model — what should it do?, what parts must exist?, and how do those parts talk to each other? — that the rest of the series uses to actually draw it. By the end you'll have a bounded project and a clear mental model, the foundation Lesson 2 builds on.

Introduction — From Idea to Blueprint

Imagine someone hands you a sticky note that says "build a Twitter clone" and asks you to start coding.

Where, exactly, would you put your cursor?

That question is harder than it looks, and the difficulty isn't about programming skill. It's that the instruction isn't actually buildable. "Build a Twitter clone" is a wish. It tells you the destination but none of the road. Before a single line of code makes sense, that wish has to be transformed into something with edges, parts, and order — a blueprint.

Figure 1. From Idea to Blueprint

This lesson is about making that transformation, and doing it with a tool most people underrate: the humble diagram.

Why an idea isn't a plan

A sentence like "build a Twitter clone" hides an enormous number of decisions. Consider just a few:

Can users edit a post after publishing it, or is it permanent?
What happens when someone submits an empty message? An error? Silence?
Does a post appear instantly, or is there a moment of "sending..."?
Where does the message go once it's submitted — and who is responsible for keeping it?

None of these answers live in the original sentence. They have to be decided, and a working app is really just the sum of hundreds of such decisions made consistent with one another. The gap between "an idea" and "a plan" is precisely the gap of unmade decisions — and code is a terrible place to discover you forgot one.

Figure 2. Why Idea is not Enough

A blueprint isn't bureaucracy. It's the cheapest place to be wrong. Moving a wall on paper costs a pencil eraser; moving it after the concrete is poured costs a demolition crew.

The same logic applies to software. A confused diagram can be fixed in thirty seconds. The same confusion, discovered three weeks into coding, can cost days of rework. Thinking before building isn't slower — it's the thing that makes building fast.

Diagrams: thinking you can point at

When we say diagram, we don't mean decorative boxes-and-arrows that get drawn after the work is done and quietly go stale. We mean something more useful: a diagram is a way of thinking that you can point at.

Plain prose is bad at certain kinds of thought. A paragraph describing how six components pass data around is exhausting to read and easy to get wrong, because language forces ideas into a single line, one word after another. But a system isn't a line — it has shape. It branches. It loops. It has parts that exist alongside each other. A diagram lets that shape sit still on the page so you can inspect it, find the gap, and point a teammate straight at the problem.

That's the real job of the diagrams in this series. They are not documentation. They are the instrument you use to turn the wish into the plan — and, just as importantly, a shared picture a whole team can argue over and agree on.

What this series is — and what this lesson does

This is the first lesson in a hands-on series with one practical goal: build a working Twitter-style application, from blank page to running software.

We won't begin with code, a database, or a framework. We'll begin where every real project begins — with the awkward question "what should this thing even do?" — and we'll answer it the way experienced engineers do: by modelling the system before constructing it.

Across this lesson you'll meet three kinds of diagrams, and the order matters, because each one answers a question the next one depends on:

Behaviour — What should the app do? We'll capture this with a flowchart: the step-by-step flow of a single user action, including its decision points and outcomes.
Structure — What parts must exist to make that behaviour possible? We'll capture this with a functional diagram (also called a block diagram): the system broken into its major components.
Interaction — How do those parts actually talk to each other at runtime? We'll capture this with a sequence diagram: the precise back-and-forth of messages between components.

Figure 3. The Cube

Behaviour first, then the structure that delivers it, then the detailed conversation inside that structure. Each diagram is derived from the one before — you'll watch structure literally fall out of behaviour rather than being invented from thin air.

Don't worry if those three terms mean nothing to you yet. Each gets a plain-language introduction, a worked example, and a step-by-step recipe in its own section. Right now, the only thing to hold onto is the progression: what it does → what it's made of → how the parts interact.

A deliberately small project

There's one honest catch we should name immediately. The real Twitter is gigantic — feeds, ranking algorithms, notifications, direct messages, media uploads, advertising, moderation at planetary scale. We are not building that. Trying to would teach you nothing except how to feel overwhelmed.

Instead, we'll build the smallest thing that is still recognisably a short-messaging app — a deliberately tiny slice we'll call Bird. Choosing that slice on purpose is not a shortcut around the engineering. It is the engineering. Knowing what to leave out — scoping a minimum viable product, or MVP, the leanest version that still does the core job — is one of the most valuable skills in software, and it's the first thing the next section will practise.

So let's pick up the sticky note that says "build a Twitter clone" — and turn it into something you can actually build.

Meet the Project: `Bird`

Before we can draw a single diagram, we need something to draw about. So let's give the vague sticky note a concrete shape.

We're building Bird: a small short-messaging app where people post brief public updates and read other people's. That's the whole pitch. If Twitter is a city, Bird is a single well-built room in it.

The hardest decision is what to leave out

It's tempting to start listing everything Bird could do — likes, replies, hashtags, profile photos, search, notifications, a ranked feed. Resist that. A feature list that grows without limit is the single most common reason beginner projects never ship.

The discipline here has a name: scoping a minimum viable product, or MVP — the leanest version of an app that still does its core job and nothing more. The goal isn't a crippled product. It's a complete one with a deliberately narrow definition of "complete."

A useful test for the MVP line: for each candidate feature, ask "If this were missing, would the app still be recognisably a short-messaging app?" If the answer is yes, the feature is out of Lesson 1 — not deleted forever, just postponed.

Run that test and most of the obvious features fall away. Likes? The app still works without them. Hashtags? Still works. A ranked feed? Still works — you'd just see posts newest-first. But remove the ability to write a post, or to see_posts, and there's no app left at all. Those survive the test. Almost nothing else does.

This gives us our scope. We'll describe it as a short list of use cases — a use case is simply one complete thing a user can accomplish with the app, named from the user's point of view. Use cases are the raw material everything else in this lesson is derived from, so naming them precisely matters.

Figure 4. Defining the MVP - Minimal Viable Product

For now, Bird has exactly three use cases:

#	Use case	What the user accomplishes
1	Post a message	Write a short message (a "bird") and publish it so others can see it.
2	View the timeline	See a list of recently posted messages, newest first.
3	Register an account	Create an identity so posts can be attributed to a person.

Three use cases. That's the entire behavioural scope of the app — and it's meant to feel small.

One use case becomes our worked example

Three use cases is a small scope, but it's still more than we need to teach diagramming. Drawing all three through all three diagram types would be repetitive — you'd learn the technique once and then watch it twice more.

So for the rest of this lesson, we'll follow a single use case end to end: "Post a message."

We chose it deliberately. Of the three, it's the one with real internal richness — it involves the user typing, the app checking the input, something deciding whether to accept or reject it, and the message being stored somewhere. That makes it the ideal teaching subject:

it has a clear start and end, so it suits a flowchart;
it requires several distinct components, so it suits a functional diagram;
it involves a genuine back-and-forth between those components, so it suits a sequence diagram.

"View the timeline" and "Register an account" are real and necessary, but we'll leave them as exercises. Once you've watched "Post a message" travel through all three diagram types, you'll have the pattern — and applying it to the other two becomes straightforward practice rather than new material.

Eventually, we've turned a vague wish ("build a Twitter clone") into a bounded one: an app called Bird, three named use cases, an explicit out-of-scope list, and one chosen worked example. That bounded definition is the first real artifact of the series. The next section introduces the three lenses we'll view it through — and explains why their order is not arbitrary.

Three Lenses, One Progression

We now have a bounded project: the Bird app, three use cases, and one worked example — "Post a message." The next move is to look at that example through three different lenses, each a kind of diagram.

The word lens is the right one. A lens doesn't change the thing you're looking at — it changes what you can see clearly. Point a microscope and a telescope at the same night sky and you learn completely different truths. Our three diagrams work the same way: one system, three views, three distinct questions answered.

The three questions

Every software system can be interrogated from three angles. Each diagram type is just a tool purpose-built to answer one of them.

Lens	The question it answers	The diagram	What you see
Behaviour	What should the app do?	Flowchart	Steps, decisions, and outcomes — the system as a _process_unfolding in time.
Structure	What parts must exist?	Functional diagram	Components and their connections — the system as a thing made of pieces.
Interaction	How do the parts talk?	Sequence diagram	Messages exchanged between components — the system as a conversation.

Notice that none of these is "more correct" than the others. A flowchart can't tell you what components to build; a functional diagram can't tell you what happens when input is invalid. They are not competing descriptions — they are complementary ones. You need all three because no single one is complete.

![[Behaviour, Structure, Interaction - 3 Diagram Types for Multi-Dimensional Software Modelling.png]]
Figure 4. Behaviour, Structure, Interaction - 3 Diagram Types for Multi-Dimensional Software Modelling

An analogy. Think of how you'd describe a car. Behaviour: "you press the pedal, fuel ignites, the wheels turn." Structure: "it has an engine, a transmission, four wheels, a fuel tank." Interaction: "the engine sends torque to the transmission, which sends it to the axle." Each description is true. Each is useless on its own. Together, they let you actually understand — or build — the car.

Why the order is not arbitrary

Here is the part that matters most, and the habit this whole series is built on: the three lenses form a progression, and the progression has a direction.

Figure 5. The order is important

You move behaviour → structure → interaction, and each step is derived from the one before it. This is the single most important idea in the lesson, so it's worth saying slowly:

Start with behaviour. Before you can know what parts a system needs, you must know what it has to do. Behaviour is the requirement; everything else serves it. So we draw the flowchart first.
Let structure fall out of behaviour. Once the flowchart exists, the components almost name themselves. The flowchart has a step "check the message is valid" — so there must be a part that does validation. It has a step "save the message" — so there must be a part that handles storage. You don't invent the structure; you read it offthe behaviour. Each thing the app must do implies a part that does it.
Detail the interaction inside the structure. Now that you know the parts, you can zoom into one slice of the action and show the precise back-and-forth between them — which component sends what message to which other component, and in what order. The sequence diagram lives inside the structure the previous step produced.

Each diagram is the input to the next. That's why the order can't be shuffled. Trying to design components before knowing the required behaviour means guessing — and guessed structure is the kind of mistake that's expensive to discover later. Behaviour-first isn't a stylistic preference; it's how you keep every later decision grounded in something real.

The thread you'll be able to trace

Because each diagram derives from the last, something valuable happens: you can follow a single feature all the way through.

The use case "Post a message" will appear three times in this lesson — once as a flow of steps, once as a set of components, once as a conversation between them. And crucially, you'll be able to point at any element in any diagram and trace it back. "This validation step in the flowchart" becomes "this validator block in the functional diagram"_becomes "this validate() message in the sequence diagram."_ Same idea, three depths of focus.

That property has a name — traceability — and it is the quiet engine of good engineering. When you can trace a feature from "what the user wanted" down to "the specific parts that deliver it," nothing falls through the cracks. When you can't, things do.

The takeaway. Three diagrams, one progression: what it does → what it's made of → how the parts interact. Each is a different lens on the same Bird app, and each is derived from the one before. Hold onto that shape — the rest of the lesson simply walks it, one lens at a time.

Closing this lesson's opening half

That completes the framing half of Lesson 1. You now have everything you need before the hands-on diagramming begins:

a vague wish turned into a bounded project — the Bird app, three use cases, one worked example;
a clear understanding of why blueprints matter — they are the cheapest place to be wrong;
the three-lens mental model — behaviour, structure, interaction — and the reason its order is fixed.

What comes next. That brings Lesson 1 to a close. You haven't drawn anything yet — and that's exactly right. A diagram drawn before the project is bounded and the lenses are understood is a diagram you'll redraw. Lesson 2 is where the building starts: first we define the data model — the precise structure of the information Bird must store and pass between its components — and then we return to the three lenses, drawing the flowchart, functional diagram, and sequence diagram for "Post a message" on top of that data model. The framing you've just finished is what makes those diagrams quick to draw and hard to get wrong.

API Authentication: Part III. JWT Tokens

Eugene Zimin — Sun, 17 May 2026 02:55:46 +0000

Why API Keys Aren't Always Enough

In Part II we saw that an API key is essentially a long, secret password your software shows to a server. It works, but it has a hidden cost: every time the key is used, the server must look it up in a database to find out what the key is allowed to do, whether it has expired, and whether it has been switched off. A JSON Web Token (JWT) removes that lookup by carrying all of that information inside the token itself. This article explains the problem JWT solves and shows where it sits in the larger story of web authentication.

Part I covered Basic Authentication — sending a username and password with every request. Part II covered API keys — replacing that reusable password with a single opaque secret string that identifies an application rather than a person.

Both approaches share a quiet assumption: the server already knows things about the credential, and it has to go and remember them.

Think of an API key as a numbered coat-check ticket. The ticket itself tells you nothing — it's just a number. To find out what the ticket entitles you to, the attendant has to walk into the back room, find the matching record, and read it. The ticket is meaningless without the back room.

That "back room" is a database, and consulting it is called a database roundtrip — the server pauses, sends a question to the database, and waits for an answer before it can continue.

The hidden cost of a roundtrip

For most applications, one database lookup per request is perfectly fine. But consider what the server actually has to check each time an API key arrives:

Permissions — what is this key allowed to do?
Expiry — is this key still valid, or has it passed its end date?
Status — has someone revoked or suspended this key?

All three answers live in the database, not in the key. So every single request triggers a roundtrip before any real work happens.

Figure 1. API Keys and Database Roundtrips

This becomes a problem at scale in two specific situations:

High request volume. A popular API might handle thousands of requests per second. Thousands of identity lookups per second is real, measurable load — and load that does nothing for the user except verify who they are.
Distributed systems. Modern applications are often split into many small, independent services (commonly called microservices — a single application built as a collection of small, separately running programs that talk to each other). If a request passes through five services, and each one independently re-checks the caller's identity against the database, that's five roundtrips for one user action. The database becomes a bottleneck that every service depends on.

Figure 2. API Key Validation Scaling Problem

The deeper issue is state. A server that must consult a database to understand a credential is a stateful server — it cannot make a decision on its own. It always needs the back room.

The core idea behind JWT

JWT flips the model. Instead of handing the server a meaningless ticket and forcing it to look up the details, JWT hands the server a ticket that has the details printed on it — and stamped in a way that proves the printing wasn't forged.

Figure 3. JWT and Stateless Validation

To stay with the analogy: a JWT is less like a coat-check number and more like a boarding pass. A boarding pass already states your name, your flight, your seat, and your boarding time, right there on the paper. The gate agent doesn't phone headquarters to look you up — they read the pass and check that it's genuine. The information travels with the traveler.

This property has a precise name. A server that can validate a token using only the token itself (plus a verification key it already holds) is stateless — it holds no per-request memory and needs no back room. We will unpack exactly how a JWT proves it hasn't been forged later, when we open up its three-part structure. For now, the one idea to carry forward is the trade:

An API key keeps the metadata in the database and gives you a pointer to it. A JWT puts the metadata in the token and gives you a way to trust it.

Figure 4. API Keys vs. JWT comparison

That trade is powerful, but — as Part III will explore honestly — it is not free. Information printed on a token cannot be un-printed, which creates a genuine difficulty when you need to cancel a token early.

A Few Words of History

Every technology arrives at a particular moment for a particular reason. JWT is no exception — it appeared just as the web was shifting away from a model that had quietly dominated for over a decade.

The world before: the session cookie

For most of the 2000s, web authentication worked through server-side sessions. When you logged in, the server created a record of your session in its own memory or database and handed your browser a small identifier — a session cookie. On every later request, your browser sent that cookie back, and the server looked it up to remember who you were.

This is the same coat-check pattern from the previous session above: the cookie is a meaningless number, and the server holds all the real information. It worked well when a website was a single server. But it tied every logged-in user to a specific machine's memory. Spread your traffic across many servers, and you hit a problem — a user logged in on Server A is a stranger to Server B, because Server B never created that session record.

The pressure that created JWT

Two trends in the early 2010s made server-side sessions increasingly awkward.

First, applications stopped being single servers. They were spread across fleets of machines, and increasingly split into microservices — many small programs, each handling one job. A shared, central session store became a bottleneck that every service had to consult.

Second, the clients changed. The web was no longer just browsers loading full pages. It was single-page applications (SPAs) — websites that load once and then behave like desktop apps — and native mobile apps, often talking to APIs hosted on entirely different domains. Cookies, which are tightly bound to a single domain by design, fit this new world poorly.

The industry needed a credential that any server could verify on its own, without a shared session store, and that wasn't anchored to one domain. The answer was to make the token carry its own proof of identity.

RFC 7519: the standard

The work happened inside the IETF (the Internet Engineering Task Force, the body that standardizes the protocols underlying the internet) as part of its OAuth working group — the same group designing the next generation of authorization standards. They needed a compact, secure token format to pass identity information around, and JWT was built to fill that gap.

The result was published in May 2015 as RFC 7519 — the formal specification that defines what a JSON Web Token is. An RFC ("Request for Comments") is the document format the IETF uses for internet standards; despite the tentative-sounding name, a published RFC is the authoritative definition.

A point worth noting: JWT was never a lone invention. It is the centerpiece of a small family of related standards — collectively called JOSE (JavaScript Object Signing and Encryption) — that also define how to sign tokens (JWS), encrypt them (JWE), and represent cryptographic keys (JWK).

For most practical purposes, though, "JWT" is the term everyone uses, and it's the one we'll use here.

From standard to ubiquity

A specification only matters if people adopt it, and JWT's adoption was unusually fast. The identity company Auth0, founded in 2013, built much of its product and developer education around JWT and promoted the format heavily — including jwt.io, a free online debugger that let developers paste in a token and see its decoded contents instantly. For a great many engineers, that tool was their first hands-on encounter with the format.

From there, JWT spread through the ecosystem. It became the default token format for OpenID Connect — the identity layer built on top of OAuth 2.0, and the subject we'll reach later in this series. Cloud platforms adopted it for service-to-service authentication. Web frameworks in nearly every programming language shipped libraries to create and verify it. Within a few years of RFC 7519, JWT had moved from a new proposal to a default assumption.

Why the history matters

This background isn't trivia — it explains the shape of the technology you're about to take apart. JWT looks the way it does because it was designed for a specific world: distributed systems with no shared memory, and clients scattered across domains and devices.

That origin also explains its central trade-off. JWT was built to let a server make a decision on its own, without calling back to a central store. That independence is exactly the feature that makes JWT fast and scalable — and, as we'll see, exactly the feature that makes a token hard to cancel once it's been issued. The strength and the weakness are the same design decision, viewed from two sides.

With that context in place, we can now open up an actual token and see what those three dot-separated parts really contain.

The Anatomy of a JWT

This is the heart of subject. Once you can see what a JWT actually is, every later topic — claims, signatures, validation, security — becomes far easier to follow. So we'll take a real token apart, piece by piece.

Three parts, two dots

A JWT, when written out, looks like a long, slightly intimidating string of gibberish:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NSIsIm5hbWUiOiJBZGEifQ.dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk

It looks random. It isn't. Look closely and you'll spot two dots dividing it into three parts:

header  .  payload  .  signature

That structure never changes. Every JWT, everywhere, is exactly three parts separated by dots:

The header — describes the token itself.
The payload — carries the actual information.
The signature — proves the first two parts haven't been tampered with.

The boarding-pass analogy from the first section holds up neatly here. The header is the small print at the top that says what kind of document this is. The payload is the part you actually care about — your name, your flight, your seat. The signature is the official stamp that proves the pass is genuine and not something printed at home.

Why it looks like gibberish: Base64URL

The reason a JWT looks unreadable is not encryption. It is encoding — and the distinction matters.

Encryption scrambles data so that no one can read it without a secret key.
Encoding simply rewrites data into a different alphabet so it can travel safely. Anyone can reverse it.

A JWT's header and payload are merely encoded, not encrypted. They use a scheme called Base64URL — a way of representing data using only letters, digits, and a couple of symbols that are safe to put inside a web address (a URL).

This leads to the single most important — and most misunderstood — fact about JWTs:

The contents of a JWT are not secret. Anyone who holds the token can decode the header and payload and read everything inside. The signature stops people from changing a token; it does nothing to hide it.

Figure 5. JWT Decoding vs. Hiding Secrets

We'll return to the security consequences of this later. For now, just hold onto it: a JWT protects against forgery, not against reading. Never put a password, a credit-card number, or any other secret inside it.

Part 1: The header

Take the first segment of our example token and reverse the Base64URL encoding, and you get a small block of JSON— JavaScript Object Notation, the simple, human-readable key: value format used everywhere on the modern web:

{
  "alg": "HS256",
  "typ": "JWT"
}

The header is metadata about the token. It answers two questions:

alg ("algorithm") — how the signature was created. Here, HS256. We'll examine the algorithm choices in detail in part of the article; for now, treat it as a label naming the method used to make the stamp.
typ ("type") — what kind of object this is. For a JWT, this is simply "JWT".

The header is short, and it's mostly there so the receiving server knows how to check the signature later.

Part 2: The payload

Decode the second segment and you get another block of JSON — and this is the part that does the real work:

{
  "sub": "12345",
  "name": "Ada"
}

The payload carries the claims — the individual statements the token is making. The word is well chosen: each entry is a claim, an assertion that "this is true." Here the token claims two things: the subject (sub) of this token is user 12345, and that user's name is Ada. The payload is where identity, permissions, expiry times, and your own custom data live.

Part 3: The signature

The third segment is the only part that is genuinely cryptographic — and it's the part that makes a JWT trustworthy.

Here is the problem the signature solves. We just established that anyone holding a token can read the payload. But could they also change it — swap "sub": "12345" for "sub": "99999" and impersonate another user? The answer is "YES". Does it stay hidden? And the most important answer is here - "NO", as the signature is what makes that attack fail because it becomes easily recognizable and server knows how to react on it.

Figure 6. JWT Forgery Protection - The Signature

When the server first creates the token, it performs a calculation. It takes the encoded header and the encoded payload, joins them with a dot, and feeds that — together with a secret key that only the server knows — into a one-way mathematical function. For our HS256 example, that function is HMAC-SHA256. The output is the signature:

signature = HMAC-SHA256(
    base64url(header) + "." + base64url(payload),
    secret
)

Two properties of this function are what make the whole scheme work:

It depends on every input. Change a single character of the header or payload, and the output is completely different.
It can't be reversed or faked without the secret. Knowing the inputs but not the secret, there is no practical way to compute the correct signature.

So when an attacker edits the payload to say "sub": "99999", the signature attached to the token no longer matches the modified content. When the server recomputes the signature from the tampered payload and compares it to the one in the token, the two don't match — and the token is rejected. Because the attacker doesn't have the secret key, they can't produce a fresh, valid signature for their forged payload either.

Figure 7. JWT Signature- The Key to Integrity and Honesty

And this is the core idea worth carrying out of this section:

The signature doesn't keep a JWT private. It keeps a JWT honest. It guarantees that the header and payload are exactly what the issuing server wrote, untouched since.

Building one by hand

The best way to dispel the sense that this is magic is to build a token with no library at all — just the standard tools any programming language provides. Here it is in Python:

import base64, json, hmac, hashlib

def base64url_encode(data: bytes) -> str:
    # Standard Base64, then made URL-safe: trailing '=' padding removed.
    return base64.urlsafe_b64encode(data).rstrip(b'=').decode()

def create_jwt(payload: dict, secret: str) -> str:
    # 1. The header: declare the algorithm and type.
    header = {"alg": "HS256", "typ": "JWT"}

    # 2. Encode the header and payload as Base64URL.
    h = base64url_encode(json.dumps(header).encode())
    p = base64url_encode(json.dumps(payload).encode())

    # 3. Sign "header.payload" with the secret using HMAC-SHA256.
    sig = hmac.new(secret.encode(), f"{h}.{p}".encode(), hashlib.sha256).digest()

    # 4. Join all three parts with dots.
    return f"{h}.{p}.{base64url_encode(sig)}"


token = create_jwt({"sub": "12345", "name": "Ada"}, secret="my-secret-key")
print(token)

That is the entire mechanism. There is nothing hidden. A JWT is two pieces of JSON, encoded for safe transport, plus a signature computed from them. The four numbered steps in the code are the whole story.

The production path: use a library

Building a token by hand is the right way to understand JWTs. It is the wrong way to use them in real software.

In production, always reach for a well-maintained, audited library — PyJWT or python-jose in Python, and direct equivalents in every other major language. The same task becomes a single call:

import jwt  # the PyJWT library

token = jwt.encode(
    {"sub": "12345", "name": "Ada"},
    "my-secret-key",
    algorithm="HS256"
)

This isn't laziness — it's safety. The hand-written version above is correct for creating tokens, but verifying them safely is full of subtle traps. A naive verifier can be tricked into accepting forged tokens, expired tokens, or tokens it should never have trusted. Mature libraries have already absorbed those lessons; your own code, written from scratch, has not.

So build one by hand once, to see that the magic is just arithmetic. Then never do it again in production.

With the structure of a token clear, we can turn to its most important part in detail — the payload, and the claims it carries.

Claims: What a Token Actually Says

In previous section we opened up a JWT and found the payload — the middle segment carrying the token's real content. Each piece of information in that payload is called a claim. This section is about what those claims are, which ones the standard defines for you, and which ones you create yourself.

A claim is just a statement

The terminology sounds formal, but the idea is plain. A claim is a single statement the token makes about its subject — one key: value pair in the payload's JSON. The token "claims" these things are true, and the signature is what makes that claim trustworthy.

A payload is simply a collection of these statements:

{
  "iss": "auth.myapp.com",
  "sub": "12345",
  "aud": "api.myapp.com",
  "iat": 1716120000,
  "exp": 1716123600,
  "role": "editor",
  "tenant_id": "acme-corp"
}

Some of those keys — iss, sub, aud, iat, exp — are part of the JWT standard and mean the same thing everywhere. Others — role, tenant_id — are invented by the application. That split is the central idea of this section.

Registered claims: the standard vocabulary

RFC 7519 defines a small set of registered claims. These are not required — a JWT is free to omit any of them — but if you do use them, you must use them with the meaning the standard assigns. They have short, three-letter names to keep tokens compact.

There are seven worth knowing:

Claim	Name	What it means
`iss`	Issuer	Who created and signed this token.
`sub`	Subject	Who or what the token is about — typically the user ID.
`aud`	Audience	Who the token is intended for — which service should accept it.
`exp`	Expiration Time	The moment after which the token must be rejected.
`nbf`	Not Before	The moment before which the token is not yet valid.
`iat`	Issued At	The moment the token was created.
`jti`	JWT ID	A unique identifier for this specific token.

A few of these deserve a closer look, because they do more than they first appear to.

iss (Issuer) identifies the authority that minted the token — usually an authentication server like auth.myapp.com. A receiving service checks this so it only trusts tokens from a source it recognizes, and ignores tokens minted by anyone else.

aud (Audience) is the one most often misunderstood, and it matters for security. It names the intended recipient. If your authentication server issues tokens for several different services, the aud claim lets each service confirm "this token was meant for me." A token minted for the billing service should be refused by the email service, even though both trust the same issuer. Skipping this check is a real vulnerability.

jti (JWT ID) is a unique serial number for one individual token. On its own it does little. Its importance shows up later: when you need to cancel a specific token before it expires, the jti is the handle you use to do it. That's the revocation problem, and jti is the thread that connects to it.

The temporal claims: `iat`, `nbf`, and `exp`

Three of the registered claims — iat, nbf, and exp — are about time, and together they give a token a lifespan. They are the mechanism behind one of JWT's most important properties: a token that automatically stops working.

Laid out on a timeline, they mark out a window of validity:

   iat            nbf              now                      exp
    │              │                │                        │
    ●──────────────●════════════════●════════════════════════●─────────►  time
  issued       valid from      this moment              expires after
                 │◄───────── token is VALID here ───────────►│

Figure 8. Timeline for JWT claims

iat (Issued At) is the token's birth timestamp — when it was created.
nbf (Not Before) is the moment the token starts being valid. Usually this is the same as iat, but it can be set in the future to issue a token now that only "switches on" later.
exp (Expiration Time) is the moment the token stops being valid. After this instant, every correct server must reject it, no matter what else the token says.

The exp claim is what gives a JWT its self-limiting nature. An API key, by contrast, tends to live until a human remembers to revoke it. A JWT carries its own deadline. Set exp to 15 minutes after iat, and the token simply expires 15 minutes later — no database, no cleanup job, no human action. This is central to managing the risk of a stolen token.

One technical detail: these timestamps are stored as Unix time — the number of seconds since midnight UTC on January 1, 1970. That's why exp appears as a large integer like 1716123600 rather than a human-readable date. It's a universal, timezone-free way to pin down an exact instant, and every JWT library converts it for you.

Custom claims: your own data

The registered claims cover identity and timing. They say nothing about what a user is allowed to do, because that is specific to your application. For everything else, you add custom claims — key: value pairs you define yourself, with whatever names and meanings you choose.

Custom claims are where JWT delivers on the promise from the section above — moving metadata into the token so the server doesn't need a database lookup. Typical examples:

role or permissions — what the user is authorized to do ("editor", "admin", or a list like ["read:invoices", "write:invoices"]). The receiving service reads this straight from the token and decides what to allow — no roundtrip required.
tenant_id — in applications that serve many separate organizations from one system, this identifies which organization the user belongs to.
email, name — basic profile fields, included to save a lookup when displaying the user's identity.

This is exactly the design that makes a server stateless: the answer to "who is this, and what may they do?" travels inside the request itself.

Two cautions, both flowing directly from facts already established.

First — and this bears repeating because it is the most common JWT mistake — the payload is not secret. Anyone holding the token can decode and read it. So custom claims may contain a user's role; they must never contain anything sensitive — no passwords, no API secrets, no private personal data.

Second, keep the payload small. A JWT travels with every single request to your API. Every claim you add makes every request slightly larger. Stick to what the receiving service genuinely needs to make a decision; resist the temptation to pack a user's entire profile into the token.

Avoiding name collisions

If your application invents a custom claim called role, and some other system your token passes through also uses role to mean something different, the two meanings collide. The JWT standard's recommended fix is to namespace custom claims — prefix them with a URL you control, so the name is globally unique:

{
  "sub": "12345",
  "https://myapp.com/role": "editor"
}

The URL doesn't have to point anywhere real; it's used purely as a unique prefix. For a closed system where you control every service, plain names like role are common and perfectly workable. For tokens that travel between organizations, namespacing prevents a class of subtle, hard-to-diagnose bugs.

The payload, in summary

A JWT's payload is a set of claims — statements the token asserts and the signature makes trustworthy. Some claims are registered by the standard: iss, sub, and aud answer who; iat, nbf, and exp answer when; jti gives the token a unique name. The rest are custom claims you define, and they are how identity and permissions ride along inside the request, sparing the server a database lookup.

We now know what a token says. The next question is how it proves it — which means looking closely at the signature, and the different algorithms used to create it.

Signature Algorithms: HS256, RS256, and ES256

Above we established what the signature does: it makes a token honest, guaranteeing the header and payload haven't been altered since the issuer wrote them. We used HS256 as the example and treated it as a black box. Now we open the box — because the choice of signing algorithm is one of the most consequential decisions you'll make when building a real system, and it turns entirely on a single question: who is allowed to verify the token?

Two families: symmetric and asymmetric

Every JWT signing algorithm belongs to one of two families. The difference between them is the difference between a shared password and a wax seal.

Figure 9. JWT Signing - Symmetric vs. Asymmetric

A symmetric algorithm uses one secret key for both jobs — creating the signature and checking it. The same key signs and verifies. It's like a password shared between two parties: whoever knows it can both lock and unlock.

An asymmetric algorithm uses a pair of mathematically linked keys: a private key and a public key. The private key creates signatures and is guarded closely. The public key only verifies signatures and can be shared freely — handed to anyone, posted publicly — without weakening anything. It's like a wax seal: the issuer owns the unique stamp (the private key), but anyone with a picture of the seal (the public key) can recognize a genuine one. Holding the public key lets you check a seal but never forge one.

That single distinction — one shared key versus a public/private pair — drives everything that follows.

HS256: the symmetric option

HS256 stands for HMAC using SHA-256. It is the symmetric choice, and it's the algorithm in every example so far.

With HS256, one secret string both signs and verifies tokens. The server that issues a token uses the secret to compute the signature; the server that receives a token uses the same secret to recompute the signature and compare. If they match, the token is genuine.

This is simple, fast, and produces compact signatures. But it carries one unavoidable consequence: every party that needs to verify a token must hold the secret key. And here is the catch — the verifying key and the signing key are the same key. Any service that can check a token can therefore also mint a brand-new token that everyone else will trust.

In a single, self-contained application, that's fine. The same server issues tokens and checks them. There's one secret, in one place, and no one else needs it.

The trouble begins when the system grows.

RS256: the asymmetric option

RS256 stands for RSA Signature using SHA-256. RSA is a long-established public-key cryptosystem; what matters here is that RS256 is asymmetric — it uses the private/public key pair.

With RS256, the authentication server holds a private key and uses it — and only it — to sign tokens. It then publishes the matching public key for the world to see. Any other service can take that public key and verify a token's signature. But the public key cannot sign anything. A service holding only the public key can confirm a token is genuine, yet has no power to forge one.

That asymmetry solves the exact problem HS256 created.

The "aha": why this matters for microservices

Recall the microservices picture from Figure 1 — an application split into many small, independent services. Suppose one of them, the authentication service, issues tokens, and a dozen others need to verify them.

With HS256, every one of those dozen services must hold the shared secret. That's a real problem on two fronts:

A wider attack surface. The secret now lives in twelve places instead of one. A breach of _any single service_leaks the key — and with it, the power to forge tokens that all twelve will trust.
Misplaced trust. Every service that can verify can also issue. A minor, low-privilege service now holds the keys to impersonate anyone. That violates a basic security principle: a component should hold only the power it actually needs.

With RS256, the picture changes completely:

The private key lives in exactly one place — the authentication service. Only it can mint tokens.
The public key is distributed to all twelve verifying services. It's not a secret; if one leaks, nothing is compromised, because a public key cannot forge anything.
Verifying services can do their job — confirming a token is genuine — without ever holding the power to create one.

Figure 10. Token Validation in Microservices - Securing the Keys

This is the central insight of the section, and it's worth stating plainly:

In a distributed system, use an asymmetric algorithm. It lets you separate the power to issue tokens from the power to verify them — concentrating the dangerous capability in one guarded place while distributing the harmless one freely.

ES256: asymmetric, but leaner

ES256 stands for ECDSA using SHA-256, where ECDSA is the Elliptic Curve Digital Signature Algorithm.

For decision-making purposes, ES256 belongs in the same category as RS256: it is asymmetric, with a private key for signing and a public key for verifying, and it solves the microservices problem in exactly the same way. The difference is mechanical. ES256 is built on elliptic-curve cryptography, a more modern branch of public-key cryptography that achieves the same security strength with much smaller keys and signatures.

The practical payoff is size. An RS256 signature is large; an ES256 signature, at comparable security, is substantially smaller. Since a JWT travels with every request, a smaller signature means a smaller token on every call. For a high-traffic API, that adds up.

The trade-off is maturity and ubiquity. RSA has been in use for decades and is supported absolutely everywhere; elliptic-curve support, while now excellent, is marginally less universal in older systems. In practice, both are solid choices — RS256 is the safe, conventional default, and ES256 is the leaner, more modern alternative.

Putting it side by side

	HS256	RS256	ES256
Family	Symmetric	Asymmetric	Asymmetric
Key(s)	One shared secret	RSA private/public pair	EC private/public pair
Who can verify?	Only holders of the secret	Anyone with the public key	Anyone with the public key
Can a verifier also forge tokens?	Yes — same key signs and verifies	No — public key can't sign	No — public key can't sign
Signature size	Small	Large	Small
Best fit	A single, self-contained service	Distributed systems / microservices	Distributed systems where token size matters

The decision rule is short. If your application is a single service that both issues and verifies its own tokens, HS256 is simple, fast, and entirely appropriate — there's no second party, so a shared secret shares nothing. The moment more than one independent service must verify tokens, switch to an asymmetric algorithm: RS256 as the well-supported default, ES256 when you want smaller tokens and your platform supports it. The question is never "which algorithm is best" in the abstract — it is "who, in my system, needs to verify a token, and should those same parties be able to create one?"

We now know how a token is signed. But verifying a token correctly involves much more than checking that one signature. The next section lays out the full validation checklist — and shows why skipping any step on it opens a door.

Validation: The Full Checklist

Here is a mistake that has shipped to production in countless real systems: a developer wires up JWT authentication, confirms the signature checks out, and considers the job done. The signature is valid — so the token is trustworthy. Right?

Not quite. A valid signature answers only one question: was this token altered since it was issued? It says nothing about when the token was issued, who issued it, who it was meant for, or whether it has been cancelled in the meantime. A token can have a perfect signature and still be one your server must refuse.

Proper validation is therefore a checklist, run in order. This section walks through all six steps, mirroring the security rigor of the earlier parts of this series.

Why the order matters

The steps are sequenced deliberately, and the first one comes first for a reason: until the signature is fully verified, you cannot trust a single byte of the payload.

Every later check — expiry, issuer, audience — reads a value from the payload. But if the signature hasn't been confirmed, the payload might be an attacker's forgery, and an attacker writes their exp claim to say whatever they like. Checking expiry on an unverified payload is theatre. So the signature is verified first; only once it passes does the payload become trustworthy enough to inspect.

The six-step checklist

Step 1 — Verify the signature. Recompute the signature from the token's header and payload (using the secret for HS256, or the public key for RS256/ES256) and confirm it matches the signature attached to the token. If it doesn't, the token is forged or corrupted — reject it immediately and run no further checks. If it matches, the payload is now proven authentic, and the remaining steps can rely on it.

Step 2 — Check exp (not expired). Read the expiration timestamp and confirm the current time is before it. An expired token is rejected even though its signature is flawless. This is the check that makes a stolen token stop working on its own — and the entire reason short expiry times are worth setting.

Step 3 — Check nbf (not used before valid). If the token carries a "not before" claim, confirm the current time is past it. A token presented before its activation moment is not yet valid and must be refused. In practice this rarely fires, but a complete validator checks it.

Step 4 — Check iss (expected issuer). Confirm the iss claim names an authentication server you actually trust. A token can be perfectly signed by some issuer, but if it isn't your issuer, it's irrelevant — reject it. This stops your service from honouring tokens minted by an unrelated, untrusted source.

Step 5 — Check aud (intended audience). Confirm the aud claim names your service. This is the check which is most often skipped — and skipping it is a genuine vulnerability. Without it, a token issued for the low-privilege analytics service would be accepted by the high-privilege billing service, simply because both trust the same issuer. The aud check is what keeps a token usable only where it was meant to be used.

Step 6 — Check jti against a revocation list (if needed). For most stateless setups this step is deliberately omitted — checking a token against a list means consulting a database, which sacrifices the very statelessness JWT was chosen for. But when you genuinely need the ability to cancel an individual token before it expires, this is where it happens: look up the token's jti and reject it if it appears on a blocklist. This step is the bridge revocation problem, and the reason it's marked "if needed" is itself the subject of that section.

What this looks like in code

Previously we said: build a token by hand once to understand it, then use a library in production. Validation is precisely where that advice earns its keep. A good library performs the entire checklist for you — if you tell it what to expect:

import jwt  # the PyJWT library

def validate_jwt(token: str, secret: str, expected_audience: str) -> dict:
    try:
        payload = jwt.decode(
            token,
            secret,
            algorithms=["HS256"],          # Step 1: which algorithm(s) to trust
            audience=expected_audience,    # Step 5: the aud claim must match this
        )
        # Steps 2 and 3 — exp and nbf — are checked automatically by decode().
        return payload

    except jwt.ExpiredSignatureError:
        raise AuthError("Token has expired")          # Step 2 failed
    except jwt.InvalidAudienceError:
        raise AuthError("Invalid audience")           # Step 5 failed
    except jwt.InvalidSignatureError:
        raise AuthError("Signature verification failed")  # Step 1 failed
    # ... and so on for the other failure cases

Three details in that small function carry the section's whole lesson.

First, jwt.decode() does several checks at once. A single call verifies the signature, the expiry, and the "not before" time. The library runs the checklist — but only the parts you've configured.

Second, a check you don't ask for doesn't happen. Notice audience=expected_audience is passed explicitly. Leave that argument out, and many libraries simply skip the audience check entirely — Step 5 silently vanishes, and no error is ever raised. The token validates "successfully," and a real vulnerability sits quietly in your code. The same applies to the issuer. Safe validation means actively telling the library what to expect; its defaults are not your security policy.

Third — and this is the most important line in the whole snippet — algorithms=["HS256"] is not optional. It tells the library exactly which signing algorithm(s) to accept. Omitting it, or filling it in carelessly, opens the single most infamous JWT vulnerability of all. We've now mentioned it twice as a forward pointer; later we finally show the attack itself.

The takeaway

A valid signature is the first line of a six-line checklist, not the whole of it. Full validation confirms a token is authentic(signature), current (exp, nbf), from a source you trust (iss), meant for you (aud), and not revoked (jti, when required). A production-grade library will run every one of those checks — but only the ones you explicitly configure. The unsafe defaults are silent, and silence, in security, is exactly the danger.

That sixth step — revocation — has been flagged twice now as something deferred. It's deferred because it isn't a simple checklist item; it's a genuine tension at the heart of the whole JWT design. The next section confronts it directly.

The Stateless Advantage — and the Revocation Problem

This is the most important section in the article, and the most honest. Everything so far has built toward a single payoff — and that same payoff comes bundled with a single, genuine flaw. A clear-eyed engineer needs both halves. This section gives you both.

The win: a server that needs no memory

Return to the very first idea of this series. An API key is a meaningless ticket; to learn what it permits, the server must make a database roundtrip to the back room. A JWT prints the details on the ticket itself and stamps them so they can't be forged.

Now we can state precisely what that buys you. When a JWT arrives, the receiving server checks it using only the token and a verification key the server already holds — the shared secret for HS256, or the public key for RS256/ES256. The signature confirms the token is authentic; the claims inside answer who the user is and what they may do. Nothing else is consulted.

That property is statelessness: the server keeps no per-user, per-session memory. It needs no shared session store, no identity database lookup on the request path. And the benefits compound:

Speed. No network trip to a database before real work begins. Verification is local arithmetic.
Scale. Add a hundred new servers and not one of them needs a connection to a session store. Each verifies tokens entirely on its own.
Resilience. Recall the microservices picture from figure 2 — a dozen services each verifying tokens. With JWT, none of them shares a session-store bottleneck. If that store would have existed, it would have been a single point of failure every service depended on. JWT removes it.

This is the killer feature. It is the reason JWT became the default token of the distributed-systems era.

The problem: you can't un-issue a token

Now the honest half.

The very thing that makes a JWT powerful — it is self-contained, valid purely on its own evidence — is the thing that makes it dangerous. A stateless server's logic is simply: "the signature is good and the token hasn't expired, therefore I trust it." That logic has no step that asks anyone's permission. It never phones home.

So consider: a token is stolen. An attacker copies a user's valid, unexpired JWT — through a compromised device, an intercepted request, a leaked log file.

With an old-style server-side session, the fix is instant. You delete the session record from the server. The next request carrying that session is a stranger; access is gone forever.

With a JWT, there is no record to delete. The token's authority lives inside the token, in the attacker's possession. Every server that sees it performs the same local check, gets the same answer — signature good, not expired — and grants access. The token keeps working, in the attacker's hands, until its exp timestamp passes. And nothing you do can hurry that moment along.

State this plainly, because it is the crux of the entire JWT trade-off:

A JWT is self-contained by design, which means it is non-revocable by design. The server gave up its memory in exchange for speed — and a server with no memory has no way to remember that one particular token should now be refused.

This is not a bug to be patched. It is the direct, unavoidable shadow of the feature. Statelessness and instant revocation are two ends of the same stick: you cannot pick up one end without the other coming with it.

Living with the trade-off

You can't eliminate the problem, but you can manage it. Three patterns are used in practice, and they sit on a deliberate spectrum — from "fully stateless, accept some risk" to "give back some statelessness, regain control."

1. Short expiry times

The simplest response: if a stolen token is valid until exp, then make exp soon.

Set tokens to expire in fifteen minutes rather than twenty-four hours. A stolen token is still unstoppable — but only for a short window. The damage is time-boxed. This keeps the server fully stateless; you simply accept a small, bounded exposure instead of trying to abolish it.

The objection is obvious, and it's a usability one: a token that expires every fifteen minutes would log the user out every fifteen minutes. Forcing someone to re-enter their password that often is unworkable. Which is exactly why short expiry never travels alone — it travels with the next pattern.

2. The access token + refresh token pattern

This is the pattern most production systems actually implement, and it resolves the usability objection cleanly by splitting one token into two tokens with two different jobs.

The access token is a JWT, short-lived (say, 15 minutes). It rides along with every API request and does the real authorization work. Being a JWT, it is verified statelessly — fast, local, no database. If stolen, it's dangerous for only 15 minutes.
The refresh token is long-lived (days or weeks), but it is not used for ordinary API calls. Its only power is to ask the authentication server for a fresh access token. It is sent rarely, to one endpoint only, and it is typically a stored, revocable credential — the auth server does keep a record of it.

Figure 11. Access & Refresh Token Split Pattern

The whole lifecycle is as follows:

1. User logs in.
   → Auth server issues:  a short-lived ACCESS token  +  a long-lived REFRESH token.

2. For each API request:
   → Client sends the ACCESS token.  Service verifies it statelessly. Fast.

3. After ~15 minutes:
   → The ACCESS token expires. The next API call is rejected.

4. Client quietly sends the REFRESH token to the auth server.
   → Auth server checks the refresh token is still valid (and not revoked),
     then issues a brand-new short-lived ACCESS token.

5. Loop back to step 2. The user notices nothing.

The elegance is in the division of labour. The token used constantly (the access token) is stateless and fast, exactly where speed matters — but barely dangerous, because it expires almost immediately. The token that is long-lived and genuinely dangerous (the refresh token) is used rarely, hits only one endpoint, and can be revoked at that one chokepoint. To cut off a compromised user, you invalidate their refresh token: within fifteen minutes their access token expires, the refresh fails, and they're locked out.

You haven't achieved truly instant revocation — there's still that ≤15-minute tail — but you've shrunk the unstoppable window to something tolerable while keeping the request path stateless. For most systems, that is the right balance.

3. A revocation blocklist

When even a fifteen-minute window is unacceptable — banking, healthcare, anything where a compromised session must die now — you need true immediate revocation. And here you must pay the honest price.

The pattern: recall the jti claim, the token's unique serial number. To revoke a token immediately, add its jti to a blocklist — a list of "tokens that must now be refused" — and have every service check incoming tokens against it. To keep that check fast, the list lives in a high-speed in-memory store such as Redis, not a slow disk database.

Be clear-eyed about what this costs. Checking a blocklist means consulting a shared store on every request — which is precisely the database roundtrip that JWT was chosen to eliminate. You have, deliberately, given back the stateless advantage.

But the surrender is partial, and that nuance matters:

A blocklist holds only revoked tokens — a tiny set — not every session. The store stays small.
An in-memory store like Redis is far faster than the relational-database lookup the original API-key model implied.
Entries can be evicted the moment a token's natural exp passes — a revoked token already rejected by the expiry check needs no blocklist entry. The list stays small on its own.

So this isn't a wholesale return to stateful sessions. It's a precise, conscious concession: trade a little speed, on a small fast lookup, to buy back the immediate-revocation capability — but only when the application's risk profile genuinely demands it.

The honest summary

JWT's defining feature and JWT's defining flaw are the same fact, seen from two sides. A self-contained token lets a server decide on its own — fast, scalable, memoryless. A self-contained token also cannot be un-decided — once issued, it is authoritative until it expires.

There is no pattern that makes that flaw vanish. There are only positions on a spectrum:

Approach	Statelessness	Revocation speed	Typical use
Short expiry only	Fully stateless	None — wait for `exp`	Low-risk, simple systems
Access + refresh tokens	Stateless request path	Bounded — within the access token's lifetime	The mainstream default
`jti` blocklist	Partly stateful again	Immediate	High-stakes systems only

The engineering question is never "how do I make JWT revocable?" — it can't be made so without giving something up. The question is: how much immediate-revocation power does this system genuinely need, and how much statelessness am I willing to trade for it? Answer that honestly, pick your point on the spectrum, and you are using JWT well.

We've now covered what a JWT is, what it carries, how it's signed, how it's validated, and its central trade-off. One thing remains: the specific ways JWT implementations get attacked — including the single most infamous JWT vulnerability of all.

Security Considerations

A JWT is a security mechanism, which makes its own failure modes worth studying with care. Most JWT disasters are not failures of the cryptography — the math is sound. They are failures of implementation: a check skipped, a default trusted, a token stored carelessly. This section covers the four that matter most, beginning with the most famous of them all.

The algorithm confusion attack: `alg: none`

We have pointed at this vulnerability twice and promised to show it here. It is the most infamous JWT bug ever, and what makes it memorable is that it turns the token's own honesty mechanism against itself.

Recall the structure of the token. The header declares which algorithm was used to sign the token, in its alg field — "alg": "HS256". The signature is the stamp that proves the token wasn't altered.

Now recall a detail almost no one expects: the JWT standard defines a legitimate algorithm value called none. It means "this token is unsigned." It exists for narrow cases where a token's integrity is already guaranteed by some other layer. It also creates a trapdoor.

Here is the attack, step by step:

1. The attacker takes a real, valid token and decodes it
   (remember - the payload is NOT secret, anyone can read it).

2. The attacker EDITS the payload freely:
        "sub": "regular-user"   →   "sub": "admin"

3. The attacker changes the header:
        "alg": "HS256"          →   "alg": "none"

4. The attacker DELETES the signature entirely,
   leaving a token that ends in a final dot and nothing after it:
        header.payload.

5. The token is sent to the server.

Now everything depends on one line of the server's code. A naively written verifier reads the header, sees "alg": "none", and reasons: "The header says this token is unsigned, so there is no signature to check. No signature to check means nothing fails. Token accepted."

Figure 12. JWT Attack with "alg" set to "none"

The forged token sails through. The attacker is now admin. They never needed the secret key, never broke any cryptography — they simply let the token tell the server how to verify it, and the token, under attacker control, said "don't bother."

That is the heart of the trap: a naive verifier trusts the token's own header to decide how the token should be checked. But the header is attacker-controlled. You have let the thing being inspected dictate the terms of its own inspection. (If this sounds like the embedded-public-key problem discussed earlier — a token vouching for its own trust — it is exactly the same mistake, in a different costume.)

*The fix is one line, and you have already seen it:

jwt.decode(token, secret, algorithms=["HS256"])

That algorithms=["HS256"] argument is the entire defense. It instructs the library: "I will accept HS256 and nothing else. I don't care what the token's header claims." A token arriving with alg: none is now rejected before any verification logic runs, because none is not on the list the server chose.

The principle, stated generally:

Never let the token decide how it should be verified. The set of acceptable algorithms is a decision the server makes in advance and enforces — not a value it reads out of the untrusted header.

Modern, well-maintained libraries now refuse alg: none by default and require you to specify allowed algorithms — which is exactly why we insisted on using a real library rather than hand-rolling verification. But the lesson outlives this one bug: a verifier must hold its own policy, never inherit it from the token.

A second face of algorithm confusion: HS256 vs RS256

There is a subtler cousin of the same attack.

Recall the asymmetric model: with RS256, the server signs with a private key and publishes the public key for anyone to verify with. The public key is, by design, not secret.

Now picture a server configured for RS256 but with a careless verifier that accepts whatever algorithm the header names. An attacker takes the public key — which is freely available — and uses it as if it were an HS256 shared secret. They forge a token, set the header to "alg": "HS256", and sign it using the public key as the HMAC secret.

Figure 13. RS256/HS256 Key Confusion Attack Path

When the token arrives, the confused server sees alg: HS256, fetches its RS256 public key as the "secret," runs the HS256 check — and it passes. The attacker has forged a valid token using only public information.

The fix is the same fix: pin the accepted algorithm. A server expecting RS256 must specify algorithms=["RS256"] and refuse everything else, so a token claiming HS256 is rejected outright. Once again — the server decides, not the token.

Secret strength for HS256

This one is simple and easily neglected. With HS256, the security of every token rests entirely on one shared secret. The signature is only as strong as that secret is hard to guess.

A secret like "secret", "password123", or your application's name is not a secret — it is an invitation. An attacker who suspects HS256 can run an offline brute-force attack: take any real token they've captured, and rapidly try millions of candidate secrets, checking each by recomputing the signature. A weak secret falls in seconds. Once it's found, the attacker can mint unlimited valid tokens for anyone.

The defense is unglamorous but absolute:

Use a long, high-entropy, randomly generated secret — at least 256 bits (32 random bytes), produced by a cryptographic random generator, never typed by a human.
Store it as a secret — in a secrets manager or environment configuration, never hard-coded in source, never committed to version control.
This concern is HS256-specific. With RS256/ES256 there is no shared secret to guess; the private key is a different kind of object, guarded differently.

Token storage in the browser

When a JWT is used to authenticate a user in a web browser, a practical question arises: where does the browser keep the token between requests? This debate is mostly relevant to browser-based apps rather than the pure API-to-API use that is this series' main focus, so we'll keep it brief — but it's worth knowing the shape of it.

There are two common choices, each with a different weakness:

localStorage is a simple in-browser storage area. It's easy to use, but any JavaScript running on the page can read it. If an attacker manages to inject a malicious script into your site — a cross-site scripting (XSS) attack — that script can read the token straight out of localStorage and steal it.
An httpOnly cookie is a cookie the browser marks as off-limits to JavaScript. Even a successful XSS attack cannot read it directly, which closes the theft route above. The trade-off: cookies are sent by the browser automatically, which can expose the app to a different attack class — cross-site request forgery (CSRF) — that needs its own separate defenses.

There is no universally "correct" answer; each option trades one risk for another. The genuinely important point sits underneath both:

A JWT is a bearer token — like cash, whoever holds it can spend it. The token itself contains no proof that the right person is presenting it. So the security of any browser-based JWT system depends heavily on simply not letting the token get stolen — and that depends on the surrounding application being free of injection flaws.

Short expiry as a discipline

The final consideration is not a new attack but a habit that blunts all of them — and we've already met it.

Every threat in this section ends the same way: an attacker obtains a token they shouldn't have. And we already know JWT's hard truth — a stolen token cannot be recalled; it is valid until its exp.

This makes the expiry time itself a security control. A long-lived token turns any single theft into a long-lived breach. A short-lived one — minutes, not days — means a stolen token is a briefly useful prize, paired with the refresh-token pattern so users never feel the churn.

Short expiry doesn't prevent theft. It caps the cost of theft. It is the safety net beneath every other mistake, and it should be treated as non-negotiable discipline rather than a tuning knob.

The thread connecting all four

Step back and the four considerations rhyme. The algorithm confusion attack is trusting the token to define its own verification. Weak HS256 secrets are trusting an easily guessed value to be hard. Careless token storage is trusting a hostile environment to keep a bearer token safe. And the antidote running through all of them is the same instinct: trust must be anchored in decisions the server makes and controls — never in something the token, the attacker, or the environment supplies. The cryptography in JWT is rarely what fails. The trust assumptions around it are.

With the failure modes mapped, only the practical verdict remains: when JWT is the right tool for the job — and when it isn't.

When to Use JWT — and When Not To

We've taken JWT apart completely: its structure, its claims, its signatures, its validation, its central trade-off, and its failure modes. Knowing how a tool works is not the same as knowing when to reach for it. This section is the practical verdict — a decision guide for an engineer building a real system.

The honest framing is this: JWT is not a better credential than the API key from Part II. It is a different credential, built for a different job. Picking the right one means matching the tool to the situation.

When JWT is the right choice

JWT earns its place whenever its defining feature — a self-contained, statelessly verifiable token — solves a problem you actually have.

Stateless microservices — the killer use case. This is the scenario JWT was built for, and the one where it has no real rival. When an application is split into many small services, and a request may pass through several of them, a JWT lets each service verify the caller independently — with only a public key, no shared session store, no database roundtrip. The token carries identity and permissions with it. If there is one situation where JWT is unambiguously the answer, this is it.

Cross-domain authentication — SPAs and mobile apps. Recall that JWT emerged partly because session cookies are bound to a single domain and fit modern clients poorly. A JWT is just a string. It travels comfortably from a single-page app or a native mobile application to an API hosted on an entirely different domain, with none of the domain-binding friction cookies impose. When your client and your API don't share an origin, JWT removes a real obstacle.

Short-lived service-to-service tokens. When one internal service needs to call another and prove "this request is genuinely from me," a short-lived JWT is an excellent fit. Its built-in expiry means these machine-to-machine tokens clean up after themselves — no long-lived credential left lying around, no revocation process to run. They are minted, used, and expire, all within minutes.

As a session-scoped layer on top of a long-term API key. This is the subtlest fit, and it shows JWT and the API key working together rather than competing. An API key (Part II) is excellent for stable, long-term identity — "this is the Acme Corp account." A JWT is excellent for short-lived, fine-grained authorization — "this particular request, in this 15-minute window, may write to invoices." A common production pattern: a client authenticates once with its long-lived API key, and in exchange receives a series of short-lived JWTs that carry the actual per-request permissions. The API key answers who you are over time; the JWT answers what you may do right now.

When to reach for something else

JWT's weaknesses are not hidden and we stated them plainly. Two situations expose them directly, and in both, a different tool is the honest choice.

When you need immediate, reliable revocation. This is JWT's defining flaw, and it is worth refusing to paper over. A standard JWT cannot be un-issued — it is valid until exp, full stop. If your system has a hard requirement that a compromised credential must die the instant you say so — with no fifteen-minute tail, no acceptable window — then a self-contained token is fighting your requirement.

The alternative here is an opaque token checked by introspection: the token is a meaningless reference string (like the API key of Part II — a coat-check ticket), and the receiving service asks a central authority, on every request, "is this token still valid?" That is a database roundtrip. You are deliberately giving up statelessness — but in exchange you get exactly the instant, authoritative revocation that statelessness cost you. For high-stakes systems, that is the right trade.

When the token is genuinely long-lived. If a credential is meant to last for months — a stable identifier for a server, a long-running integration — then JWT's machinery is working against you. Its central feature, the self-contained expiring claim set, brings no benefit when nothing is meant to expire soon, and its central flaw, non-revocability, becomes a serious liability over a long lifespan. For durable, long-term identity, a plain API key is simpler, and — because it's checked against a database anyway — it can be revoked the moment you need to. Don't reach for a JWT's complexity to do an API key's job.

The decision in one table

Your situation	Reach for	Why
Many services must verify a caller independently	JWT	Stateless verification, no shared session store
Client and API live on different domains	JWT	A plain string travels anywhere; no cookie domain-binding
Short-lived, self-expiring service-to-service calls	JWT	Built-in expiry; no revocation process needed
Stable, long-term identity for an account or server	API key (Part II)	Simpler; revocable; nothing needs to expire soon
A compromised credential must be killable instantly	Opaque token + introspection	True immediate revocation, at the cost of statelessness

The underlying judgment

Reduce all of the above to a single question and it becomes easy to remember:

Does this system need a server to decide on its own, fast and at scale — or does it need a central authority to stay in control, able to revoke instantly?

JWT is the tool for the first. It trades central control for local, stateless speed — a brilliant trade when speed and scale are what you need, a poor one when control is. Opaque tokens and API keys are the tools for the second.

There is no "best" credential, only a credential that fits. An engineer who can name why they chose JWT — and name what they gave up to get it — is using it well. One who reaches for it by default, because it is the modern-sounding option, has skipped the only question that matters.

That leaves one piece of this series unfinished. JWT answers, beautifully, the question who are you? It does not answer a different and harder one: how do you let a third party act on your behalf — without ever handing them your password? That is the problem the final part was built to solve.

Conclusion: Three Tools, Three Jobs

We began this article with a coat-check ticket and we end it with a boarding pass. That shift — from a meaningless token the server must look up, to a self-describing token the server can read and trust on sight — is the whole of what JWT contributes. It's worth gathering the journey into a single picture before we close.

What JWT actually gave us

Strip away the Base64URL, the claims tables, the algorithm names, and one idea remains: a JWT moves metadata out of the database and into the token itself, stamped so it cannot be forged.

That single move is the source of everything else in this article. It is why a server can be stateless — verifying a request with only a key it already holds, no roundtrip, no shared session store. It is why JWT scales so gracefully across microservices and travels so freely across domains. And it is — unavoidably — why a JWT cannot be un-issued: a server that gave up its memory for speed has no memory in which to record that one token should now be refused.

The strength and the flaw are the same fact seen from two sides. An engineer who holds both halves in mind at once understands JWT. One who remembers only the strength will, sooner or later, be surprised by the flaw.

Three methods, side by side

This series has now equipped us with three distinct credentials. They are not a ranking — not worse, better, best. They are three tools for three different jobs:

	Basic Auth (Part I)	API Key (Part II)	JWT (Part III)
What it is	Username + password sent on every request	A single opaque secret string	A signed, self-describing token
Carries its own data?	No	No — it's just a reference	Yes — identity, permissions, expiry, all inside
Server needs a lookup?	Yes, every request	Yes, every request	No — verified statelessly with a key
Built-in expiry?	No	No	Yes — the `exp` claim
Revocable immediately?	Yes (change the password)	Yes (delete the key)	No — valid until it expires
Best at	Simple, internal, low-stakes access	Stable, long-term identity	Stateless, distributed, short-lived authorization

Read the table top to bottom and the trade-off snaps into focus. Basic Auth and the API key keep their information in the database — which makes them slower per request, but instantly revocable. JWT carries its information in the token — which makes it fast and stateless, but stubbornly non-revocable. You cannot have both. Every authentication design is, at heart, a choice about where the trust lives: in a central store the server must consult, or in the token the server can read on its own.

Choosing well means naming that trade-off out loud — not reaching for the most modern-sounding option by reflex, but asking what this system actually needs, and what it can afford to give up.

The one question that remains

And yet, for all three of these methods, notice what they have in common. Basic Auth, the API key, and the JWT all answer the same single question:

"Who are you?"

They are mechanisms of authentication — proving identity. Each one assumes the party presenting the credential is the party it belongs to, acting on its own behalf.

Figure 14. Authentication vs. Authorization

But a great deal of the modern web does not work that way. When you let a photo-printing service reach into your cloud storage, or allow an analytics dashboard to read your calendar, something more delicate is happening. You are granting a third party access to your resources — and you would never dream of handing them your password to do it.

None of the three tools in this series solves that. A JWT can prove who you are beautifully; it has nothing to say about how you safely delegate a sliver of your access to someone else.

That problem — how to grant a third party limited access to your resources without ever sharing your credentials — is precisely the problem the next and final part of this series was built around. It is the problem of authorization and delegation, and the answer is the framework that quietly underpins nearly every "Sign in with…" button and connected app on the internet: OAuth 2.0.

We have spent three articles learning, in ever-greater depth, how to answer who are you? It is time to ask the harder question.