DEV Community: Eugene Zimin

Drawing the Blueprint: Flowchart, Functional Diagram, and Sequence Diagram

Eugene Zimin — Sat, 23 May 2026 20:45:57 +0000

Lesson 3 of Build a Twitter Clone - A Practical Guide to Software Modelling

In Lesson 2, we defined what Bird must remember: two databases, tables within, every field traced back to a use case. Now it's time to draw what Bird must do. Using "Post a message" as our single worked example, we construct all three diagrams in sequence - first a flowchart that traces the user action step by step, then a functional diagram that reads the required components directly off that flowchart, then a sequence diagram that shows exactly how those components talk. By the end, the use case that started as a sentence on a sticky note has a complete, traceable blueprint.

Introduction - The Foundation Is Ready

Lesson 2 ended with a promise, and it's worth quoting precisely:

"That's where Lesson 3 picks up: with the data model as the foundation, we draw all three diagrams."

That moment has arrived. But it's worth pausing for a second to notice what's different now compared to where we started.

In Lesson 1, we introduced the idea of three diagrams - a flowchart, a functional diagram, and a sequence diagram - as three lenses on a single system. We explained the mental model, argued for the order, and left it there. No diagrams were actually drawn, and that was deliberate: a diagram drawn before the project is bounded and the data understood is a diagram you'll redraw.

Figure 1. The Foundation Is Ready

In Lesson 2, we did the bounding. We read Bird's three use cases for data clues, named two entities (User and Message), defined five tables across two databases (ums and twitter), and established the relationships that tie them together. The schema is real now: a messages row has a specific shape - an id, a content field capped at 280 characters, an author_idpointing to a row in ums.users, and a created_at timestamp. The users row has a shape too. Every "save the message" step in every diagram we draw will refer to something concrete.

That concreteness is what changes everything. A diagram drawn on top of a real data model isn't a sketch - it's a specification. The boxes mean something. The arrows carry named fields. When a step says "validate the message," we already know what the validation is checking (which is called validation criteria): a non-empty content string, no longer than 280 characters, attached to an authenticated author_id. Nothing is left as a placeholder.

This lesson draws all three diagrams. We'll work in order - behaviour first, then the structure that behaviour implies, then the detailed conversation inside that structure - for the same reason we always work in that order: each diagram is the input to the next, and the input has to exist before the output can be correct.

A Quick Reminder: The Three-Lens Progression

Before we pick up the pen, one short bridge back to Lesson 1 - not a full re-teach, just enough to make sure the scaffold is standing.

The core idea is this: any software system can be understood from three distinct angles, and each angle demands a different kind of diagram.

Lens	The question it answers	The diagram	What you see
Behaviour	What should the app do?	Flowchart	Steps, decisions, and outcomes - the system as a process unfolding in time
Structure	What parts must exist to make that behaviour happen?	Functional diagram	Components and their connections - the system as a thing made of pieces
Interaction	How do those parts talk to each other at runtime?	Sequence diagram	Messages exchanged between components - the system as a conversation

None of these is more correct than the others. A flowchart can't tell you what components to build. A functional diagram can't tell you what happens when a user submits an empty message. A sequence diagram can't tell you whether you've correctly scoped the feature. They are complementary, not competing - and you need all three because no single one is complete.

The critical habit is the order. You move behaviour → structure → interaction, and each step is derived from the one before:

Figure 2. The three-lens progression - each diagram derives from the one before

The reason the order is fixed: you can't know what components to build until you know what the system must do. And you can't describe the conversation between components until you know which components exist. Behaviour is the requirement; structure serves it; interaction lives inside structure.

An analogy for the progression. Imagine designing a kitchen. First you decide what the kitchen must do- prepare meals, store food, clean dishes. That's behaviour. Then you ask what parts must exist to support each of those tasks: a cooktop, a refrigerator, a sink. That's structure, falling out of behaviour. Finally, you trace the specific workflow - the chef moves from the refrigerator to the prep surface to the cooktop, in that order, passing ingredients along. That's interaction, living inside the structure. You couldn't have designed the workflow without knowing the layout. You couldn't have designed the layout without knowing the tasks. The order is the method.

Lens 1: Behaviour - Drawing the Flowchart

A flowchart answers one question: what does the system do, step by step? Not what parts it has. Not which component calls which. Just the sequence of actions and decisions that carry a user from "I want to post a message" to "it's done" - or to an error, if something goes wrong along the way.

That makes the flowchart the right place to start. It captures the requirement in its purest form, before any implementation choices have been made.

Building it step by step

Step 1 - Name the start and end points.

The flow begins the moment the user presses "Post" - that is, when the system receives a submission. It ends either with a new row in twitter.messages and a confirmation to the user, or with an error message and no row written. Name those endpoints first, before filling in anything in between.

Step 2 - List the actions in order.

Walk the happy path first - what happens when everything goes right? For "Post a message," that sequence is:

Submit a message
Pass the authentication check - is the user logged in?
Pass the permissions check - is the user allowed to post?
Pass content validation - is the message valid?
Save the message to the database
Confirm success to the user

Six steps. Two are actions, three are decision gates, one is a confirmation. That's a healthy breakdown - the logic is explicit without becoming an implementation manual.

Step 3 - Add the decision points.

Three decision diamonds gate the happy path:

Is the user logged in? If no → redirect to the login page. After a successful login the user is returned to the submission step - a loop, not a dead end.
Is the user allowed to post? Not every account has posting rights - a read-only or suspended role is blocked here. If no → shared Error node → END.
Is the message valid? (Non-empty? Within 280 characters?) If invalid → shared Error node → END.

Notice that the two error branches converge on a single Error node before reaching END. This is an intentional design choice in the diagram: the system's response to a permissions failure and a content failure is the same kind of thing - an error returned to the user. Merging them keeps the diagram clean and makes that equivalence visible.

Step 4 - Connect and label.

Arrange the steps top to bottom, connect them with arrows, and label every arrow leaving a decision diamond. The labels on the branches ("yes / no", or more descriptively "valid / invalid") are what make the diagram readable at a glance.

The flowchart

Figure 3. Flowchart for "Post a message"

What this diagram tells us - and what it doesn't

The flowchart is intentionally silent on how each step is implemented. It doesn't say which component checks authentication, or what library validates the string length, or whether the database write is synchronous. Those details don't belong here. What the flowchart does - and does well - is make the logic visible: the decisions, the branches, the outcomes. That's the entire job of Lens 1.

Notice that the data model already sharpens the language. The "Save message" step doesn't just say "save the message"- it names the fields: author_id, content, created_at, writing into twitter.messages. That specificity comes directly from Lesson 2. The flowchart is still high-level, but it's no longer vague.

With the behaviour fully drawn, the next question asks itself: what parts must exist to make each of these steps happen? That's Lens 2 - and the answer comes directly from reading this flowchart.

Lens 2: Structure - Reading the Functional Diagram off the Flowchart

The flowchart is done. We know what "Post a message" must do - the steps, the decisions, the branches. Now a different question takes over: what parts must exist inside the system to make each of those steps happen?

This is where most beginners get into trouble. They open a new diagram and start inventing components - a generic AuthService, an APIGateway, a DatabaseLayer - based on instinct or prior experience. The result is a diagram that reflects what the developer already knew, not what the use case actually requires.

There is a better method. The functional diagram doesn't need to be invented. It can be read directly off the flowchart.

The method: scan every step, ask one question

For each step in the flowchart, ask: what component must own this responsibility? The answer names a block. Do that for every step, merge the ones that belong together, and the structure falls out of the behaviour.

Let's walk through the flowchart step by step.

Flowchart step	Responsibility	Component
User submits message	Sends the HTTP request	Frontend Client
Is the user logged in?	Identifies who is making the request	Twitter Service
Redirect to login	Returns the login page	Frontend Client
Is the user allowed to post?	Fetches the user record and checks for the `PRODUCER`role	Twitter Service → UMS Service
Is the message valid?	Enforces content rules (non-empty, ≤ 280 chars)	Twitter Service
Save message	Writes the row to `twitter.messages`	Twitter Service → MySQL Database
Confirm success	Returns `201` with the created message	Twitter Service → Frontend Client
Return error	Returns `403 Forbidden` to the client	Twitter Service → Frontend Client

Two things stand out immediately. First, the Twitter Service carries the most weight: it receives the request, performs content validation, and orchestrates the database write. But it doesn't hold user data - that lives in UMS. So for the permission check, the Twitter Service calls the UMS Service via an internal HTTP request, retrieves the user's roles, and decides whether to proceed. This cross-service call is the architectural heart of the feature.

Second, there is no separate API Gateway or Auth Service in Bird. The Frontend Client talks directly to the Twitter Service on port 9001. Role checking is a responsibility of the Twitter Service itself, delegated to UMS on demand - not a standalone component sitting in between.

This gives us four components: the Frontend Client, the Twitter Service, the UMS Service, and MySQL Database.

The functional diagram

Figure 4. Functional diagram for Bird

The diagram reads as a set of named blocks connected by labelled arrows. The arrows carry the nature of the relationship - not the step-by-step logic (that's the flowchart's job) but the structural dependency: the Frontend Client sends requests to the Twitter Service; the Twitter Service calls UMS to validate the user; the Twitter Service reads and writes MySQL directly. Each arrow represents a real, named communication channel in the codebase - HTTP on :9001 for the UI-to-service call, WebClient for the service-to-service call, JDBC for the database connection.

What this diagram is - and isn't

A functional diagram is a map of components, not a map of steps. It answers "what pieces exist and how are they connected?" - not "what happens first?" The flowchart and the functional diagram are not redundant; they answer different questions and each is incomplete without the other.

Notice also what the functional diagram deliberately omits: there are no arrows labelled with field names, no decision branches, no error paths, no sequence numbers. Those details live in the sequence diagram. Here, the goal is structural clarity - a reader should be able to see at a glance what Bird is made of and how the pieces relate.

With the components named and connected, one question remains: how exactly do they talk to each other during "Post a message"? That's the sequence diagram - and now that we know which components exist, we can finally draw it precisely.

Lens 3: Interaction - The Sequence Diagram

The flowchart told us what "Post a message" does. The functional diagram told us what components exist to do it. The sequence diagram answers the last question: what do those components say to each other, in what order, when the feature actually runs?

This is where the previous two diagrams earn their investment. Every component named in the functional diagram becomes a lifeline. Every step in the flowchart becomes a message on one of those lifelines. Nothing is invented - it's assembled from what we already know.

From components to lifelines

A sequence diagram is built around lifelines - vertical columns, one per component, representing each participant in the interaction. Reading the functional diagram directly, our lifelines are:

Frontend Client - the browser, initiating the request
MessageController - the entry point inside the Twitter Service, receiving the HTTP call
MessagesService - the orchestrator: calls UMS, checks roles, triggers the database write
UMSConnector - the thin HTTP client that talks to UMS via WebClient
UMS Service - the external service that owns user identity and roles
JdbcMessageRepository - the DAO that executes the SQL insert
MySQL - the database that persists the row

Reading the flow

The messages between lifelines follow the exact path the codebase executes. Let's trace it:

The Frontend Client sends POST /messages/message with { author, content } to the MessageController.
MessageController deserializes the body into a Message DTO and delegates to MessagesService via createMessage(message).
MessagesService needs to verify the author - it doesn't hold user data, so it calls UMSConnector with retrieveUmsData(/users/user/{id}).
UMSConnector fires a non-blocking WebClient GET to the UMS Service.
UMS Service returns a UserDto containing the user's roles.
MessagesService unpacks the response via HttpResponseExtractor and inspects the Roles. If the user is not a PRODUCER - the flow terminates here with a 403 Forbidden back to the client.
If the role check passes, MessagesService calls createMessage(message) on JdbcMessageRepository.
JdbcMessageRepository executes INSERT INTO messages against MySQL (converting the UUID via DaoHelper).
MySQL returns ok.
The saved Message travels back up: JdbcMessageRepository → MessagesService → MessageController → Frontend Client as 201 { message }.

The sequence diagram

Figure 5. Sequence diagram for "Post a message"

What this diagram shows that the others couldn't

Look at the alt block in the middle of the diagram. That's the permission check - the same diamond in the flowchart, the same role-checking responsibility assigned to MessagesService in the functional diagram. But now it has an exact form: a comparison against the PRODUCER role, returning 403 Forbidden if it fails, with the response passing back through MessageController to the client.

This is traceability working in practice. The validation step in the flowchart → the MessagesService block in the functional diagram → the HttpResponseExtractor → User → Roles call chain in the sequence diagram. One idea, three levels of resolution.

Notice also what the sequence diagram adds that neither previous diagram could: the reactive chain. The UMSConnectorcall is non-blocking - MessagesService issues the WebClient request and the rest of the chain executes inside a .flatMap(). The sequence diagram makes this visible through the activation boxes: MS stays active while UC and UMS do their work, then resumes. That concurrency detail is invisible in both the flowchart and the functional diagram.

With all three diagrams drawn, one use case - "Post a message" - now has a complete, layered blueprint. The next section puts all three side by side and shows what traceability looks like end to end.

Let's pause before moving on and do something the three diagrams individually don't do: put them side by side and trace a single idea across all of them.

Take the permission check - the moment the system decides whether the author is allowed to post. Here is where it lives in each diagram:

Diagram	Where the permission check appears	What you can see
Flowchart	The diamond "Is user allowed to post?" branching to `Error`or continuing to validation	The logic - two outcomes, one decision
Functional diagram	The arrow from `MessagesService` labelled "check roles contains PRODUCER" pointing to `Roles`	The ownership - which component holds this responsibility
Sequence diagram	The `alt` block: `HttpResponseExtractor → User → Roles`, branching to `403 Forbidden` or `INSERT`	The mechanics - what data moves, in what order, producing what result

Same check. Three levels of resolution. You could point at the diamond in the flowchart, follow it to the MessagesServicearrow in the functional diagram, and land on the alt block in the sequence diagram - and at each step you'd learn something the previous diagram couldn't tell you.

That property is traceability: the ability to follow a single requirement from its highest-level expression down to its lowest-level implementation without losing the thread. It's what separates a set of diagrams that decorate a document from a set that actually functions as a blueprint.

Figure 6. One feature, three views - the permission check traced across all three diagrams

Notice also what traceability reveals in the other direction. The sequence diagram shows that the 403 Forbidden response travels back through MessageController before reaching the client - a detail invisible in both the flowchart (which just says "Error → END") and the functional diagram (which shows MessageController connected to GlobalExceptionHandler without explaining when). The diagrams don't just repeat each other. Each one completes the picture in a direction the others leave open.

Limitations and Open Questions

Three diagrams drawn, one use case traced. Before closing, it's worth being clear about what this process is - and what it isn't.

Diagrams go stale. Every diagram in this lesson reflects the codebase as it stands today. The moment a new field is added to messages, or the permission model changes, or UMSConnector is replaced by a shared auth library - the diagrams need updating. A diagram that isn't maintained becomes misleading faster than no diagram at all. The discipline of drawing diagrams is only half the work; the other half is keeping them honest.

This is a snapshot, not a specification. The diagrams here describe what Bird does, not what it must do. A specification would go further - stating invariants, error contracts, performance bounds, and edge cases. These diagrams are the foundation of a spec, not the spec itself. They answer "how does it work?" not "how must it always work?"

Over-modelling is a real risk. Drawing all three diagrams for every feature in a large system would consume more time than it saves. The three-lens approach is most valuable at decision points: when designing a new feature from scratch, when onboarding someone to an unfamiliar part of the system, or when debugging behaviour that contradicts the mental model. Applied indiscriminately, it becomes ceremony rather than engineering.

These diagrams are not UML. Readers familiar with the Unified Modelling Language (UML) will notice that the diagrams in this lesson borrow UML conventions - sequence diagram lifelines, flowchart decision diamonds - without following the full UML standard. That's deliberate. UML is precise and complete, and its precision comes at the cost of accessibility. However when you move to production systems with formal specification requirements, UML or C4 may be the right choice.

What's Next

Three lessons in, and Bird has a bounded scope, a real data model, and a complete three-lens blueprint for its core feature.

What it doesn't have yet is a working API. The diagrams show that a POST /messages/message endpoint should exist, that it must accept { author, content }, that it must call UMS before writing to the database - but none of that is running code yet.

Lesson 4 is where the diagrams meet the wire. We'll define the API contracts and endpoints that connect Bird's components - what each service exposes, what it expects, and what it returns. The sequence diagram already told us that POST /messages/message must exist, that it calls GET /users/user/{id} on UMS, and that both exchanges carry specific payloads. Lesson 4 makes those contracts explicit and formal: request shapes, response shapes, status codes, and the communication protocol between the Twitter Service and UMS.

The diagrams were never the destination. They were the clearest possible way to know what to build before building it.

The Blueprint Beneath the Blueprint: Designing Data Model and Choosing Its Database

Eugene Zimin — Sat, 23 May 2026 06:36:22 +0000

Lesson 2 of Build a Twitter Clone - A Practical Guide to Software Modelling

A diagram shows you what a system does; a data model tells you what it remembers. Before drawing a single flowchart, you need to know what information Bird must store - and how that information is shaped. In this lesson we read our three use cases for data clues, name the entities the system must track, define their fields and relationships, and translate all of it into a real MySQL schema split across two purposefully separated databases.

Introduction - Data Before Diagrams

Lesson 1 was closed with a promise: in Lesson 2, we draw the diagrams. Flowchart, functional diagram, sequence diagram - the works. At the moment we're going to defer it for a while. The reason is quite simple - because of something that becomes obvious the moment you try to draw the flowchart for "Post a message" without it: a diagram that doesn't know what data it's moving is vague in exactly the wrong places.

You need to consider the following - the flowchart for posting a message will eventually have a step called something like "save the message." Straightforward enough on paper. But the moment you try to build it, questions pile up fast:

Save what, exactly? The text? The author? A timestamp? All three?
Where does the author's identity come from - a name typed into a field, or a reference to a stored account?
What does "the author" even mean to the system - a row in a table somewhere, or just a string?
If the same author posts twice, how does the system know both messages belong to the same person?

Figure 1. What is the Message?

A diagram that leaves those questions open isn't a blueprint. It's a sketch - useful for thinking, but not yet useful for building. The answers live in the data model: the formal description of what the system stores, and how the pieces of stored information relate to one another.

Think of the data model as the system's long-term memory. The diagrams describe what the system does- its behaviour, its structure, its conversations. The data model describes what it remembers between those moments of action. Without memory, each request starts from nothing. With it, a message posted today is still there tomorrow, and the author who posted it can be found.

Getting the data model right before drawing the diagrams isn't pedantry. It's the thing that turns vague boxes into precise components. When you know that a message is a row with an id, a content field capped at 280 characters, and an author_id that points to a specific user - then the "save message" step in your flowchart stops being a placeholder and starts being an instruction. The functional block that does the saving has a clear contract. The sequence diagram can name what passes between components.

One System, Two Databases - and Why That's the Right Call

Before we write a single table definition, there's a structural question to answer: should all of Bird's data live in one database, or more than one?

The instinct is usually to start with one. It's simpler, it requires less setup, and for a small project it feels like the obvious default. We're going to make a different call - and the reasoning behind it is worth understanding clearly, because the same question will come up in every non-trivial system you ever build.

The problem with putting everything in one place

Start with the simpler option: one database called bird, with all tables inside it - users, messages, sessions, roles, everything. Will it work?

The answer is clear - it would work. In fact, it's how most beginner projects start, and there's nothing wrong with it as a starting point. But consider what happens as the system grows:

A security audit requires changes to how passwords are stored. To make that change safely, you need to understand every table that touches user data - except now it's tangled up with message tables, timeline queries, and subscription records. The scope of "change one thing" has quietly expanded.
Message volume spikes. You want to move the messages table to faster storage, or archive old records. But the table is in the same database as your user credentials, which have entirely different performance and retention requirements. You can't move one without the other.
A colleague is working on authentication while you're working on the timeline feature. Both of you are making schema changes in the same database, to tables that are conceptually unrelated. You're in each other's way.

Figure 2. All in one Database - Will it Work?

The deeper problem is this: a single database couples concerns that have no real reason to be coupled. They are in the same place not because they belong together, but because it was convenient to put them there. Convenience now, complexity later.

The principle: each domain owns its data

The solution is to give each distinct area of the problem its own data store - its own database that it, and only it, is responsible for. Nothing else writes directly to that data; anything that needs information from another domain has to go through the interface that domain exposes.

This is a well-established pattern in software design called Database per Service: each logical service or domain owns its storage, and the boundary between services is enforced at the data layer, not just at the code layer. The pattern makes the separation real and durable - you can't accidentally reach across a boundary you'd have to explicitly cross.

![[Database per Service Design Pattern.jpg]]Figure 3. Database per Service Design Pattern

For Bird, two domains emerge clearly once you ask the question:

Identity and access - who people are, how they prove it, what permissions they hold, which sessions are currently active. This is security-critical data, relatively stable, and completely self-contained. It has nothing to do with messages.
Content and social graph - the messages people post, and the follow relationships between them. This data is high-volume and fast-moving, with entirely different performance and retention characteristics.

These two concerns have different owners, different change rates, and different security requirements. They will be given separate databases: ums (User Management System) for identity and access, and twitter for content and social graph.

Database	Domain	What it owns
`ums`	Identity & access	Who people are, how they authenticate, what permissions they hold, which sessions are active
`twitter`	Content & social graph	The messages people post, and who follows whom

Separating them means each can be optimized, scaled, or secured independently - a schema change in one won't touch the other.

The idea behind the split: bounded contexts

The deeper principle at work here comes from Domain-Driven Design (DDD) - a way of thinking about how to organize software around the real-world problems it solves, rather than around technical convenience.

DDD would describe ums and twitter as separate bounded contexts: distinct areas of the problem, each with its own vocabulary, rules, and data. The word user in the identity context means something specific - an account with credentials, roles, and a session history. The word author in the messaging context means something different - a source of content, identified by an ID, whose full identity details live elsewhere. These concepts correspond to the same human being, but they are different models of that person, serving different purposes. Keeping them in separate databases makes that distinction visible and enforces it.

Figure 4. How DDD Helps and Works

An analogy. Think of a hospital. The billing department and the medical records department both deal with the same patients - but they maintain completely separate files. The billing system doesn't need to know a patient's diagnosis; the medical records system doesn't need to know their payment history. Each department owns its data. Information is shared only when explicitly requested, through defined channels. The separation isn't bureaucracy - it's what keeps sensitive data appropriately contained, and what lets each department evolve independently.

If you want to go deeper on Domain-Driven Design, this article covers the core ideas without requiring a computer science background.

Reading the Use Cases for Data Clues

A use case describes what a user accomplishes. But read carefully, and it also tells you what the system must remember in order to make that possible. Each of Bird's three use cases leaves a trail of data requirements - we just have to follow it.

Post a message

A user writes some text and publishes it. For this to work, the system must store the text itself, know who posted it, and record when it was posted so the timeline can be ordered. That's three pieces of information: content, author, timestamp.

Scope	Field	Type	Why it's needed
`messages`	`id`	UUID	A stable, unique identifier for this message
`messages`	`author_id`	UUID	The `id` of the user who posted this message - links back to `users.id`
`messages`	`content`	String (max 280 chars)	The text of the message, capped at 280 characters
`messages`	`created_at`	Timestamp	When the message was posted - used to order the timeline

View the timeline

This use case produces no new fields. It reads existing messages rows, ordered by created_at descending, and joins to users on author_id to display the author's name. Every field it depends on was already required by Post a message. Here we have to introduce what it is called - subscriptions and set a relation between author and and consumer of the message, i.e. - subscriber.

Scope	Field	Type	Why it's needed
`subscriptions`	`subscriber_id`	UUID	The user doing the following - logical FK to `users.id`
`subscriptions`	`producer_id`	UUID	The user being followed - logical FK to `users.id`
`subscriptions`	`created_at`	Timestamp	When the follow relationship was created

Register an account

A user creates an identity. The system must store a name to display, credentials to authenticate with (stored as a hashed password, never plain text), and again a timestamp. It also needs a way to distinguish one account from every other - a unique identifier that never changes even if the username does.

Scope	Field	Type	Why it's needed
`users`	`id`	UUID	A stable, unique identifier for this user - never changes, even if name or email does
`users`	`name`	String	The display name shown alongside every post
`users`	`email`	String	Login credential; unique across all users - no two accounts can share an address
`users`	`password_hash`	String	The user's password after a one-way hashing function - never the plain-text password
`users`	`created_at`	Timestamp	When the account was registered
`users`	`updated_at`	Timestamp	When any field on this record last changed; refreshed automatically on every write

Scope	Field	Type	Why it's needed
`roles`	`id`	UUID	Unique identifier for this role
`roles`	`name`	String	The role label (e.g. `admin`, `member`) - must be unique
`roles`	`description`	String	Optional human-readable explanation of what this role permits

Scope	Field	Type	Why it's needed
`sessions`	`id`	UUID	Unique identifier for this login session
`sessions`	`user_id`	UUID	References `users.id` - whose session this is
`sessions`	`logged_in_at`	Timestamp	When the session started
`sessions`	`logged_out_at`	Timestamp	When the session ended; `NULL` if the user is still logged in

Across all three use cases, two distinct categories of information emerge: things the system needs to know about people, and things it needs to know about messages. That observation is the foundation of the data model.

Naming the Entities

An entity is a category of information the system tracks as a distinct thing - something that has its own identity, its own set of properties, and its own lifetime. Naming entities is the first act of data modelling: you're deciding what the system considers a noun.

From the use cases, two entities name themselves:

User - a registered account. Every message is posted by a user; the timeline attributes each post to one. Users exist independently of any message they've posted, and they persist even if all their messages were deleted. They are a distinct thing in their own right.

Message - a piece of content posted by a user. Messages depend on users (a message with no author makes no sense), but they are not part of a user - they are their own thing, with their own timestamp and their own text.

Two entities. That matches the two databases we decided to create: ums owns User, and twitter owns Message. The database boundary and the entity boundary are the same line drawn twice.

You'll notice there's no Timeline entity, no Feed entity, no Notification. The timeline is not a thing the system stores - it's a query - give me all messages, ordered by created_at, descending. It exists at runtime, not at rest. This is a useful distinction to internalise: not everything the user sees needs to be stored.

In the next section we'll define the fields each entity carries - and introduce the mechanism that connects a Messageback to the User who wrote it.

Defining the Fields

An entity is just a name until you give it fields - the individual pieces of data it holds. This is where the model gets concrete. For each field, we'll state what it is, what type of value it holds, and why it exists, tracing every decision back to a use case or a design constraint.

A note on identifiers

Every entity needs a primary key - a field whose sole job is to uniquely identify one row among all others, forever. A common choice is an auto-incrementing integer (1, 2, 3...), but Bird uses something different: a UUID (Universally Unique Identifier), stored as 16 bytes (or 128 bits) of binary data.

A UUID looks like 550e8400-e29b-41d4-a716-446655440000 - a 128-bit value generated in a way that makes collisions statistically impossible, even across separate systems. The binary storage (BINARY(16)) keeps it compact and fast to index. The reason to prefer UUIDs over integers here is forward-looking: if Bird ever scales to multiple servers generating records simultaneously, each can produce its own IDs without coordinating with the others. Integers can't do that safely.

Every table uses this same pattern for its primary key: a BINARY(16) column called id, defaulting to a freshly generated UUID.

`users` - the identity record

The users table lives in the ums database. It is the authoritative record of everyone who has registered an account.

Field	Type	Purpose
`id`	`BINARY(16)`	Primary key - uniquely identifies this user across the entire system
`name`	`VARCHAR(100)`	The display name shown on posts
`email`	`VARCHAR(255)`	Login credential and contact address; must be unique across all users
`password_hash`	`VARCHAR(255)`	The result of running the user's password through a one-way hashing function - never the password itself
`created_at`	`DATETIME`	When the account was registered
`updated_at`	`DATETIME`	When any field on this record was last changed; updated automatically

Two fields deserve a word of explanation. password_hash stores a hashed password, not a plain-text one. A hash functionis a one-way transformation: you can turn a password into a hash, but you cannot reverse the process. When a user logs in, the system hashes what they typed and compares the result to the stored hash - the original password never needs to be stored or retrieved. This is standard practice; storing plain passwords is a serious security failure.

updated_at tracks the last modification time and refreshes itself automatically on every write. It's low-cost to store and invaluable for debugging, auditing, and cache invalidation later.

Supporting the `User`: roles and sessions

The use cases named registration - but a real identity system has two more concerns lurking just beneath the surface: what is this user allowed to do, and is this user currently logged in? These concerns get their own tables.

roles is a lookup table - a simple list of named permission levels (for example, admin, moderator, member). Roles don't come from the use cases directly; they come from the reality that not all users have the same permissions.

Field	Type	Purpose
`id`	`BINARY(16)`	Primary key — uniquely identifies this role
`name`	`VARCHAR(50)`	The role label (e.g. `admin`, `member`) — must be unique
`description`	`VARCHAR(255)`	Optional human-readable explanation of what this role permits

users_roles links users to roles. Because one user can hold multiple roles and one role can be held by many users, this is a many-to-many relationship - and the standard way to model that in a relational database is a join table: a table with two columns, each a reference to one side of the relationship.

sessions records active login sessions. When a user authenticates, a session row is created with a logged_in_at timestamp. When they log out, logged_out_at is filled in. This gives the system a full audit trail of who was logged in and when - and lets it invalidate specific sessions without forcing a global logout.

Field	Type	Purpose
`id`	`BINARY(16)`	Primary key — uniquely identifies this session
`user_id`	`BINARY(16)`	References `ums.users.id` — whose session this is
`logged_in_at`	`DATETIME`	When the session started
`logged_out_at`	`DATETIME`	When the session ended; `NULL` if the user is still logged in

None of these tables store messages or content. They belong entirely to the ums database and the identity domain.

`messages` - the content record

The messages table lives in the twitter database. It is the record of everything posted.

Field	Type	Purpose
`id`	`BINARY(16)`	Primary key - uniquely identifies this message
`author_id`	`BINARY(16)`	The `id` of the user who posted this message, from `ums.users`
`content`	`VARCHAR(280)`	The text of the message - capped at 280 characters
`created_at`	`DATETIME`	When the message was posted

content is capped at 280 characters - the same limit Twitter uses, and a deliberate product constraint, not a technical one. The database enforces it with VARCHAR(280).

author_id is the field that links a message to its author. It holds the id value of a row in ums.users. Because the two tables live in separate databases, MySQL cannot enforce this link with a formal foreign key constraint - but the relationship is real. The application layer is responsible for ensuring that no message is ever written with an author_id that doesn't correspond to a real user.

`subscriptions` - the social graph

There is one more table in the twitter database: subscriptions. It didn't appear in the original three use cases, but it's present in the schema for a good reason - it's the data that would power a personalised timeline.

A subscription is a directional relationship between two users: one subscriber who follows one producer. The table has just three fields:

Field	Type	Purpose
`subscriber_id`	`BINARY(16)`	The user doing the following
`producer_id`	`BINARY(16)`	The user being followed
`created_at`	`DATETIME`	When the follow happened

The combination of subscriber_id and producer_id is the primary key - you can only follow someone once. Both columns are logical foreign keys to ums.users.id, subject to the same cross-database constraint limitation as author_id in messages.

subscriptions is infrastructure for a feature - "Follow a user" - that isn't in scope for the current lesson's use cases but is correct to model now, because adding it later would require a migration. Modelling it upfront costs nothing; omitting it and adding it later costs a schema change and a deployment. This is the kind of forward-thinking that separates a considered data model from a reactive one.

How the Entities Relate

Individual entities are only half the picture. A data model also defines how entities connect to one another - their relationships. For Bird, there are two:

A User has many Messages (one-to-many). One user can post any number of messages; each message belongs to exactly one user. This relationship is expressed through author_id in the messages table - a field that holds the id of the user who owns that row. Following the author_id from a message leads you to its author.

A User can follow many Users, and be followed by many Users (many-to-many). This is the social graph, modelled through the subscriptions table. To find everyone a user follows, query subscriptions where subscriber_id matches. To find everyone following a user, query where producer_id matches.

Figure 5. Types of Relationships

The pattern to remember. A one-to-many relationship is expressed by putting the "one" side's id as a field on the "many" side - the foreign key lives in the child table. A many-to-many relationship requires its own table, with one column for each side. Both patterns appear in Bird, and together they cover the vast majority of real-world data relationships you'll encounter.

With the entities named, the fields defined, and the relationships mapped, the data model is complete. What remains is to choose a database engine and write the schema.

What the Diagrams Will Now Be Built On

Lesson 1 introduced three lenses for looking at a system: functional structure, behaviour, and component interaction. Lesson 1 also made a promise — that diagrams would follow.

They will. But notice what's changed since then.

Before the data model existed, a flowchart for "Post a message" would have had a step labelled something like "save the message" — a placeholder that gestures at an action without defining it. Now that step has a precise meaning: create a row in twitter.messages with a content value, an author_id pointing to the authenticated user in ums.users, and a created_attimestamp set to now.

The data model doesn't just inform the diagrams — it creates clear mechanics how they work. Every box that reads or writes data now has a contract: specific fields, specific tables, specific relationships. The sequence diagram will name what passes between components. The functional diagram will know what each block owns. The flowchart steps will correspond to real operations.

That's where Lesson 3 picks up: with the data model as the foundation, we draw all three diagrams.

Conclusion

A data model is the contract the rest of the system is written against.

We started with three use cases and asked a simple question: what must the system remember for each of these to work? That question led us to two entities — User and Message — and from there to five tables across two purposefully separated databases.

The separation itself was a design decision, not a convenience. Identity and access belong to one domain; content and social graph belong to another. Keeping them apart makes each easier to change, scale, and reason about independently. The trade-off — enforcing cross-database relationships in application code rather than at the database level — is a known and manageable cost.

Every field in the schema exists for a reason traceable back to a use case. Every relationship reflects a real dependency between things the system tracks. Nothing was added for completeness or anticipation — except subscriptions, which earns its place by being cheaper to model now than to migrate in later.

The diagrams come next. They'll have something real to say.

From Idea to Blueprint: Turning a Vague App Concept into Something You Can Actually Build

Eugene Zimin — Sun, 17 May 2026 22:03:47 +0000

Lesson 1 of Build a Twitter Clone — A Practical Guide to Software Modelling

"Build a Twitter clone" is a wish, not a plan — you can't write code from a single sentence. This lesson shows you how to think about turning a fuzzy app idea into a concrete blueprint: we scope a deliberately small messaging app called Bird, then introduce the three-lens model — what should it do?, what parts must exist?, and how do those parts talk to each other? — that the rest of the series uses to actually draw it. By the end you'll have a bounded project and a clear mental model, the foundation Lesson 2 builds on.

Introduction — From Idea to Blueprint

Imagine someone hands you a sticky note that says "build a Twitter clone" and asks you to start coding.

Where, exactly, would you put your cursor?

That question is harder than it looks, and the difficulty isn't about programming skill. It's that the instruction isn't actually buildable. "Build a Twitter clone" is a wish. It tells you the destination but none of the road. Before a single line of code makes sense, that wish has to be transformed into something with edges, parts, and order — a blueprint.

Figure 1. From Idea to Blueprint

This lesson is about making that transformation, and doing it with a tool most people underrate: the humble diagram.

Why an idea isn't a plan

A sentence like "build a Twitter clone" hides an enormous number of decisions. Consider just a few:

Can users edit a post after publishing it, or is it permanent?
What happens when someone submits an empty message? An error? Silence?
Does a post appear instantly, or is there a moment of "sending..."?
Where does the message go once it's submitted — and who is responsible for keeping it?

None of these answers live in the original sentence. They have to be decided, and a working app is really just the sum of hundreds of such decisions made consistent with one another. The gap between "an idea" and "a plan" is precisely the gap of unmade decisions — and code is a terrible place to discover you forgot one.

Figure 2. Why Idea is not Enough

A blueprint isn't bureaucracy. It's the cheapest place to be wrong. Moving a wall on paper costs a pencil eraser; moving it after the concrete is poured costs a demolition crew.

The same logic applies to software. A confused diagram can be fixed in thirty seconds. The same confusion, discovered three weeks into coding, can cost days of rework. Thinking before building isn't slower — it's the thing that makes building fast.

Diagrams: thinking you can point at

When we say diagram, we don't mean decorative boxes-and-arrows that get drawn after the work is done and quietly go stale. We mean something more useful: a diagram is a way of thinking that you can point at.

Plain prose is bad at certain kinds of thought. A paragraph describing how six components pass data around is exhausting to read and easy to get wrong, because language forces ideas into a single line, one word after another. But a system isn't a line — it has shape. It branches. It loops. It has parts that exist alongside each other. A diagram lets that shape sit still on the page so you can inspect it, find the gap, and point a teammate straight at the problem.

That's the real job of the diagrams in this series. They are not documentation. They are the instrument you use to turn the wish into the plan — and, just as importantly, a shared picture a whole team can argue over and agree on.

What this series is — and what this lesson does

This is the first lesson in a hands-on series with one practical goal: build a working Twitter-style application, from blank page to running software.

We won't begin with code, a database, or a framework. We'll begin where every real project begins — with the awkward question "what should this thing even do?" — and we'll answer it the way experienced engineers do: by modelling the system before constructing it.

Across this lesson you'll meet three kinds of diagrams, and the order matters, because each one answers a question the next one depends on:

Behaviour — What should the app do? We'll capture this with a flowchart: the step-by-step flow of a single user action, including its decision points and outcomes.
Structure — What parts must exist to make that behaviour possible? We'll capture this with a functional diagram (also called a block diagram): the system broken into its major components.
Interaction — How do those parts actually talk to each other at runtime? We'll capture this with a sequence diagram: the precise back-and-forth of messages between components.

Figure 3. The Cube

Behaviour first, then the structure that delivers it, then the detailed conversation inside that structure. Each diagram is derived from the one before — you'll watch structure literally fall out of behaviour rather than being invented from thin air.

Don't worry if those three terms mean nothing to you yet. Each gets a plain-language introduction, a worked example, and a step-by-step recipe in its own section. Right now, the only thing to hold onto is the progression: what it does → what it's made of → how the parts interact.

A deliberately small project

There's one honest catch we should name immediately. The real Twitter is gigantic — feeds, ranking algorithms, notifications, direct messages, media uploads, advertising, moderation at planetary scale. We are not building that. Trying to would teach you nothing except how to feel overwhelmed.

Instead, we'll build the smallest thing that is still recognisably a short-messaging app — a deliberately tiny slice we'll call Bird. Choosing that slice on purpose is not a shortcut around the engineering. It is the engineering. Knowing what to leave out — scoping a minimum viable product, or MVP, the leanest version that still does the core job — is one of the most valuable skills in software, and it's the first thing the next section will practise.

So let's pick up the sticky note that says "build a Twitter clone" — and turn it into something you can actually build.

Meet the Project: `Bird`

Before we can draw a single diagram, we need something to draw about. So let's give the vague sticky note a concrete shape.

We're building Bird: a small short-messaging app where people post brief public updates and read other people's. That's the whole pitch. If Twitter is a city, Bird is a single well-built room in it.

The hardest decision is what to leave out

It's tempting to start listing everything Bird could do — likes, replies, hashtags, profile photos, search, notifications, a ranked feed. Resist that. A feature list that grows without limit is the single most common reason beginner projects never ship.

The discipline here has a name: scoping a minimum viable product, or MVP — the leanest version of an app that still does its core job and nothing more. The goal isn't a crippled product. It's a complete one with a deliberately narrow definition of "complete."

A useful test for the MVP line: for each candidate feature, ask "If this were missing, would the app still be recognisably a short-messaging app?" If the answer is yes, the feature is out of Lesson 1 — not deleted forever, just postponed.

Run that test and most of the obvious features fall away. Likes? The app still works without them. Hashtags? Still works. A ranked feed? Still works — you'd just see posts newest-first. But remove the ability to write a post, or to see_posts, and there's no app left at all. Those survive the test. Almost nothing else does.

This gives us our scope. We'll describe it as a short list of use cases — a use case is simply one complete thing a user can accomplish with the app, named from the user's point of view. Use cases are the raw material everything else in this lesson is derived from, so naming them precisely matters.

Figure 4. Defining the MVP - Minimal Viable Product

For now, Bird has exactly three use cases:

#	Use case	What the user accomplishes
1	Post a message	Write a short message (a "bird") and publish it so others can see it.
2	View the timeline	See a list of recently posted messages, newest first.
3	Register an account	Create an identity so posts can be attributed to a person.

Three use cases. That's the entire behavioural scope of the app — and it's meant to feel small.

One use case becomes our worked example

Three use cases is a small scope, but it's still more than we need to teach diagramming. Drawing all three through all three diagram types would be repetitive — you'd learn the technique once and then watch it twice more.

So for the rest of this lesson, we'll follow a single use case end to end: "Post a message."

We chose it deliberately. Of the three, it's the one with real internal richness — it involves the user typing, the app checking the input, something deciding whether to accept or reject it, and the message being stored somewhere. That makes it the ideal teaching subject:

it has a clear start and end, so it suits a flowchart;
it requires several distinct components, so it suits a functional diagram;
it involves a genuine back-and-forth between those components, so it suits a sequence diagram.

"View the timeline" and "Register an account" are real and necessary, but we'll leave them as exercises. Once you've watched "Post a message" travel through all three diagram types, you'll have the pattern — and applying it to the other two becomes straightforward practice rather than new material.

Eventually, we've turned a vague wish ("build a Twitter clone") into a bounded one: an app called Bird, three named use cases, an explicit out-of-scope list, and one chosen worked example. That bounded definition is the first real artifact of the series. The next section introduces the three lenses we'll view it through — and explains why their order is not arbitrary.

Three Lenses, One Progression

We now have a bounded project: the Bird app, three use cases, and one worked example — "Post a message." The next move is to look at that example through three different lenses, each a kind of diagram.

The word lens is the right one. A lens doesn't change the thing you're looking at — it changes what you can see clearly. Point a microscope and a telescope at the same night sky and you learn completely different truths. Our three diagrams work the same way: one system, three views, three distinct questions answered.

The three questions

Every software system can be interrogated from three angles. Each diagram type is just a tool purpose-built to answer one of them.

Lens	The question it answers	The diagram	What you see
Behaviour	What should the app do?	Flowchart	Steps, decisions, and outcomes — the system as a _process_unfolding in time.
Structure	What parts must exist?	Functional diagram	Components and their connections — the system as a thing made of pieces.
Interaction	How do the parts talk?	Sequence diagram	Messages exchanged between components — the system as a conversation.

Notice that none of these is "more correct" than the others. A flowchart can't tell you what components to build; a functional diagram can't tell you what happens when input is invalid. They are not competing descriptions — they are complementary ones. You need all three because no single one is complete.

![[Behaviour, Structure, Interaction - 3 Diagram Types for Multi-Dimensional Software Modelling.png]]
Figure 4. Behaviour, Structure, Interaction - 3 Diagram Types for Multi-Dimensional Software Modelling

An analogy. Think of how you'd describe a car. Behaviour: "you press the pedal, fuel ignites, the wheels turn." Structure: "it has an engine, a transmission, four wheels, a fuel tank." Interaction: "the engine sends torque to the transmission, which sends it to the axle." Each description is true. Each is useless on its own. Together, they let you actually understand — or build — the car.

Why the order is not arbitrary

Here is the part that matters most, and the habit this whole series is built on: the three lenses form a progression, and the progression has a direction.

Figure 5. The order is important

You move behaviour → structure → interaction, and each step is derived from the one before it. This is the single most important idea in the lesson, so it's worth saying slowly:

Start with behaviour. Before you can know what parts a system needs, you must know what it has to do. Behaviour is the requirement; everything else serves it. So we draw the flowchart first.
Let structure fall out of behaviour. Once the flowchart exists, the components almost name themselves. The flowchart has a step "check the message is valid" — so there must be a part that does validation. It has a step "save the message" — so there must be a part that handles storage. You don't invent the structure; you read it offthe behaviour. Each thing the app must do implies a part that does it.
Detail the interaction inside the structure. Now that you know the parts, you can zoom into one slice of the action and show the precise back-and-forth between them — which component sends what message to which other component, and in what order. The sequence diagram lives inside the structure the previous step produced.

Each diagram is the input to the next. That's why the order can't be shuffled. Trying to design components before knowing the required behaviour means guessing — and guessed structure is the kind of mistake that's expensive to discover later. Behaviour-first isn't a stylistic preference; it's how you keep every later decision grounded in something real.

The thread you'll be able to trace

Because each diagram derives from the last, something valuable happens: you can follow a single feature all the way through.

The use case "Post a message" will appear three times in this lesson — once as a flow of steps, once as a set of components, once as a conversation between them. And crucially, you'll be able to point at any element in any diagram and trace it back. "This validation step in the flowchart" becomes "this validator block in the functional diagram"_becomes "this validate() message in the sequence diagram."_ Same idea, three depths of focus.

That property has a name — traceability — and it is the quiet engine of good engineering. When you can trace a feature from "what the user wanted" down to "the specific parts that deliver it," nothing falls through the cracks. When you can't, things do.

The takeaway. Three diagrams, one progression: what it does → what it's made of → how the parts interact. Each is a different lens on the same Bird app, and each is derived from the one before. Hold onto that shape — the rest of the lesson simply walks it, one lens at a time.

Closing this lesson's opening half

That completes the framing half of Lesson 1. You now have everything you need before the hands-on diagramming begins:

a vague wish turned into a bounded project — the Bird app, three use cases, one worked example;
a clear understanding of why blueprints matter — they are the cheapest place to be wrong;
the three-lens mental model — behaviour, structure, interaction — and the reason its order is fixed.

What comes next. That brings Lesson 1 to a close. You haven't drawn anything yet — and that's exactly right. A diagram drawn before the project is bounded and the lenses are understood is a diagram you'll redraw. Lesson 2 is where the building starts: first we define the data model — the precise structure of the information Bird must store and pass between its components — and then we return to the three lenses, drawing the flowchart, functional diagram, and sequence diagram for "Post a message" on top of that data model. The framing you've just finished is what makes those diagrams quick to draw and hard to get wrong.

API Authentication: Part III. JWT Tokens

Eugene Zimin — Sun, 17 May 2026 02:55:46 +0000

Why API Keys Aren't Always Enough

In Part II we saw that an API key is essentially a long, secret password your software shows to a server. It works, but it has a hidden cost: every time the key is used, the server must look it up in a database to find out what the key is allowed to do, whether it has expired, and whether it has been switched off. A JSON Web Token (JWT) removes that lookup by carrying all of that information inside the token itself. This article explains the problem JWT solves and shows where it sits in the larger story of web authentication.

Part I covered Basic Authentication — sending a username and password with every request. Part II covered API keys — replacing that reusable password with a single opaque secret string that identifies an application rather than a person.

Both approaches share a quiet assumption: the server already knows things about the credential, and it has to go and remember them.

Think of an API key as a numbered coat-check ticket. The ticket itself tells you nothing — it's just a number. To find out what the ticket entitles you to, the attendant has to walk into the back room, find the matching record, and read it. The ticket is meaningless without the back room.

That "back room" is a database, and consulting it is called a database roundtrip — the server pauses, sends a question to the database, and waits for an answer before it can continue.

The hidden cost of a roundtrip

For most applications, one database lookup per request is perfectly fine. But consider what the server actually has to check each time an API key arrives:

Permissions — what is this key allowed to do?
Expiry — is this key still valid, or has it passed its end date?
Status — has someone revoked or suspended this key?

All three answers live in the database, not in the key. So every single request triggers a roundtrip before any real work happens.

Figure 1. API Keys and Database Roundtrips

This becomes a problem at scale in two specific situations:

High request volume. A popular API might handle thousands of requests per second. Thousands of identity lookups per second is real, measurable load — and load that does nothing for the user except verify who they are.
Distributed systems. Modern applications are often split into many small, independent services (commonly called microservices — a single application built as a collection of small, separately running programs that talk to each other). If a request passes through five services, and each one independently re-checks the caller's identity against the database, that's five roundtrips for one user action. The database becomes a bottleneck that every service depends on.

Figure 2. API Key Validation Scaling Problem

The deeper issue is state. A server that must consult a database to understand a credential is a stateful server — it cannot make a decision on its own. It always needs the back room.

The core idea behind JWT

JWT flips the model. Instead of handing the server a meaningless ticket and forcing it to look up the details, JWT hands the server a ticket that has the details printed on it — and stamped in a way that proves the printing wasn't forged.

Figure 3. JWT and Stateless Validation

To stay with the analogy: a JWT is less like a coat-check number and more like a boarding pass. A boarding pass already states your name, your flight, your seat, and your boarding time, right there on the paper. The gate agent doesn't phone headquarters to look you up — they read the pass and check that it's genuine. The information travels with the traveler.

This property has a precise name. A server that can validate a token using only the token itself (plus a verification key it already holds) is stateless — it holds no per-request memory and needs no back room. We will unpack exactly how a JWT proves it hasn't been forged later, when we open up its three-part structure. For now, the one idea to carry forward is the trade:

An API key keeps the metadata in the database and gives you a pointer to it. A JWT puts the metadata in the token and gives you a way to trust it.

Figure 4. API Keys vs. JWT comparison

That trade is powerful, but — as Part III will explore honestly — it is not free. Information printed on a token cannot be un-printed, which creates a genuine difficulty when you need to cancel a token early.

A Few Words of History

Every technology arrives at a particular moment for a particular reason. JWT is no exception — it appeared just as the web was shifting away from a model that had quietly dominated for over a decade.

The world before: the session cookie

For most of the 2000s, web authentication worked through server-side sessions. When you logged in, the server created a record of your session in its own memory or database and handed your browser a small identifier — a session cookie. On every later request, your browser sent that cookie back, and the server looked it up to remember who you were.

This is the same coat-check pattern from the previous session above: the cookie is a meaningless number, and the server holds all the real information. It worked well when a website was a single server. But it tied every logged-in user to a specific machine's memory. Spread your traffic across many servers, and you hit a problem — a user logged in on Server A is a stranger to Server B, because Server B never created that session record.

The pressure that created JWT

Two trends in the early 2010s made server-side sessions increasingly awkward.

First, applications stopped being single servers. They were spread across fleets of machines, and increasingly split into microservices — many small programs, each handling one job. A shared, central session store became a bottleneck that every service had to consult.

Second, the clients changed. The web was no longer just browsers loading full pages. It was single-page applications (SPAs) — websites that load once and then behave like desktop apps — and native mobile apps, often talking to APIs hosted on entirely different domains. Cookies, which are tightly bound to a single domain by design, fit this new world poorly.

The industry needed a credential that any server could verify on its own, without a shared session store, and that wasn't anchored to one domain. The answer was to make the token carry its own proof of identity.

RFC 7519: the standard

The work happened inside the IETF (the Internet Engineering Task Force, the body that standardizes the protocols underlying the internet) as part of its OAuth working group — the same group designing the next generation of authorization standards. They needed a compact, secure token format to pass identity information around, and JWT was built to fill that gap.

The result was published in May 2015 as RFC 7519 — the formal specification that defines what a JSON Web Token is. An RFC ("Request for Comments") is the document format the IETF uses for internet standards; despite the tentative-sounding name, a published RFC is the authoritative definition.

A point worth noting: JWT was never a lone invention. It is the centerpiece of a small family of related standards — collectively called JOSE (JavaScript Object Signing and Encryption) — that also define how to sign tokens (JWS), encrypt them (JWE), and represent cryptographic keys (JWK).

For most practical purposes, though, "JWT" is the term everyone uses, and it's the one we'll use here.

From standard to ubiquity

A specification only matters if people adopt it, and JWT's adoption was unusually fast. The identity company Auth0, founded in 2013, built much of its product and developer education around JWT and promoted the format heavily — including jwt.io, a free online debugger that let developers paste in a token and see its decoded contents instantly. For a great many engineers, that tool was their first hands-on encounter with the format.

From there, JWT spread through the ecosystem. It became the default token format for OpenID Connect — the identity layer built on top of OAuth 2.0, and the subject we'll reach later in this series. Cloud platforms adopted it for service-to-service authentication. Web frameworks in nearly every programming language shipped libraries to create and verify it. Within a few years of RFC 7519, JWT had moved from a new proposal to a default assumption.

Why the history matters

This background isn't trivia — it explains the shape of the technology you're about to take apart. JWT looks the way it does because it was designed for a specific world: distributed systems with no shared memory, and clients scattered across domains and devices.

That origin also explains its central trade-off. JWT was built to let a server make a decision on its own, without calling back to a central store. That independence is exactly the feature that makes JWT fast and scalable — and, as we'll see, exactly the feature that makes a token hard to cancel once it's been issued. The strength and the weakness are the same design decision, viewed from two sides.

With that context in place, we can now open up an actual token and see what those three dot-separated parts really contain.

The Anatomy of a JWT

This is the heart of subject. Once you can see what a JWT actually is, every later topic — claims, signatures, validation, security — becomes far easier to follow. So we'll take a real token apart, piece by piece.

Three parts, two dots

A JWT, when written out, looks like a long, slightly intimidating string of gibberish:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NSIsIm5hbWUiOiJBZGEifQ.dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW1gFWFOEjXk

It looks random. It isn't. Look closely and you'll spot two dots dividing it into three parts:

header  .  payload  .  signature

That structure never changes. Every JWT, everywhere, is exactly three parts separated by dots:

The header — describes the token itself.
The payload — carries the actual information.
The signature — proves the first two parts haven't been tampered with.

The boarding-pass analogy from the first section holds up neatly here. The header is the small print at the top that says what kind of document this is. The payload is the part you actually care about — your name, your flight, your seat. The signature is the official stamp that proves the pass is genuine and not something printed at home.

Why it looks like gibberish: Base64URL

The reason a JWT looks unreadable is not encryption. It is encoding — and the distinction matters.

Encryption scrambles data so that no one can read it without a secret key.
Encoding simply rewrites data into a different alphabet so it can travel safely. Anyone can reverse it.

A JWT's header and payload are merely encoded, not encrypted. They use a scheme called Base64URL — a way of representing data using only letters, digits, and a couple of symbols that are safe to put inside a web address (a URL).

This leads to the single most important — and most misunderstood — fact about JWTs:

The contents of a JWT are not secret. Anyone who holds the token can decode the header and payload and read everything inside. The signature stops people from changing a token; it does nothing to hide it.

Figure 5. JWT Decoding vs. Hiding Secrets

We'll return to the security consequences of this later. For now, just hold onto it: a JWT protects against forgery, not against reading. Never put a password, a credit-card number, or any other secret inside it.

Part 1: The header

Take the first segment of our example token and reverse the Base64URL encoding, and you get a small block of JSON— JavaScript Object Notation, the simple, human-readable key: value format used everywhere on the modern web:

{
  "alg": "HS256",
  "typ": "JWT"
}

The header is metadata about the token. It answers two questions:

alg ("algorithm") — how the signature was created. Here, HS256. We'll examine the algorithm choices in detail in part of the article; for now, treat it as a label naming the method used to make the stamp.
typ ("type") — what kind of object this is. For a JWT, this is simply "JWT".

The header is short, and it's mostly there so the receiving server knows how to check the signature later.

Part 2: The payload

Decode the second segment and you get another block of JSON — and this is the part that does the real work:

{
  "sub": "12345",
  "name": "Ada"
}

The payload carries the claims — the individual statements the token is making. The word is well chosen: each entry is a claim, an assertion that "this is true." Here the token claims two things: the subject (sub) of this token is user 12345, and that user's name is Ada. The payload is where identity, permissions, expiry times, and your own custom data live.

Part 3: The signature

The third segment is the only part that is genuinely cryptographic — and it's the part that makes a JWT trustworthy.

Here is the problem the signature solves. We just established that anyone holding a token can read the payload. But could they also change it — swap "sub": "12345" for "sub": "99999" and impersonate another user? The answer is "YES". Does it stay hidden? And the most important answer is here - "NO", as the signature is what makes that attack fail because it becomes easily recognizable and server knows how to react on it.

Figure 6. JWT Forgery Protection - The Signature

When the server first creates the token, it performs a calculation. It takes the encoded header and the encoded payload, joins them with a dot, and feeds that — together with a secret key that only the server knows — into a one-way mathematical function. For our HS256 example, that function is HMAC-SHA256. The output is the signature:

signature = HMAC-SHA256(
    base64url(header) + "." + base64url(payload),
    secret
)

Two properties of this function are what make the whole scheme work:

It depends on every input. Change a single character of the header or payload, and the output is completely different.
It can't be reversed or faked without the secret. Knowing the inputs but not the secret, there is no practical way to compute the correct signature.

So when an attacker edits the payload to say "sub": "99999", the signature attached to the token no longer matches the modified content. When the server recomputes the signature from the tampered payload and compares it to the one in the token, the two don't match — and the token is rejected. Because the attacker doesn't have the secret key, they can't produce a fresh, valid signature for their forged payload either.

Figure 7. JWT Signature- The Key to Integrity and Honesty

And this is the core idea worth carrying out of this section:

The signature doesn't keep a JWT private. It keeps a JWT honest. It guarantees that the header and payload are exactly what the issuing server wrote, untouched since.

Building one by hand

The best way to dispel the sense that this is magic is to build a token with no library at all — just the standard tools any programming language provides. Here it is in Python:

import base64, json, hmac, hashlib

def base64url_encode(data: bytes) -> str:
    # Standard Base64, then made URL-safe: trailing '=' padding removed.
    return base64.urlsafe_b64encode(data).rstrip(b'=').decode()

def create_jwt(payload: dict, secret: str) -> str:
    # 1. The header: declare the algorithm and type.
    header = {"alg": "HS256", "typ": "JWT"}

    # 2. Encode the header and payload as Base64URL.
    h = base64url_encode(json.dumps(header).encode())
    p = base64url_encode(json.dumps(payload).encode())

    # 3. Sign "header.payload" with the secret using HMAC-SHA256.
    sig = hmac.new(secret.encode(), f"{h}.{p}".encode(), hashlib.sha256).digest()

    # 4. Join all three parts with dots.
    return f"{h}.{p}.{base64url_encode(sig)}"


token = create_jwt({"sub": "12345", "name": "Ada"}, secret="my-secret-key")
print(token)

That is the entire mechanism. There is nothing hidden. A JWT is two pieces of JSON, encoded for safe transport, plus a signature computed from them. The four numbered steps in the code are the whole story.

The production path: use a library

Building a token by hand is the right way to understand JWTs. It is the wrong way to use them in real software.

In production, always reach for a well-maintained, audited library — PyJWT or python-jose in Python, and direct equivalents in every other major language. The same task becomes a single call:

import jwt  # the PyJWT library

token = jwt.encode(
    {"sub": "12345", "name": "Ada"},
    "my-secret-key",
    algorithm="HS256"
)

This isn't laziness — it's safety. The hand-written version above is correct for creating tokens, but verifying them safely is full of subtle traps. A naive verifier can be tricked into accepting forged tokens, expired tokens, or tokens it should never have trusted. Mature libraries have already absorbed those lessons; your own code, written from scratch, has not.

So build one by hand once, to see that the magic is just arithmetic. Then never do it again in production.

With the structure of a token clear, we can turn to its most important part in detail — the payload, and the claims it carries.

Claims: What a Token Actually Says

In previous section we opened up a JWT and found the payload — the middle segment carrying the token's real content. Each piece of information in that payload is called a claim. This section is about what those claims are, which ones the standard defines for you, and which ones you create yourself.

A claim is just a statement

The terminology sounds formal, but the idea is plain. A claim is a single statement the token makes about its subject — one key: value pair in the payload's JSON. The token "claims" these things are true, and the signature is what makes that claim trustworthy.

A payload is simply a collection of these statements:

{
  "iss": "auth.myapp.com",
  "sub": "12345",
  "aud": "api.myapp.com",
  "iat": 1716120000,
  "exp": 1716123600,
  "role": "editor",
  "tenant_id": "acme-corp"
}

Some of those keys — iss, sub, aud, iat, exp — are part of the JWT standard and mean the same thing everywhere. Others — role, tenant_id — are invented by the application. That split is the central idea of this section.

Registered claims: the standard vocabulary

RFC 7519 defines a small set of registered claims. These are not required — a JWT is free to omit any of them — but if you do use them, you must use them with the meaning the standard assigns. They have short, three-letter names to keep tokens compact.

There are seven worth knowing:

Claim	Name	What it means
`iss`	Issuer	Who created and signed this token.
`sub`	Subject	Who or what the token is about — typically the user ID.
`aud`	Audience	Who the token is intended for — which service should accept it.
`exp`	Expiration Time	The moment after which the token must be rejected.
`nbf`	Not Before	The moment before which the token is not yet valid.
`iat`	Issued At	The moment the token was created.
`jti`	JWT ID	A unique identifier for this specific token.

A few of these deserve a closer look, because they do more than they first appear to.

iss (Issuer) identifies the authority that minted the token — usually an authentication server like auth.myapp.com. A receiving service checks this so it only trusts tokens from a source it recognizes, and ignores tokens minted by anyone else.

aud (Audience) is the one most often misunderstood, and it matters for security. It names the intended recipient. If your authentication server issues tokens for several different services, the aud claim lets each service confirm "this token was meant for me." A token minted for the billing service should be refused by the email service, even though both trust the same issuer. Skipping this check is a real vulnerability.

jti (JWT ID) is a unique serial number for one individual token. On its own it does little. Its importance shows up later: when you need to cancel a specific token before it expires, the jti is the handle you use to do it. That's the revocation problem, and jti is the thread that connects to it.

The temporal claims: `iat`, `nbf`, and `exp`

Three of the registered claims — iat, nbf, and exp — are about time, and together they give a token a lifespan. They are the mechanism behind one of JWT's most important properties: a token that automatically stops working.

Laid out on a timeline, they mark out a window of validity:

   iat            nbf              now                      exp
    │              │                │                        │
    ●──────────────●════════════════●════════════════════════●─────────►  time
  issued       valid from      this moment              expires after
                 │◄───────── token is VALID here ───────────►│

Figure 8. Timeline for JWT claims

iat (Issued At) is the token's birth timestamp — when it was created.
nbf (Not Before) is the moment the token starts being valid. Usually this is the same as iat, but it can be set in the future to issue a token now that only "switches on" later.
exp (Expiration Time) is the moment the token stops being valid. After this instant, every correct server must reject it, no matter what else the token says.

The exp claim is what gives a JWT its self-limiting nature. An API key, by contrast, tends to live until a human remembers to revoke it. A JWT carries its own deadline. Set exp to 15 minutes after iat, and the token simply expires 15 minutes later — no database, no cleanup job, no human action. This is central to managing the risk of a stolen token.

One technical detail: these timestamps are stored as Unix time — the number of seconds since midnight UTC on January 1, 1970. That's why exp appears as a large integer like 1716123600 rather than a human-readable date. It's a universal, timezone-free way to pin down an exact instant, and every JWT library converts it for you.

Custom claims: your own data

The registered claims cover identity and timing. They say nothing about what a user is allowed to do, because that is specific to your application. For everything else, you add custom claims — key: value pairs you define yourself, with whatever names and meanings you choose.

Custom claims are where JWT delivers on the promise from the section above — moving metadata into the token so the server doesn't need a database lookup. Typical examples:

role or permissions — what the user is authorized to do ("editor", "admin", or a list like ["read:invoices", "write:invoices"]). The receiving service reads this straight from the token and decides what to allow — no roundtrip required.
tenant_id — in applications that serve many separate organizations from one system, this identifies which organization the user belongs to.
email, name — basic profile fields, included to save a lookup when displaying the user's identity.

This is exactly the design that makes a server stateless: the answer to "who is this, and what may they do?" travels inside the request itself.

Two cautions, both flowing directly from facts already established.

First — and this bears repeating because it is the most common JWT mistake — the payload is not secret. Anyone holding the token can decode and read it. So custom claims may contain a user's role; they must never contain anything sensitive — no passwords, no API secrets, no private personal data.

Second, keep the payload small. A JWT travels with every single request to your API. Every claim you add makes every request slightly larger. Stick to what the receiving service genuinely needs to make a decision; resist the temptation to pack a user's entire profile into the token.

Avoiding name collisions

If your application invents a custom claim called role, and some other system your token passes through also uses role to mean something different, the two meanings collide. The JWT standard's recommended fix is to namespace custom claims — prefix them with a URL you control, so the name is globally unique:

{
  "sub": "12345",
  "https://myapp.com/role": "editor"
}

The URL doesn't have to point anywhere real; it's used purely as a unique prefix. For a closed system where you control every service, plain names like role are common and perfectly workable. For tokens that travel between organizations, namespacing prevents a class of subtle, hard-to-diagnose bugs.

The payload, in summary

A JWT's payload is a set of claims — statements the token asserts and the signature makes trustworthy. Some claims are registered by the standard: iss, sub, and aud answer who; iat, nbf, and exp answer when; jti gives the token a unique name. The rest are custom claims you define, and they are how identity and permissions ride along inside the request, sparing the server a database lookup.

We now know what a token says. The next question is how it proves it — which means looking closely at the signature, and the different algorithms used to create it.

Signature Algorithms: HS256, RS256, and ES256

Above we established what the signature does: it makes a token honest, guaranteeing the header and payload haven't been altered since the issuer wrote them. We used HS256 as the example and treated it as a black box. Now we open the box — because the choice of signing algorithm is one of the most consequential decisions you'll make when building a real system, and it turns entirely on a single question: who is allowed to verify the token?

Two families: symmetric and asymmetric

Every JWT signing algorithm belongs to one of two families. The difference between them is the difference between a shared password and a wax seal.

Figure 9. JWT Signing - Symmetric vs. Asymmetric

A symmetric algorithm uses one secret key for both jobs — creating the signature and checking it. The same key signs and verifies. It's like a password shared between two parties: whoever knows it can both lock and unlock.

An asymmetric algorithm uses a pair of mathematically linked keys: a private key and a public key. The private key creates signatures and is guarded closely. The public key only verifies signatures and can be shared freely — handed to anyone, posted publicly — without weakening anything. It's like a wax seal: the issuer owns the unique stamp (the private key), but anyone with a picture of the seal (the public key) can recognize a genuine one. Holding the public key lets you check a seal but never forge one.

That single distinction — one shared key versus a public/private pair — drives everything that follows.

HS256: the symmetric option

HS256 stands for HMAC using SHA-256. It is the symmetric choice, and it's the algorithm in every example so far.

With HS256, one secret string both signs and verifies tokens. The server that issues a token uses the secret to compute the signature; the server that receives a token uses the same secret to recompute the signature and compare. If they match, the token is genuine.

This is simple, fast, and produces compact signatures. But it carries one unavoidable consequence: every party that needs to verify a token must hold the secret key. And here is the catch — the verifying key and the signing key are the same key. Any service that can check a token can therefore also mint a brand-new token that everyone else will trust.

In a single, self-contained application, that's fine. The same server issues tokens and checks them. There's one secret, in one place, and no one else needs it.

The trouble begins when the system grows.

RS256: the asymmetric option

RS256 stands for RSA Signature using SHA-256. RSA is a long-established public-key cryptosystem; what matters here is that RS256 is asymmetric — it uses the private/public key pair.

With RS256, the authentication server holds a private key and uses it — and only it — to sign tokens. It then publishes the matching public key for the world to see. Any other service can take that public key and verify a token's signature. But the public key cannot sign anything. A service holding only the public key can confirm a token is genuine, yet has no power to forge one.

That asymmetry solves the exact problem HS256 created.

The "aha": why this matters for microservices

Recall the microservices picture from Figure 1 — an application split into many small, independent services. Suppose one of them, the authentication service, issues tokens, and a dozen others need to verify them.

With HS256, every one of those dozen services must hold the shared secret. That's a real problem on two fronts:

A wider attack surface. The secret now lives in twelve places instead of one. A breach of _any single service_leaks the key — and with it, the power to forge tokens that all twelve will trust.
Misplaced trust. Every service that can verify can also issue. A minor, low-privilege service now holds the keys to impersonate anyone. That violates a basic security principle: a component should hold only the power it actually needs.

With RS256, the picture changes completely:

The private key lives in exactly one place — the authentication service. Only it can mint tokens.
The public key is distributed to all twelve verifying services. It's not a secret; if one leaks, nothing is compromised, because a public key cannot forge anything.
Verifying services can do their job — confirming a token is genuine — without ever holding the power to create one.

Figure 10. Token Validation in Microservices - Securing the Keys

This is the central insight of the section, and it's worth stating plainly:

In a distributed system, use an asymmetric algorithm. It lets you separate the power to issue tokens from the power to verify them — concentrating the dangerous capability in one guarded place while distributing the harmless one freely.

ES256: asymmetric, but leaner

ES256 stands for ECDSA using SHA-256, where ECDSA is the Elliptic Curve Digital Signature Algorithm.

For decision-making purposes, ES256 belongs in the same category as RS256: it is asymmetric, with a private key for signing and a public key for verifying, and it solves the microservices problem in exactly the same way. The difference is mechanical. ES256 is built on elliptic-curve cryptography, a more modern branch of public-key cryptography that achieves the same security strength with much smaller keys and signatures.

The practical payoff is size. An RS256 signature is large; an ES256 signature, at comparable security, is substantially smaller. Since a JWT travels with every request, a smaller signature means a smaller token on every call. For a high-traffic API, that adds up.

The trade-off is maturity and ubiquity. RSA has been in use for decades and is supported absolutely everywhere; elliptic-curve support, while now excellent, is marginally less universal in older systems. In practice, both are solid choices — RS256 is the safe, conventional default, and ES256 is the leaner, more modern alternative.

Putting it side by side

	HS256	RS256	ES256
Family	Symmetric	Asymmetric	Asymmetric
Key(s)	One shared secret	RSA private/public pair	EC private/public pair
Who can verify?	Only holders of the secret	Anyone with the public key	Anyone with the public key
Can a verifier also forge tokens?	Yes — same key signs and verifies	No — public key can't sign	No — public key can't sign
Signature size	Small	Large	Small
Best fit	A single, self-contained service	Distributed systems / microservices	Distributed systems where token size matters

The decision rule is short. If your application is a single service that both issues and verifies its own tokens, HS256 is simple, fast, and entirely appropriate — there's no second party, so a shared secret shares nothing. The moment more than one independent service must verify tokens, switch to an asymmetric algorithm: RS256 as the well-supported default, ES256 when you want smaller tokens and your platform supports it. The question is never "which algorithm is best" in the abstract — it is "who, in my system, needs to verify a token, and should those same parties be able to create one?"

We now know how a token is signed. But verifying a token correctly involves much more than checking that one signature. The next section lays out the full validation checklist — and shows why skipping any step on it opens a door.

Validation: The Full Checklist

Here is a mistake that has shipped to production in countless real systems: a developer wires up JWT authentication, confirms the signature checks out, and considers the job done. The signature is valid — so the token is trustworthy. Right?

Not quite. A valid signature answers only one question: was this token altered since it was issued? It says nothing about when the token was issued, who issued it, who it was meant for, or whether it has been cancelled in the meantime. A token can have a perfect signature and still be one your server must refuse.

Proper validation is therefore a checklist, run in order. This section walks through all six steps, mirroring the security rigor of the earlier parts of this series.

Why the order matters

The steps are sequenced deliberately, and the first one comes first for a reason: until the signature is fully verified, you cannot trust a single byte of the payload.

Every later check — expiry, issuer, audience — reads a value from the payload. But if the signature hasn't been confirmed, the payload might be an attacker's forgery, and an attacker writes their exp claim to say whatever they like. Checking expiry on an unverified payload is theatre. So the signature is verified first; only once it passes does the payload become trustworthy enough to inspect.

The six-step checklist

Step 1 — Verify the signature. Recompute the signature from the token's header and payload (using the secret for HS256, or the public key for RS256/ES256) and confirm it matches the signature attached to the token. If it doesn't, the token is forged or corrupted — reject it immediately and run no further checks. If it matches, the payload is now proven authentic, and the remaining steps can rely on it.

Step 2 — Check exp (not expired). Read the expiration timestamp and confirm the current time is before it. An expired token is rejected even though its signature is flawless. This is the check that makes a stolen token stop working on its own — and the entire reason short expiry times are worth setting.

Step 3 — Check nbf (not used before valid). If the token carries a "not before" claim, confirm the current time is past it. A token presented before its activation moment is not yet valid and must be refused. In practice this rarely fires, but a complete validator checks it.

Step 4 — Check iss (expected issuer). Confirm the iss claim names an authentication server you actually trust. A token can be perfectly signed by some issuer, but if it isn't your issuer, it's irrelevant — reject it. This stops your service from honouring tokens minted by an unrelated, untrusted source.

Step 5 — Check aud (intended audience). Confirm the aud claim names your service. This is the check which is most often skipped — and skipping it is a genuine vulnerability. Without it, a token issued for the low-privilege analytics service would be accepted by the high-privilege billing service, simply because both trust the same issuer. The aud check is what keeps a token usable only where it was meant to be used.

Step 6 — Check jti against a revocation list (if needed). For most stateless setups this step is deliberately omitted — checking a token against a list means consulting a database, which sacrifices the very statelessness JWT was chosen for. But when you genuinely need the ability to cancel an individual token before it expires, this is where it happens: look up the token's jti and reject it if it appears on a blocklist. This step is the bridge revocation problem, and the reason it's marked "if needed" is itself the subject of that section.

What this looks like in code

Previously we said: build a token by hand once to understand it, then use a library in production. Validation is precisely where that advice earns its keep. A good library performs the entire checklist for you — if you tell it what to expect:

import jwt  # the PyJWT library

def validate_jwt(token: str, secret: str, expected_audience: str) -> dict:
    try:
        payload = jwt.decode(
            token,
            secret,
            algorithms=["HS256"],          # Step 1: which algorithm(s) to trust
            audience=expected_audience,    # Step 5: the aud claim must match this
        )
        # Steps 2 and 3 — exp and nbf — are checked automatically by decode().
        return payload

    except jwt.ExpiredSignatureError:
        raise AuthError("Token has expired")          # Step 2 failed
    except jwt.InvalidAudienceError:
        raise AuthError("Invalid audience")           # Step 5 failed
    except jwt.InvalidSignatureError:
        raise AuthError("Signature verification failed")  # Step 1 failed
    # ... and so on for the other failure cases

Three details in that small function carry the section's whole lesson.

First, jwt.decode() does several checks at once. A single call verifies the signature, the expiry, and the "not before" time. The library runs the checklist — but only the parts you've configured.

Second, a check you don't ask for doesn't happen. Notice audience=expected_audience is passed explicitly. Leave that argument out, and many libraries simply skip the audience check entirely — Step 5 silently vanishes, and no error is ever raised. The token validates "successfully," and a real vulnerability sits quietly in your code. The same applies to the issuer. Safe validation means actively telling the library what to expect; its defaults are not your security policy.

Third — and this is the most important line in the whole snippet — algorithms=["HS256"] is not optional. It tells the library exactly which signing algorithm(s) to accept. Omitting it, or filling it in carelessly, opens the single most infamous JWT vulnerability of all. We've now mentioned it twice as a forward pointer; later we finally show the attack itself.

The takeaway

A valid signature is the first line of a six-line checklist, not the whole of it. Full validation confirms a token is authentic(signature), current (exp, nbf), from a source you trust (iss), meant for you (aud), and not revoked (jti, when required). A production-grade library will run every one of those checks — but only the ones you explicitly configure. The unsafe defaults are silent, and silence, in security, is exactly the danger.

That sixth step — revocation — has been flagged twice now as something deferred. It's deferred because it isn't a simple checklist item; it's a genuine tension at the heart of the whole JWT design. The next section confronts it directly.

The Stateless Advantage — and the Revocation Problem

This is the most important section in the article, and the most honest. Everything so far has built toward a single payoff — and that same payoff comes bundled with a single, genuine flaw. A clear-eyed engineer needs both halves. This section gives you both.

The win: a server that needs no memory

Return to the very first idea of this series. An API key is a meaningless ticket; to learn what it permits, the server must make a database roundtrip to the back room. A JWT prints the details on the ticket itself and stamps them so they can't be forged.

Now we can state precisely what that buys you. When a JWT arrives, the receiving server checks it using only the token and a verification key the server already holds — the shared secret for HS256, or the public key for RS256/ES256. The signature confirms the token is authentic; the claims inside answer who the user is and what they may do. Nothing else is consulted.

That property is statelessness: the server keeps no per-user, per-session memory. It needs no shared session store, no identity database lookup on the request path. And the benefits compound:

Speed. No network trip to a database before real work begins. Verification is local arithmetic.
Scale. Add a hundred new servers and not one of them needs a connection to a session store. Each verifies tokens entirely on its own.
Resilience. Recall the microservices picture from figure 2 — a dozen services each verifying tokens. With JWT, none of them shares a session-store bottleneck. If that store would have existed, it would have been a single point of failure every service depended on. JWT removes it.

This is the killer feature. It is the reason JWT became the default token of the distributed-systems era.

The problem: you can't un-issue a token

Now the honest half.

The very thing that makes a JWT powerful — it is self-contained, valid purely on its own evidence — is the thing that makes it dangerous. A stateless server's logic is simply: "the signature is good and the token hasn't expired, therefore I trust it." That logic has no step that asks anyone's permission. It never phones home.

So consider: a token is stolen. An attacker copies a user's valid, unexpired JWT — through a compromised device, an intercepted request, a leaked log file.

With an old-style server-side session, the fix is instant. You delete the session record from the server. The next request carrying that session is a stranger; access is gone forever.

With a JWT, there is no record to delete. The token's authority lives inside the token, in the attacker's possession. Every server that sees it performs the same local check, gets the same answer — signature good, not expired — and grants access. The token keeps working, in the attacker's hands, until its exp timestamp passes. And nothing you do can hurry that moment along.

State this plainly, because it is the crux of the entire JWT trade-off:

A JWT is self-contained by design, which means it is non-revocable by design. The server gave up its memory in exchange for speed — and a server with no memory has no way to remember that one particular token should now be refused.

This is not a bug to be patched. It is the direct, unavoidable shadow of the feature. Statelessness and instant revocation are two ends of the same stick: you cannot pick up one end without the other coming with it.

Living with the trade-off

You can't eliminate the problem, but you can manage it. Three patterns are used in practice, and they sit on a deliberate spectrum — from "fully stateless, accept some risk" to "give back some statelessness, regain control."

1. Short expiry times

The simplest response: if a stolen token is valid until exp, then make exp soon.

Set tokens to expire in fifteen minutes rather than twenty-four hours. A stolen token is still unstoppable — but only for a short window. The damage is time-boxed. This keeps the server fully stateless; you simply accept a small, bounded exposure instead of trying to abolish it.

The objection is obvious, and it's a usability one: a token that expires every fifteen minutes would log the user out every fifteen minutes. Forcing someone to re-enter their password that often is unworkable. Which is exactly why short expiry never travels alone — it travels with the next pattern.

2. The access token + refresh token pattern

This is the pattern most production systems actually implement, and it resolves the usability objection cleanly by splitting one token into two tokens with two different jobs.

The access token is a JWT, short-lived (say, 15 minutes). It rides along with every API request and does the real authorization work. Being a JWT, it is verified statelessly — fast, local, no database. If stolen, it's dangerous for only 15 minutes.
The refresh token is long-lived (days or weeks), but it is not used for ordinary API calls. Its only power is to ask the authentication server for a fresh access token. It is sent rarely, to one endpoint only, and it is typically a stored, revocable credential — the auth server does keep a record of it.

Figure 11. Access & Refresh Token Split Pattern

The whole lifecycle is as follows:

1. User logs in.
   → Auth server issues:  a short-lived ACCESS token  +  a long-lived REFRESH token.

2. For each API request:
   → Client sends the ACCESS token.  Service verifies it statelessly. Fast.

3. After ~15 minutes:
   → The ACCESS token expires. The next API call is rejected.

4. Client quietly sends the REFRESH token to the auth server.
   → Auth server checks the refresh token is still valid (and not revoked),
     then issues a brand-new short-lived ACCESS token.

5. Loop back to step 2. The user notices nothing.

The elegance is in the division of labour. The token used constantly (the access token) is stateless and fast, exactly where speed matters — but barely dangerous, because it expires almost immediately. The token that is long-lived and genuinely dangerous (the refresh token) is used rarely, hits only one endpoint, and can be revoked at that one chokepoint. To cut off a compromised user, you invalidate their refresh token: within fifteen minutes their access token expires, the refresh fails, and they're locked out.

You haven't achieved truly instant revocation — there's still that ≤15-minute tail — but you've shrunk the unstoppable window to something tolerable while keeping the request path stateless. For most systems, that is the right balance.

3. A revocation blocklist

When even a fifteen-minute window is unacceptable — banking, healthcare, anything where a compromised session must die now — you need true immediate revocation. And here you must pay the honest price.

The pattern: recall the jti claim, the token's unique serial number. To revoke a token immediately, add its jti to a blocklist — a list of "tokens that must now be refused" — and have every service check incoming tokens against it. To keep that check fast, the list lives in a high-speed in-memory store such as Redis, not a slow disk database.

Be clear-eyed about what this costs. Checking a blocklist means consulting a shared store on every request — which is precisely the database roundtrip that JWT was chosen to eliminate. You have, deliberately, given back the stateless advantage.

But the surrender is partial, and that nuance matters:

A blocklist holds only revoked tokens — a tiny set — not every session. The store stays small.
An in-memory store like Redis is far faster than the relational-database lookup the original API-key model implied.
Entries can be evicted the moment a token's natural exp passes — a revoked token already rejected by the expiry check needs no blocklist entry. The list stays small on its own.

So this isn't a wholesale return to stateful sessions. It's a precise, conscious concession: trade a little speed, on a small fast lookup, to buy back the immediate-revocation capability — but only when the application's risk profile genuinely demands it.

The honest summary

JWT's defining feature and JWT's defining flaw are the same fact, seen from two sides. A self-contained token lets a server decide on its own — fast, scalable, memoryless. A self-contained token also cannot be un-decided — once issued, it is authoritative until it expires.

There is no pattern that makes that flaw vanish. There are only positions on a spectrum:

Approach	Statelessness	Revocation speed	Typical use
Short expiry only	Fully stateless	None — wait for `exp`	Low-risk, simple systems
Access + refresh tokens	Stateless request path	Bounded — within the access token's lifetime	The mainstream default
`jti` blocklist	Partly stateful again	Immediate	High-stakes systems only

The engineering question is never "how do I make JWT revocable?" — it can't be made so without giving something up. The question is: how much immediate-revocation power does this system genuinely need, and how much statelessness am I willing to trade for it? Answer that honestly, pick your point on the spectrum, and you are using JWT well.

We've now covered what a JWT is, what it carries, how it's signed, how it's validated, and its central trade-off. One thing remains: the specific ways JWT implementations get attacked — including the single most infamous JWT vulnerability of all.

Security Considerations

A JWT is a security mechanism, which makes its own failure modes worth studying with care. Most JWT disasters are not failures of the cryptography — the math is sound. They are failures of implementation: a check skipped, a default trusted, a token stored carelessly. This section covers the four that matter most, beginning with the most famous of them all.

The algorithm confusion attack: `alg: none`

We have pointed at this vulnerability twice and promised to show it here. It is the most infamous JWT bug ever, and what makes it memorable is that it turns the token's own honesty mechanism against itself.

Recall the structure of the token. The header declares which algorithm was used to sign the token, in its alg field — "alg": "HS256". The signature is the stamp that proves the token wasn't altered.

Now recall a detail almost no one expects: the JWT standard defines a legitimate algorithm value called none. It means "this token is unsigned." It exists for narrow cases where a token's integrity is already guaranteed by some other layer. It also creates a trapdoor.

Here is the attack, step by step:

1. The attacker takes a real, valid token and decodes it
   (remember - the payload is NOT secret, anyone can read it).

2. The attacker EDITS the payload freely:
        "sub": "regular-user"   →   "sub": "admin"

3. The attacker changes the header:
        "alg": "HS256"          →   "alg": "none"

4. The attacker DELETES the signature entirely,
   leaving a token that ends in a final dot and nothing after it:
        header.payload.

5. The token is sent to the server.

Now everything depends on one line of the server's code. A naively written verifier reads the header, sees "alg": "none", and reasons: "The header says this token is unsigned, so there is no signature to check. No signature to check means nothing fails. Token accepted."

Figure 12. JWT Attack with "alg" set to "none"

The forged token sails through. The attacker is now admin. They never needed the secret key, never broke any cryptography — they simply let the token tell the server how to verify it, and the token, under attacker control, said "don't bother."

That is the heart of the trap: a naive verifier trusts the token's own header to decide how the token should be checked. But the header is attacker-controlled. You have let the thing being inspected dictate the terms of its own inspection. (If this sounds like the embedded-public-key problem discussed earlier — a token vouching for its own trust — it is exactly the same mistake, in a different costume.)

*The fix is one line, and you have already seen it:

jwt.decode(token, secret, algorithms=["HS256"])

That algorithms=["HS256"] argument is the entire defense. It instructs the library: "I will accept HS256 and nothing else. I don't care what the token's header claims." A token arriving with alg: none is now rejected before any verification logic runs, because none is not on the list the server chose.

The principle, stated generally:

Never let the token decide how it should be verified. The set of acceptable algorithms is a decision the server makes in advance and enforces — not a value it reads out of the untrusted header.

Modern, well-maintained libraries now refuse alg: none by default and require you to specify allowed algorithms — which is exactly why we insisted on using a real library rather than hand-rolling verification. But the lesson outlives this one bug: a verifier must hold its own policy, never inherit it from the token.

A second face of algorithm confusion: HS256 vs RS256

There is a subtler cousin of the same attack.

Recall the asymmetric model: with RS256, the server signs with a private key and publishes the public key for anyone to verify with. The public key is, by design, not secret.

Now picture a server configured for RS256 but with a careless verifier that accepts whatever algorithm the header names. An attacker takes the public key — which is freely available — and uses it as if it were an HS256 shared secret. They forge a token, set the header to "alg": "HS256", and sign it using the public key as the HMAC secret.

Figure 13. RS256/HS256 Key Confusion Attack Path

When the token arrives, the confused server sees alg: HS256, fetches its RS256 public key as the "secret," runs the HS256 check — and it passes. The attacker has forged a valid token using only public information.

The fix is the same fix: pin the accepted algorithm. A server expecting RS256 must specify algorithms=["RS256"] and refuse everything else, so a token claiming HS256 is rejected outright. Once again — the server decides, not the token.

Secret strength for HS256

This one is simple and easily neglected. With HS256, the security of every token rests entirely on one shared secret. The signature is only as strong as that secret is hard to guess.

A secret like "secret", "password123", or your application's name is not a secret — it is an invitation. An attacker who suspects HS256 can run an offline brute-force attack: take any real token they've captured, and rapidly try millions of candidate secrets, checking each by recomputing the signature. A weak secret falls in seconds. Once it's found, the attacker can mint unlimited valid tokens for anyone.

The defense is unglamorous but absolute:

Use a long, high-entropy, randomly generated secret — at least 256 bits (32 random bytes), produced by a cryptographic random generator, never typed by a human.
Store it as a secret — in a secrets manager or environment configuration, never hard-coded in source, never committed to version control.
This concern is HS256-specific. With RS256/ES256 there is no shared secret to guess; the private key is a different kind of object, guarded differently.

Token storage in the browser

When a JWT is used to authenticate a user in a web browser, a practical question arises: where does the browser keep the token between requests? This debate is mostly relevant to browser-based apps rather than the pure API-to-API use that is this series' main focus, so we'll keep it brief — but it's worth knowing the shape of it.

There are two common choices, each with a different weakness:

localStorage is a simple in-browser storage area. It's easy to use, but any JavaScript running on the page can read it. If an attacker manages to inject a malicious script into your site — a cross-site scripting (XSS) attack — that script can read the token straight out of localStorage and steal it.
An httpOnly cookie is a cookie the browser marks as off-limits to JavaScript. Even a successful XSS attack cannot read it directly, which closes the theft route above. The trade-off: cookies are sent by the browser automatically, which can expose the app to a different attack class — cross-site request forgery (CSRF) — that needs its own separate defenses.

There is no universally "correct" answer; each option trades one risk for another. The genuinely important point sits underneath both:

A JWT is a bearer token — like cash, whoever holds it can spend it. The token itself contains no proof that the right person is presenting it. So the security of any browser-based JWT system depends heavily on simply not letting the token get stolen — and that depends on the surrounding application being free of injection flaws.

Short expiry as a discipline

The final consideration is not a new attack but a habit that blunts all of them — and we've already met it.

Every threat in this section ends the same way: an attacker obtains a token they shouldn't have. And we already know JWT's hard truth — a stolen token cannot be recalled; it is valid until its exp.

This makes the expiry time itself a security control. A long-lived token turns any single theft into a long-lived breach. A short-lived one — minutes, not days — means a stolen token is a briefly useful prize, paired with the refresh-token pattern so users never feel the churn.

Short expiry doesn't prevent theft. It caps the cost of theft. It is the safety net beneath every other mistake, and it should be treated as non-negotiable discipline rather than a tuning knob.

The thread connecting all four

Step back and the four considerations rhyme. The algorithm confusion attack is trusting the token to define its own verification. Weak HS256 secrets are trusting an easily guessed value to be hard. Careless token storage is trusting a hostile environment to keep a bearer token safe. And the antidote running through all of them is the same instinct: trust must be anchored in decisions the server makes and controls — never in something the token, the attacker, or the environment supplies. The cryptography in JWT is rarely what fails. The trust assumptions around it are.

With the failure modes mapped, only the practical verdict remains: when JWT is the right tool for the job — and when it isn't.

When to Use JWT — and When Not To

We've taken JWT apart completely: its structure, its claims, its signatures, its validation, its central trade-off, and its failure modes. Knowing how a tool works is not the same as knowing when to reach for it. This section is the practical verdict — a decision guide for an engineer building a real system.

The honest framing is this: JWT is not a better credential than the API key from Part II. It is a different credential, built for a different job. Picking the right one means matching the tool to the situation.

When JWT is the right choice

JWT earns its place whenever its defining feature — a self-contained, statelessly verifiable token — solves a problem you actually have.

Stateless microservices — the killer use case. This is the scenario JWT was built for, and the one where it has no real rival. When an application is split into many small services, and a request may pass through several of them, a JWT lets each service verify the caller independently — with only a public key, no shared session store, no database roundtrip. The token carries identity and permissions with it. If there is one situation where JWT is unambiguously the answer, this is it.

Cross-domain authentication — SPAs and mobile apps. Recall that JWT emerged partly because session cookies are bound to a single domain and fit modern clients poorly. A JWT is just a string. It travels comfortably from a single-page app or a native mobile application to an API hosted on an entirely different domain, with none of the domain-binding friction cookies impose. When your client and your API don't share an origin, JWT removes a real obstacle.

Short-lived service-to-service tokens. When one internal service needs to call another and prove "this request is genuinely from me," a short-lived JWT is an excellent fit. Its built-in expiry means these machine-to-machine tokens clean up after themselves — no long-lived credential left lying around, no revocation process to run. They are minted, used, and expire, all within minutes.

As a session-scoped layer on top of a long-term API key. This is the subtlest fit, and it shows JWT and the API key working together rather than competing. An API key (Part II) is excellent for stable, long-term identity — "this is the Acme Corp account." A JWT is excellent for short-lived, fine-grained authorization — "this particular request, in this 15-minute window, may write to invoices." A common production pattern: a client authenticates once with its long-lived API key, and in exchange receives a series of short-lived JWTs that carry the actual per-request permissions. The API key answers who you are over time; the JWT answers what you may do right now.

When to reach for something else

JWT's weaknesses are not hidden and we stated them plainly. Two situations expose them directly, and in both, a different tool is the honest choice.

When you need immediate, reliable revocation. This is JWT's defining flaw, and it is worth refusing to paper over. A standard JWT cannot be un-issued — it is valid until exp, full stop. If your system has a hard requirement that a compromised credential must die the instant you say so — with no fifteen-minute tail, no acceptable window — then a self-contained token is fighting your requirement.

The alternative here is an opaque token checked by introspection: the token is a meaningless reference string (like the API key of Part II — a coat-check ticket), and the receiving service asks a central authority, on every request, "is this token still valid?" That is a database roundtrip. You are deliberately giving up statelessness — but in exchange you get exactly the instant, authoritative revocation that statelessness cost you. For high-stakes systems, that is the right trade.

When the token is genuinely long-lived. If a credential is meant to last for months — a stable identifier for a server, a long-running integration — then JWT's machinery is working against you. Its central feature, the self-contained expiring claim set, brings no benefit when nothing is meant to expire soon, and its central flaw, non-revocability, becomes a serious liability over a long lifespan. For durable, long-term identity, a plain API key is simpler, and — because it's checked against a database anyway — it can be revoked the moment you need to. Don't reach for a JWT's complexity to do an API key's job.

The decision in one table

Your situation	Reach for	Why
Many services must verify a caller independently	JWT	Stateless verification, no shared session store
Client and API live on different domains	JWT	A plain string travels anywhere; no cookie domain-binding
Short-lived, self-expiring service-to-service calls	JWT	Built-in expiry; no revocation process needed
Stable, long-term identity for an account or server	API key (Part II)	Simpler; revocable; nothing needs to expire soon
A compromised credential must be killable instantly	Opaque token + introspection	True immediate revocation, at the cost of statelessness

The underlying judgment

Reduce all of the above to a single question and it becomes easy to remember:

Does this system need a server to decide on its own, fast and at scale — or does it need a central authority to stay in control, able to revoke instantly?

JWT is the tool for the first. It trades central control for local, stateless speed — a brilliant trade when speed and scale are what you need, a poor one when control is. Opaque tokens and API keys are the tools for the second.

There is no "best" credential, only a credential that fits. An engineer who can name why they chose JWT — and name what they gave up to get it — is using it well. One who reaches for it by default, because it is the modern-sounding option, has skipped the only question that matters.

That leaves one piece of this series unfinished. JWT answers, beautifully, the question who are you? It does not answer a different and harder one: how do you let a third party act on your behalf — without ever handing them your password? That is the problem the final part was built to solve.

Conclusion: Three Tools, Three Jobs

We began this article with a coat-check ticket and we end it with a boarding pass. That shift — from a meaningless token the server must look up, to a self-describing token the server can read and trust on sight — is the whole of what JWT contributes. It's worth gathering the journey into a single picture before we close.

What JWT actually gave us

Strip away the Base64URL, the claims tables, the algorithm names, and one idea remains: a JWT moves metadata out of the database and into the token itself, stamped so it cannot be forged.

That single move is the source of everything else in this article. It is why a server can be stateless — verifying a request with only a key it already holds, no roundtrip, no shared session store. It is why JWT scales so gracefully across microservices and travels so freely across domains. And it is — unavoidably — why a JWT cannot be un-issued: a server that gave up its memory for speed has no memory in which to record that one token should now be refused.

The strength and the flaw are the same fact seen from two sides. An engineer who holds both halves in mind at once understands JWT. One who remembers only the strength will, sooner or later, be surprised by the flaw.

Three methods, side by side

This series has now equipped us with three distinct credentials. They are not a ranking — not worse, better, best. They are three tools for three different jobs:

	Basic Auth (Part I)	API Key (Part II)	JWT (Part III)
What it is	Username + password sent on every request	A single opaque secret string	A signed, self-describing token
Carries its own data?	No	No — it's just a reference	Yes — identity, permissions, expiry, all inside
Server needs a lookup?	Yes, every request	Yes, every request	No — verified statelessly with a key
Built-in expiry?	No	No	Yes — the `exp` claim
Revocable immediately?	Yes (change the password)	Yes (delete the key)	No — valid until it expires
Best at	Simple, internal, low-stakes access	Stable, long-term identity	Stateless, distributed, short-lived authorization

Read the table top to bottom and the trade-off snaps into focus. Basic Auth and the API key keep their information in the database — which makes them slower per request, but instantly revocable. JWT carries its information in the token — which makes it fast and stateless, but stubbornly non-revocable. You cannot have both. Every authentication design is, at heart, a choice about where the trust lives: in a central store the server must consult, or in the token the server can read on its own.

Choosing well means naming that trade-off out loud — not reaching for the most modern-sounding option by reflex, but asking what this system actually needs, and what it can afford to give up.

The one question that remains

And yet, for all three of these methods, notice what they have in common. Basic Auth, the API key, and the JWT all answer the same single question:

"Who are you?"

They are mechanisms of authentication — proving identity. Each one assumes the party presenting the credential is the party it belongs to, acting on its own behalf.

Figure 14. Authentication vs. Authorization

But a great deal of the modern web does not work that way. When you let a photo-printing service reach into your cloud storage, or allow an analytics dashboard to read your calendar, something more delicate is happening. You are granting a third party access to your resources — and you would never dream of handing them your password to do it.

None of the three tools in this series solves that. A JWT can prove who you are beautifully; it has nothing to say about how you safely delegate a sliver of your access to someone else.

That problem — how to grant a third party limited access to your resources without ever sharing your credentials — is precisely the problem the next and final part of this series was built around. It is the problem of authorization and delegation, and the answer is the framework that quietly underpins nearly every "Sign in with…" button and connected app on the internet: OAuth 2.0.

We have spent three articles learning, in ever-greater depth, how to answer who are you? It is time to ask the harder question.

API Authentication: Part II. API Keys

Eugene Zimin — Mon, 02 Dec 2024 08:38:33 +0000

Few Words of History

Following our exploration of Basic Authentication, the next in line are API Keys - a widely adopted authentication mechanism that addresses many of the limitations of Basic Authentication while maintaining simplicity in implementation. API Keys represent an evolution in API authentication, offering a balance between security and usability that has made them a popular choice for modern web services.

The concept of API Keys emerged as web services began to scale beyond simple and single client-server interactions. Unlike Basic Authentication's username-password paradigm, API Keys introduced a single, long-lived token approach that better suited programmatic access to APIs. This shift reflected the growing need for machine-to-machine communication in distributed systems, where traditional username-password combinations proved cumbersome.

The rise of public APIs in the mid-2000s catalyzed the widespread adoption of API Keys. Services like Google Maps, Twitter, and Amazon Web Services popularized this authentication method, demonstrating its effectiveness in managing access to public APIs at scale. This adoption marked a significant step forward in API security, introducing concepts like rate limiting, usage tracking, and granular access control that weren't easily achievable with Basic Authentication.

Understanding API Keys

At their core, API Keys are long, unique strings that serve as both identifier and authenticator for API clients. Unlike Basic Authentication's two-part credential system, an API Key combines identification and authentication into a single token. This architectural choice fundamentally changes how systems handle authentication and brings several important implications for system design.

Consider a practical example of how this unified approach manifests in real-world applications:

# Traditional Basic Auth approach
def authenticate_with_basic(username, password):
    stored_password_hash = database.get_password_hash(username)
    if not stored_password_hash:
        return False  # Username doesn't exist

    return verify_password(password, stored_password_hash)

# API Key approach
def authenticate_with_api_key(api_key):
    key_data = database.get_key_data(api_key)
    if not key_data:
        return False  # Invalid key

    return key_data  # Contains identity and permissions

In this example, the basic authentication process requires two distinct steps: first finding the user, then verifying their password. The API Key approach consolidates these steps - the key itself contains or points to all necessary information. When using Basic Authentication, the system must maintain separate storage for usernames and password hashes, while API Keys allow for a more simplified data structure.

Let's examine a more complex example that demonstrates the practical implications of this unified approach:

class APIKeyAuthenticator:
    def __init__(self, database_connection):
        self.db = database_connection

    def issue_key_for_service(self, service_name, permission_level):
        # Generate a cryptographically secure random key
        raw_key = secrets.token_urlsafe(32)

        # Create a structured key with embedded metadata
        key_prefix = self._generate_prefix(service_name)
        timestamp = int(time.time())
        api_key = f"{key_prefix}.{timestamp}.{raw_key}"

        # Store key metadata
        key_metadata = {
            'service': service_name,
            'permissions': permission_level,
            'created': timestamp,
            'last_used': None
        }
        self.db.store_key_metadata(api_key, key_metadata)

        return api_key, key_metadata

    def validate_request(self, api_key, requested_action):
        key_data = self.db.get_key_metadata(api_key)
        if not key_data:
            return False

        # Update usage timestamp
        key_data['last_used'] = int(time.time())
        self.db.update_key_metadata(api_key, key_data)

        return self._check_permissions(key_data['permissions'], requested_action)

This implementation demonstrates several key concepts. The issue_key_for_service method creates a structured API key that embeds useful metadata directly in the key format. The key consists of three parts:

a service-specific prefix,
a timestamp, and
a random component. This structure allows for quick identification of the key's origin and age without accessing the database.

The validate_request method shows how authentication and authorization become intertwined with API Keys. A single database lookup retrieves all necessary information about the key's associated service and permissions. This contrasts with Basic Authentication, where separate lookups might be needed for authentication and permission checking.

The practical implications of this unified approach become clear when examining system behavior. For instance, when a service needs to rotate its API key:

def rotate_service_key(authenticator, service_name, old_key):
    # Verify the old key is valid
    if not authenticator.validate_request(old_key, 'rotate_key'):
        return None, False

    # Get existing permissions from old key
    old_key_data = authenticator.db.get_key_metadata(old_key)
    permission_level = old_key_data['permissions']

    # Issue new key with same permissions
    new_key, _ = authenticator.issue_key_for_service(
        service_name, 
        permission_level
    )

    # Invalidate old key after grace period
    authenticator.db.schedule_key_deletion(old_key, grace_period_hours=24)

    return new_key, True

This rotation process demonstrates the elegance of the unified token approach. The entire service identity and permission set transfers seamlessly to the new key, while the old key can be gracefully deprecated. In a Basic Authentication system, changing credentials might require updating multiple database records and managing password change workflows.

Key Management and Storage

The management and storage of API keys present unique challenges that require careful consideration. In a production environment, the storage mechanism must balance security, performance, and scalability. Let's explore a comprehensive implementation that addresses these concerns with the class called APIKeyManader

from datetime import datetime, timedelta
import hashlib
from typing import Optional, Dict
import secrets
import base64

class APIKeyManager:
    def __init__(self, database_connection):
        self.db = database_connection
        self.hash_algorithm = 'sha256'
        self.key_prefix_length = 8

Hashing

Our class will contain few instance variables defining DB connection, hashing algorithm and the prefix length. Then we need to consider a method to create a hash key:

    def hash_key(self, api_key: str) -> str:
        return hashlib.new(self.hash_algorithm, api_key.encode()).hexdigest()

Usage of the method is pretty simple:

>>> manager.hash_key("pk_test_abc123")
"8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92"

Creating a hash instead of storing an API Key directly in the database is another security implication. If an attacker gains read access to this database through SQL injection, backup file exposure, or any other security breach, they immediately obtain valid API keys for all clients. These keys remain fully functional until manually revoked.

Here's what happens when an attacker breaches this database:

They see only hashes, which are one-way mathematical transformations
Even with modern computing power, it's practically impossible to reverse these hashes to obtain the original API keys
Each key uses a unique salt, preventing attackers from using rainbow tables (pre-computed hash databases)

# Example of what an attacker might see in the database
compromised_data = {
    'record_1': {
        'key_hash': '8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92',
        'salt': 'a1b2c3d4e5f6g7h8',
        'client_id': 'client_123'
    },
    'record_2': {
        'key_hash': '472b07b9fcf2c2451e8781e944bf5f77cd8457488d8f45f2318eb1b5b4175276',
        'salt': '9i8h7g6f5e4d3c2',
        'client_id': 'client_456'
    }
}

it won't be helpful as it is practically impossible to restore the API Key based on the hash information only. The security of this approach relies on some mathematical principles.

Hash functions are one-way operations
The same input always produces the same hash
Small changes in input produce completely different hashes
It's computationally infeasible to find an input that produces a specific hash

For instance:

# Demonstration of hash properties
api_key = "pk_live_abc123"
hash1 = hashlib.sha256(api_key.encode()).hexdigest()
print(f"Hash of '{api_key}': {hash1}")

# Changing just one character produces a completely different hash
api_key_modified = "pk_live_abc124"
hash2 = hashlib.sha256(api_key_modified.encode()).hexdigest()
print(f"Hash of '{api_key_modified}': {hash2}")

This will output two completely different hashes, even though the input strings differ by only one character:

Hash of 'pk_live_abc123': 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92
Hash of 'pk_live_abc124': 472b07b9fcf2c2451e8781e944bf5f77cd8457488d8f45f2318eb1b5b4175276

Adding Metadata

However creating an abstract key is almost meaningless as we need to associate this key with specific user information. For this purpose we need to be able to link the API Key to the use data.

    def create_key_with_metadata(self, 
                               client_id: str, 
                               permissions: list[str],
                               expiry_days: int = 90) -> Dict:
        # Generate random bytes and encode as base64
        random_bytes = secrets.token_bytes(32)
        random_part = base64.urlsafe_b64encode(random_bytes).decode()

        # Create structured API key with prefix
        prefix = "pk_live" if self.is_production() else "pk_test"
        api_key = f"{prefix}_{random_part}"

        # Calculate expiration date
        created_at = datetime.utcnow()
        expires_at = created_at + timedelta(days=expiry_days)

        # Prepare metadata for storage
        metadata = {
            'client_id': client_id,
            'permissions': permissions,
            'created_at': created_at.isoformat() + 'Z',
            'expires_at': expires_at.isoformat() + 'Z',
            'key_hash': self.hash_key(api_key),
            'status': 'active'
        }

        return {
            'raw_key': api_key,
            'metadata': metadata
        }

The create_key_with_metadata method represents a fundamental operation in API key management - the creation of a new API key for a client.

The method accepts three parameters: a client identifier (who the key is for), a list of permissions (what they can do), and an optional expiry period in days (when the key will stop working). By default, keys expire after 90 days to enforce regular rotation.

The output consists of two critical components packaged together: the API key itself and its associated metadata. The API key is what we'll provide to the client - it's their credential for accessing our API. The metadata, on the other hand, contains everything our system needs to know about this key: who owns it, what they're allowed to do with it, when it was created, when it expires, and a secure hash of the key for verification purposes.

Think of this like issuing a security badge at a facility. The physical badge (analogous to our API key) goes to the person, while all the information about that badge - who it belongs to, what doors it can open, when it expires - gets stored in the security system's database (our metadata).

The key itself follows a specific format: a prefix indicating whether it's for production or testing, followed by a cryptographically secure random string. This structured format helps with key management and debugging, while maintaining the security properties we need.

The method ensures several security properties: unpredictability through cryptographic randomness, safe storage through hashing, and automatic expiration through timestamp tracking. These elements combine to create a robust, manageable authentication token system.

Validating API Key

    def validate_key(self, api_key: str) -> Optional[Dict]:
        key_hash = self.hash_key(api_key)
        metadata = self.db.get_key_metadata(key_hash)

        if not metadata:
            return None

        # Check if key has expired
        expires_at = datetime.fromisoformat(
            metadata['expires_at'].replace('Z', '+00:00')
        )
        if datetime.utcnow() > expires_at:
            return None

        # Update last used timestamp
        metadata['last_used'] = datetime.utcnow().isoformat() + 'Z'
        self.db.update_key_metadata(key_hash, metadata)

        return metadata

The validate_key method serves as the gatekeeper for our API authentication system. When a client makes a request to our API with their key, this method determines whether that key is valid and active.

The process begins with the client's provided API key. The method first calculates its hash - the same process we used when storing the key. This hash acts as our lookup identifier in the database. Remember, we never store the actual API keys, only their hashes.

If we can't find any metadata associated with this hash in our database, it means the key either never existed or has been revoked. In this case, we immediately return None, indicating an invalid key.

For keys that do exist, we perform a temporal validation. The stored metadata includes an expiration timestamp - we parse this timestamp (converting it from ISO format with UTC indicator to a Python datetime object) and compare it with the current time. If the key has expired, we again return None. This expiration check is vital for enforcing security through key rotation.

When a key passes both these checks (exists and hasn't expired), we perform one final housekeeping task: updating the last_used timestamp. This tracking helps with key management, letting us identify unused keys that might need to be revoked.

Finally, for valid keys, we return the complete metadata associated with that key. This metadata contains essential information like the client's identity and permissions, which our API can use to make authorization decisions.

Think of this process like checking an ID card at a secure facility:

We first verify the ID exists in our system
We check if it hasn't expired
We log when it was last used
If everything checks out, we retrieve all the information about what the ID holder is allowed to do

This method is called frequently - potentially for every API request - so it needs to be efficient while maintaining strict security standards.

Implementation Patterns

When implementing API key authentication, there are few different transmission methods of API Keys between client and server. We'll examine the main approaches and their implications for real-world applications.

Header-based Authentication

The HTTP header approach represents the industry standard for API key transmission. By placing the API key in a custom header, we benefit from several security advantages:

def make_api_request(url: str, api_key: str, method: str = 'GET', 
                    data: dict = None) -> requests.Response:
    headers = {
        'X-API-Key': api_key,
        'Content-Type': 'application/json'
    }

    response = requests.request(
        method=method,
        url=url,
        headers=headers,
        json=data,
        verify=True  # Always verify SSL certificates
    )

    return response

and here is the usage example:

api_key = "pk_8a4c6f3e_1a2b3c4d5e6f7g8h9i0j"
response = make_api_request(
    url="https://api.example.com/data",
    api_key=api_key
)

The header-based approach offers several advantages. HTTP headers are not typically logged by default in most web servers and proxy systems and they're not directly visible by user. This means our API keys won't appear in log files, reducing the risk of accidental exposure. Additionally, headers aren't cached by browsers or included in browser history, providing another layer of security.

When implementing header-based authentication, we commonly prefix our custom header with X- to indicate it's a non-standard header. The X-API-Key header has become a de facto standard in the industry, though some systems use variations like Authorization with a custom prefix.

Query Parameter Authentication

Query parameter authentication places the API key directly in the URL as a parameter. Here's how this manifests in practice.

https://api.example.com/v1/data?api_key=pk_test_abc123def456

This approach faces several security challenges:

Server Logs Exposure: Most web servers include the complete URL in their logs:

`192.168.1.1 - - [01/Dec/2024:10:00:00 +0000] "GET /v1/data?api_key=pk_test_abc123def456 HTTP/1.1" 200 2326`

Browser History: The key becomes part of browsing history:

# Browser history entry
{
    'url': 'https://api.example.com/v1/data?api_key=pk_test_abc123def456',
    'timestamp': '2024-12-01T10:00:00',
    'title': 'API Request'
}

Referrer Headers: When clicking links on the API response page, the API key might be sent as a referrer:

Referer: https://api.example.com/v1/data?api_key=pk_test_abc123def456

While not recommended for production systems, query parameter authentication remains in use for some specific scenarios.

Static HTML/JavaScript environments where header modification isn't possible:

<img src="https://api.example.com/v1/image?api_key=pk_test_abc123" />

Direct browser access for development and testing:

curl "https://api.example.com/v1/test?api_key=pk_test_abc123"`

Legacy system compatibility where header modification isn't supported.

Security Considerations

When implementing API key authentication, security considerations extend far beyond the mere generation and validation of keys.

The fundamental challenge lies in the nature of API keys themselves - they are long-lived credentials that grant access to potentially sensitive resources. Unlike session-based authentication systems where credentials expire relatively quickly, API keys often persist for extended periods, making them attractive targets for malicious actors. This persistence, while convenient for legitimate users, requires us to implement robust security measures throughout the key lifecycle.

Consider an API key like a physical key to a building. Just as a physical key requires secure storage, controlled distribution, and periodic replacement, API keys demand similar careful management. The loss or compromise of an API key can have far-reaching consequences, potentially exposing not just individual resources but entire systems to unauthorized access.

Key Rotation and Revocation

Key rotation and revocation form the cornerstone of long-term API security strategy. While API keys might seem permanent, treating them as such introduces significant security risks. Let's examine how we implement these crucial security practices:

from datetime import datetime, timedelta

class APIKeyRotationManager:
    def __init__(self):
        self.rotation_period = timedelta(days=90)
        self.grace_period = timedelta(days=7)

The rotation period represents our security lifecycle. Setting it to 90 days balances security needs with operational convenience. The grace period allows systems to transition smoothly between old and new keys without service interruption.

Think of this like replacing locks in a building - we need to ensure new keys are distributed before old ones stop working. Here's how we manage this process:

def should_rotate(self, key_metadata: dict) -> bool:
    key_age = datetime.utcnow() - datetime.fromisoformat(
        key_metadata['created_at'].replace('Z', '+00:00')
    )

    # Standard age-based rotation
    if key_age >= self.rotation_period:
        return True

    # Check for suspicious activity patterns
    if self._detect_unusual_usage(key_metadata):
        return True

    return False

def rotate_key(self, old_key_metadata: dict) -> tuple[str, dict]:
    # Creates a new key while maintaining the old one during grace period.
    # Generate new key with same permissions
    new_key = self._generate_key()
    new_metadata = {
        'client_id': old_key_metadata['client_id'],
        'permissions': old_key_metadata['permissions'],
        'created_at': datetime.utcnow().isoformat() + 'Z',
        'rotated_from': old_key_metadata['key_hash'],
        'status': 'active'
    }

    # Update old key metadata
    old_key_metadata['status'] = 'rotating'
    old_key_metadata['rotates_at'] = (
        datetime.utcnow() + self.grace_period
    ).isoformat() + 'Z'

    return new_key, new_metadata

Our rotation strategy includes several critical patterns:

Regular rotation based on key age
Forced rotation when suspicious activity is detected
Overlapping validity periods to prevent service disruption
Maintenance of key lineage for audit purposes

Sometimes, we need more immediate action. Key revocation provides this capability:

def revoke_key(self, key_metadata: dict, reason: str) -> None:
    # this is a soft revoke
    revocation_data = {
        'status': 'revoked',
        'revoked_at': datetime.utcnow().isoformat() + 'Z',
        'revocation_reason': reason,
        'last_known_use': key_metadata.get('last_used')
    }

    # Update key metadata with revocation information
    key_metadata.update(revocation_data)

    # Trigger any necessary security alerts
    if reason in ['suspicious_activity', 'security_breach']:
        self._trigger_security_alert(key_metadata)

We also implement a validation system that respects both rotation and revocation states:

def validate_key_status(self, key_metadata: dict) -> bool:
    """
    Validates a key's status considering rotation and revocation.
    """
    if key_metadata['status'] == 'revoked':
        return False

    if key_metadata['status'] == 'rotating':
        rotation_deadline = datetime.fromisoformat(
            key_metadata['rotates_at'].replace('Z', '+00:00')
        )
        if datetime.utcnow() > rotation_deadline:
            return False

    return True

This approach to key lifecycle management ensures our system maintains security while remaining operationally viable. Regular rotation prevents the risks associated with permanent credentials, while our revocation system provides immediate response capability for security incidents.

Rate Limiting and Usage Tracking

Rate limiting and usage tracking serve as essential defensive mechanisms in API authentication systems. Just as a security guard monitors the frequency of entries into a building, these systems protect our API from abuse while providing valuable insights into usage patterns.

Let's examine a comprehensive implementation that handles both concerns:

from datetime import datetime, timedelta
from collections import defaultdict
import threading

class RateLimiter:
    def __init__(self):
        self.lock = threading.Lock()
        self.limits = {
            'requests_per_second': 10,
            'requests_per_minute': 300,
            'requests_per_hour': 5000,
            'requests_per_day': 50000
        }
        self.usage = defaultdict(lambda: {
            'second_count': 0,
            'minute_count': 0,
            'hour_count': 0,
            'day_count': 0,
            'windows': {
                'second': datetime.utcnow(),
                'minute': datetime.utcnow(),
                'hour': datetime.utcnow(),
                'day': datetime.utcnow()
            }
        })

This implementation introduces a multi-window rate limiting approach. Think of it like a building's security system that tracks entries across different time scales - from immediate access control to daily visitor limits. Here's how we enforce these limits:

def check_rate_limit(self, api_key: str) -> tuple[bool, dict]:
    with self.lock:
        now = datetime.utcnow()
        usage = self.usage[api_key]
        limits_status = {}

        # Reset counters if time windows have elapsed
        if now - usage['windows']['second'] >= timedelta(seconds=1):
            usage['second_count'] = 0
            usage['windows']['second'] = now

        if now - usage['windows']['minute'] >= timedelta(minutes=1):
            usage['minute_count'] = 0
            usage['windows']['minute'] = now

        # Update counters
        usage['second_count'] += 1
        usage['minute_count'] += 1

        # Check against limits
        limits_status = {
            'second': usage['second_count'] <= self.limits['requests_per_second'],
            'minute': usage['minute_count'] <= self.limits['requests_per_minute']
        }

        # Record usage pattern for analysis
        self._record_usage_pattern(api_key, now)

        return all(limits_status.values()), limits_status

Alongside rate limiting, we implement comprehensive usage tracking:

class UsageTracker:
    def __init__(self):
        self.usage_patterns = defaultdict(list)

    def record_request(self, api_key: str, request_data: dict):
        usage_record = {
            'timestamp': datetime.utcnow().isoformat() + 'Z',
            'endpoint': request_data.get('endpoint'),
            'method': request_data.get('method'),
            'response_status': request_data.get('status'),
            'response_time': request_data.get('response_time'),
            'client_ip': request_data.get('client_ip')
        }

        self.usage_patterns[api_key].append(usage_record)

    def analyze_usage_pattern(self, api_key: str) -> dict:
        patterns = self.usage_patterns[api_key]
        if not patterns:
            return {'risk_level': 'unknown'}

        recent_patterns = [p for p in patterns 
                          if self._is_recent(p['timestamp'])]

        # these methods are separate part of implementation:
        # self._calculate_risk_level
        # self._detect_anomalies
        # self._summarize_usage
        return {
            'risk_level': self._calculate_risk_level(recent_patterns),
            'unusual_activity': self._detect_anomalies(recent_patterns),
            'usage_summary': self._summarize_usage(recent_patterns)
        }

These systems work together to provide several critical security functions:

Prevention of abuse through multi-window rate limiting
Detection of unusual usage patterns that might indicate compromised keys
Collection of usage metrics for billing and capacity planning
Early warning system for potential security incidents

Finally we may create adaptive rate limiting based on observed patterns:

def adjust_limits(self, api_key: str, usage_analysis: dict):
    # get data fromt the previous method
    usage_analysis = self.analyze_usage_pattern(api_key)
    risk_level = usage_analysis['risk_level']
    normal_usage = usage_analysis['usage_summary']['average_daily_requests']

    if risk_level == 'high':
        # Implement stricter limits for high-risk usage patterns
        self.limits['requests_per_minute'] = min(
            self.limits['requests_per_minute'],
            int(normal_usage / (24 * 60) * 1.5)  # 50% above normal
        )
    elif risk_level == 'low':
        # Gradually relax limits for trusted keys
        self.limits['requests_per_minute'] = int(
            self.limits['requests_per_minute'] * 1.1  # 10% increase
        )

This approach to rate limiting and usage tracking provides both immediate protection against abuse and long-term insights into API usage patterns.

When to Use API Keys

The decision to implement API key authentication should be based on a careful analysis of system requirements, security needs, and usage patterns. Let's examine the primary scenarios where API keys excel as an authentication mechanism.

Public APIs with Moderate Security Requirements

In the context of public APIs, such as weather data services or public reference databases, API keys serve as an ideal authentication method. For instance, consider a public mapping service:

def get_location_data(api_key: str, coordinates: tuple) -> dict:
    headers = {'X-API-Key': api_key}
    return requests.get(
        f"https://maps.example.com/v1/location",
        headers=headers,
        params={'lat': coordinates[0], 'lng': coordinates[1]}
    ).json()

This approach works well because it balances accessibility with basic security and usage tracking. The data, while valuable, doesn't contain sensitive information that would require more robust authentication methods.

Developer-Focused Services

When building services primarily used by developers, API keys provide a straightforward integration path. Consider a code analysis service:

class CodeAnalysisAPI:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.base_url = "https://analysis.example.com/v1"

    def analyze_repository(self, repo_url: str) -> dict:
        response = requests.post(
            f"{self.base_url}/analyze",
            headers={'Authorization': f"Bearer {self.api_key}"},
            json={'repository': repo_url}
        )
        return response.json()

Developers appreciate this approach because it requires minimal setup and integrates easily with common development tools and workflows.

Services Requiring Usage Tracking

When business requirements demand precise usage monitoring, API keys excel. Consider a content delivery network:

class ContentDelivery:
    def __init__(self):
        self.usage_tracker = UsageTrackingSystem()

    def serve_content(self, api_key: str, content_id: str) -> Response:
        # Validate key and record usage
        if not self.usage_tracker.check_quota(api_key):
            return Response("Quota exceeded", status=429)

        # Track bandwidth consumption
        content = self.get_content(content_id)
        self.usage_tracker.record_bandwidth(
            api_key,
            len(content),
            content_type=content.type
        )

        return Response(content)

This system allows for precise tracking of resource consumption, enabling accurate billing and capacity planning.

Internal Microservices

Within secure networks, API keys provide an efficient service-to-service authentication mechanism. Consider a microservice architecture:

class InternalServiceClient:
    def __init__(self, service_name: str, api_key: str):
        self.service_name = service_name
        self.api_key = api_key

    def request_internal_data(self, endpoint: str) -> dict:
        headers = {
            'X-Service-Name': self.service_name,
            'X-API-Key': self.api_key
        }

        return requests.get(
            f"http://internal-service/{endpoint}",
            headers=headers
        ).json()

In this context, API keys provide a lightweight yet effective way to maintain service boundaries and track inter-service communications.

However, there are scenarios where API keys might not be the best choice:

High-security environments requiring user-specific actions
Systems handling sensitive personal data
Financial services requiring transaction-level authentication
Applications needing fine-grained user permissions

In these cases, more robust authentication methods like OAuth 2.0 or JWT-based systems might be more appropriate. The key is to match the authentication method to the specific security and operational requirements of your system.

This comprehensive understanding of when to use API keys helps ensure we implement the right authentication mechanism for our specific use case, balancing security needs with operational efficiency.

Conclusion

API Keys represent a significant evolution in API authentication, offering a balance of security and simplicity that makes them ideal for many modern use cases. While they may not provide the robust security features of OAuth 2.0 or JWT tokens, their straightforward implementation and management make them an excellent choice for many applications.

The key to successful API Key implementation lies in proper key management, secure transmission, and comprehensive monitoring. By following the best practices outlined in this guide and implementing appropriate security measures, developers can leverage API Keys to build secure and scalable API authentication systems.

As we continue our journey through API authentication methods, API Keys serve as a bridge between the simplicity of Basic Authentication and the complexity of token-based systems like OAuth 2.0, which we'll explore in our next article in this series.

API Authentication: Part I. Basic Authentication

Eugene Zimin — Mon, 02 Dec 2024 04:10:15 +0000

Introduction

API authentication serves as the fundamental gatekeeper in the inter service communication, establishing a mechanism for identifying and verifying the clients attempting to interact with our systems. At its core, authentication acts as a digital handshake between client and server, ensuring that only legitimate requests gain access to protected resources.

In distributed systems architecture, this process involves multiple layers of verification. When a client - whether a mobile application, web service, or another system - attempts to access an API endpoint, it must first prove its identity. This proof typically comes in the form of credentials, tokens, or cryptographic signatures. Much like a passport serves as a trusted form of identification in international travel, these digital credentials provide a way to verify the authenticity of incoming requests.

The story of API authentication begins in the early days of web services, when the primary concern was simply identifying the caller, who often was a user. The initial mechanisms were straightforward - a simple username and password combination transmitted with each request. As web applications grew more complex and interconnected, these basic authentication methods proved insufficient. Client applications, third-party services, and automated systems began consuming APIs alongside human users, each requiring different security considerations.

The rise of service-oriented architectures in the late 1990s and early 2000s introduced new challenges. APIs needed to handle machine-to-machine communication, delegate access rights, and manage varying levels of permissions - all while maintaining security and performance.

The evolution reflected a fundamental shift in how systems interact with each other. No longer was it sufficient to simply verify a username and password - systems needed to establish trust, manage sessions, handle token expiration, and implement multiple layers of security. This transformation mirrors the broader evolution of the internet from a simple information-sharing platform to a complex ecosystem of interconnected services.

Authentication vs. Authorization

Before we begin, we need to differentiate and understand the difference between authentication and authorization. In scientific terms, authentication and authorization represent distinct but complementary security processes that work together to create a comprehensive security model.

Authentication answers the question: "Who is making this request?" Think of it like airport security screening. When passing through security, travelers must present valid identification (passport or ID) to prove they are who they claim to be. Similarly, API requests must present valid credentials to prove their identity. Just as a passport contains specific security features that can be validated, API credentials contain unique identifiers and cryptographic elements that verify their authenticity.

Authorization, on the other hand, addresses the question: "What is this authenticated entity allowed to do?" Continuing with our airport analogy, once a traveler's identity is confirmed through their passport (authentication), their boarding pass determines which flight they can board and which areas of the airport they can access (authorization). A first-class ticket might grant access to exclusive lounges, while a standard ticket restricts access to general waiting areas.

In the context of API security, this distinction becomes critical to keep in mind. Consider a banking system: when logging into a banking application, providing correct username and password authenticates the user (proves identity), but the bank's authorization system determines which accounts they can view or what types of transactions they can perform. A customer service representative might be authenticated into the system but authorized only to view customer information, while a financial advisor might be authorized to perform trades on behalf of clients.

Basic Authentication: The Core

Basic Authentication represents one of the foundational authentication methods in web services. Despite its simplicity, understanding its mechanics provides crucial insights into authentication principles. Born in the early days of web protocols, Basic Authentication introduces core concepts that persist in modern authentication systems.

Figure 1. Basic Authentication Flow

At its heart, Basic Authentication operates on a straightforward premise - each request carries credentials in its header. These credentials consist of a username and password pair, combined and encoded using a base64 algorithm with the prefix word Basic before encoded string. Below is the example how to encode the username and password into the base64 string using command line interface.

$ echo -n 'username:password' | base64
dXNlcm5hbWU6cGFzc3dvcmQ=

To create a Basic Authentication token we need to get that string and simply add the word Basic before:

# This is the client code, which may be another service or browser
import base64

def create_basic_auth_header(username, password):
    # Combine username and password with a colon
    credentials = f"{username}:{password}"

    # Convert the string to bytes, encode in base64, and decode back to string
    encoded_credentials = base64.b64encode(credentials.encode('utf-8')).decode('utf-8')

    # Create the authorization header
    return f"Basic {encoded_credentials}"

# Example implementation
auth_header = create_basic_auth_header("researcher", "secure_pwd123")
print(f"Generated header: {auth_header}")

# Output example:
# Generated header: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

This generated Basic Authentication token in form of the string Basic dXNlcm5hbWU6cGFzc3dvcmQ= should be injected by client application for every request made towards the secured API endpoint. When examining network traffic, such a header should appear in every request, made by client application (web browser or another service):

GET /api/resource HTTP/1.1
Host: api.example.com
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

The server's role in this exchange involves reversing the encoding process and validating the credentials. This creates a simple yet complete authentication cycle:

# This is server code, which validates credentials on the server
def validate_basic_auth(auth_header):
    try:
        # check the authentication approach
        auth_tokens = auth_header.split(' ')

        # return False if the approach is not Basic Auth
        if ('Basic' != auth_tokens[0])
            return False

        # Decode from base64
        decoded_bytes = base64.b64decode(auth_tokens[1])

        # Convert to string and split credentials
        decoded_str = decoded_bytes.decode('utf-8')
        username, password = decoded_str.split(':')

        # In practice, compare against securely stored credentials in Database
        return verify_credentials_in_DB(username, password)

    except Exception:
        return False

Each request follows this validation cycle, establishing a stateless authentication pattern.

Base64 Encoding

When examining Basic Authentication's encoding mechanism, understanding why base64 is chosen over other encoding methods reveals fascinating insights into network protocol design. Let's take a look at it through both theoretical explanation and practical demonstrations.

import base64

# Let's examine what happens with special characters in credentials
def demonstrate_base64_necessity(username, password):
    # Original credentials with special characters
    raw_credentials = f"{username}:{password}"
    print(f"Original string: {raw_credentials}")

    # Show raw bytes representation
    raw_bytes = raw_credentials.encode('utf-8')
    print(f"Raw bytes: {raw_bytes}")

    # Show base64 encoded version
    encoded = base64.b64encode(raw_bytes).decode('utf-8')
    print(f"Base64 encoded: {encoded}")

    return encoded

# Example with special characters
result = demonstrate_base64_necessity("research@lab.com", "p@ssw:rd!123")

Considering we provide the following data for input research@lab.com as the username and p@ssw:rd!123 as the password, we should expect the following output:

# Original string: research@lab.com:p@ssw:rd!123
# Raw bytes: b'research@lab.com:p@ssw:rd!123'
# Base64 encoded: cmVzZWFyY2hAbGFiLmNvbTpwQHNzdzpyZCExMjM=

Base64 encoding serves several critical purposes in Basic Authentication.

First, HTTP headers must comply with ASCII character restrictions. Consider credentials containing non-ASCII characters or special symbols that might interfere with HTTP header parsing. Base64 encoding solves this by converting any binary data into a subset of ASCII characters (A-Z, a-z, 0-9, +, /, and =).

Let's demonstrate this with a more complex example:

def analyze_character_safety(credentials):
    # Original string might contain problematic characters
    print(f"Original: {credentials}")

    # Show problematic characters in HTTP headers
    problematic = [c for c in credentials if not (32 <= ord(c) <= 126)]
    if problematic:
        print(f"Problematic characters: {problematic}")

    # Encode to base64
    encoded = base64.b64encode(credentials.encode('utf-8')).decode('utf-8')
    print(f"Safe base64: {encoded}")

    # Verify all characters are HTTP-safe
    safe_chars = all(32 <= ord(c) <= 126 for c in encoded)
    print(f"All characters HTTP-safe: {safe_chars}")

# Example with various challenging characters
analyze_character_safety("user:パスワード")  # Japanese characters
analyze_character_safety("user:password\n\r")  # Control characters

Again, using user:パスワード" in first case and user:password\n\r in the second case we would expect to get the following output:

# Original: user:パスワード
# Problematic characters: ['パ', 'ス', 'ワ', 'ー', 'ド']
# Safe base64: dXNlcjrjg5Hjgrnjg/rjg7zjg4k=
# All characters HTTP-safe: True

Another aspect is the colon character : used as a delimiter between username and password. Without encoding, this would create ambiguity in parsing the credentials. Base64 encoding ensures the colon in the original credentials doesn't interfere with the username-password delimiter:

def demonstrate_colon_handling():
    # A username containing a colon would be problematic
    problematic_username = "research:team"
    password = "secure123"

    raw = f"{problematic_username}:{password}"
    print(f"Ambiguous raw format: {raw}")

demonstrate_colon_handling()

Without encoding, this would be ambiguous and we won't know which colon is the delimiter?:

# Ambiguous raw format: research:team:secure123

To solve that we need to add the following code at the end of the demonstrate_colon_handling method:

    # Base64 encoding resolves this ambiguity
    encoded = base64.b64encode(raw.encode('utf-8')).decode('utf-8')
    print(f"Unambiguous encoded format: {encoded}")

    # Demonstrate successful decoding
    decoded = base64.b64decode(encoded).decode('utf-8')
    print(f"Decoded back: {decoded}")

As the result now we should expect the following:

# Ambiguous raw format: research:team:secure123
# Unambiguous encoded format: cmVzZWFyY2g6dGVhbTpzZWN1cmUxMjM=
# Decoded back: research:team:secure123

Base64 encoding also provides a minor level of obfuscation. While it's important to note that this is not encryption and offers no real security, it prevents casual observers from immediately reading credentials in network traces.

It is clear that base64 encoding in Basic Authentication isn't about security but about ensuring reliable transport of credential information across HTTP protocols. It's a elegant solution to the challenge of sending potentially complex credential data within the constraints of HTTP headers.

Security Considerations

While Basic Authentication provides a straightforward implementation, its security profile requires careful consideration, as Transport Layer Security (TLS) becomes critical when using Basic Authentication. Consider the following example:

import requests
from urllib.parse import urlparse

def create_basic_auth_credentials():
    url = "http://api.example.com/data"
    auth_header = construct_basic_auth_header("researcher", "password123")

    # Examining the request details
    parsed_url = urlparse(url)
    if parsed_url.scheme != "https":
        print("Credentials will be transmitted in encoded but unencrypted form")
        print(f"Raw auth header visible in transit: {auth_header}")

If we run the following code with the HTTP schema (not HTTPS) we will see the following:

# Unencrypted HTTP request (vulnerable)
GET /api/data HTTP/1.1
Host: api.example.com
Authorization: Basic cmVzZWFyY2hlcjpwYXNzd29yZDEyMw==

and if we run it over HTTPS connection - we will get encrypted data:

# HTTPS encrypted request (secure)
[encrypted data stream]

HTTP Secured connection is important here because it provides encryption on transport layer and prevents MITM attacks.

A man-in-the-middle attack occurs when a malicious actor positions themselves between the client and server during communication. Think of it like intercepting a postal mail - the attacker can read, modify, or even replace the contents before forwarding them to the intended recipient. In Basic Authentication, since credentials are merely encoded in base64 (not encrypted), an intercepted request reveals the username and password as clearly as reading an open letter.

What makes MITM particularly dangerous for Basic Authentication? Every request to the server includes these encoded credentials. This repeated transmission creates multiple opportunities for credential theft. Once intercepted, these credentials remain valid until explicitly changed, as Basic Authentication lacks built-in expiration or revocation mechanisms. The attacker can reuse the stolen credentials indefinitely, potentially accessing sensitive data or performing unauthorized operations.

The situation changes dramatically when HTTPS enters the picture. HTTPS establishes a secure encrypted tunnel between client and server through Transport Layer Security (TLS). Continuing our postal analogy, HTTPS is like sending a letter in a secure diplomatic pouch that can only be opened by the intended recipient. Even if intercepted, the encrypted contents remain unreadable to the attacker.

When to use Basic Authentication

Despite its limitations, Basic Authentication remains relevant in specific scenarios where its simplicity and ease of implementation outweigh its constraints. Understanding these use cases helps make informed decisions about authentication strategies.

Development and Testing Environments

During the development phase, teams often need quick, straightforward authentication mechanisms to test API functionality. Basic Authentication serves this purpose effectively, allowing developers to focus on core functionality without the complexity of more sophisticated authentication systems.

In development environments, this approach facilitates rapid iteration and testing. However, it's important to ensure these simplified credentials never make their way into production systems. Development configurations should remain strictly separated from production environments and never stored in Git. When building and testing applications, secrets such as credentials, API keys, and tokens should be managed through environment variables or dedicated secrets management systems.

from os import environ
from dotenv import load_dotenv

class Config:
    def __init__(self):
        # Load environment variables from .env for development
        load_dotenv()

        self.basic_auth = {
            'username': environ.get('AUTH_USERNAME'),
            'password': environ.get('AUTH_PASSWORD')
        }

Development teams should maintain:

A .env.example file in version control showing the required variables without actual values
A .gitignore file that explicitly excludes .env files and other secret-containing configurations
Separate configuration management for each environment (development, staging, production)
Documentation explaining how to securely obtain and configure credentials

For local development, create a .env file that's never committed:

# .env
AUTH_USERNAME=dev_user
AUTH_PASSWORD=dev_password

Internal Network Communications

Within controlled, internal networks, Basic Authentication can provide adequate security when combined with proper transport layer protection. This is particularly relevant for microservices architectures where services communicate within a trusted network boundary. In such environments, services often need to authenticate each other rapidly and efficiently which makes Basic Authentication is the preferable approach. The controlled nature of internal networks, combined with additional security measures like network segmentation, firewalls, and intrusion detection systems, creates multiple layers of protection. Organizations commonly implement Basic Authentication for service-to-service communication in scenarios where the network perimeter is well-defined and access is strictly controlled through VPNs or private networks.

def internal_service_call():
    internal_url = "https://internal-api.local/data"
    response = requests.get(
        internal_url,
        auth=('service_account', 'internal_token'),
        verify=True  # Still enforce SSL certificate verification
    )

Command-Line Tools and Scripts

Basic Authentication proves valuable for command-line interfaces and automation scripts where simplicity and direct implementation are priorities. These tools often operate in controlled environments where the additional complexity of token-based authentication might be unnecessary. When building automation tools for internal use, developers need quick, reliable authentication methods that can be easily implemented across different programming languages and platforms. The simplicity of implementation allows teams to focus on core functionality while maintaining adequate security through proper credential management and secure communication channels.

Monitoring and Health Check Endpoints

For simple monitoring endpoints that provide basic system health information, Basic Authentication offers a straightforward way to prevent unauthorized access while maintaining easy integration with monitoring tools. Monitoring systems require consistent, reliable access to health check endpoints while preventing unauthorized access to potentially sensitive system information. The stateless nature of Basic Authentication makes it particularly suitable for these endpoints, as monitoring tools can easily include credentials in their requests without managing complex token lifecycles or session states. System administrators can quickly configure monitoring tools with Basic Authentication credentials, enabling automated health checks and alerts while maintaining a basic security barrier against unauthorized access attempts. The simplicity of Basic Authentication also facilitates easy integration with various monitoring platforms and tools, making it a practical choice for health check endpoints in both development and production environments.

Conclusion

Basic Authentication, despite its age and simplicity, continues to serve as a foundational element in API security architecture. The security considerations highlight the critical importance of implementing Basic Authentication exclusively over HTTPS connections. Without proper transport layer security, the base64 encoding offers no real protection against man-in-the-middle attacks, potentially exposing sensitive credentials to malicious actors. This vulnerability underscores why Basic Authentication should be deployed thoughtfully and in appropriate contexts.

Basic Authentication remains valuable when matched with the right requirements and security context. In development environments, it facilitates rapid testing and iteration. Within internal networks, it enables efficient service-to-service communication. For command-line tools and monitoring systems, it provides a straightforward authentication mechanism that balances security with simplicity.

However, it's important to recognize that Basic Authentication is just one tool in the modern API security toolkit. While it excels in specific scenarios, more complex applications often require more sophisticated authentication mechanisms like OAuth 2.0 or JWT tokens. The key to successful implementation lies in understanding these limitations and choosing Basic Authentication only when its simplicity aligns with the security requirements and operational context of your system.

As we continue to build and secure modern APIs, Basic Authentication serves as a reminder that sometimes the simplest solution, when properly implemented and appropriately deployed, can be the right choice. Its longevity in the field of web security testifies to the enduring value of straightforward, well-understood security mechanisms in specific contexts.

Debugging SSH connections: A Comprehensive Guide

Eugene Zimin — Tue, 26 Nov 2024 21:35:26 +0000

SSH (Secure Shell) is the backbone of remote system administration and secure remote access, serving millions of developers and system administrators daily. However, when SSH connections fail, the cryptographic nature of the protocol can make debugging challenging. The complex interplay between authentication mechanisms, encryption algorithms, and network layers often obscures the root cause of connection issues. This complexity is further compounded by the protocol's security-first design, where error messages are intentionally vague to prevent potential attackers from gathering system information. Whether we're dealing with key authentication failures, network connectivity issues, or configuration mismatches, understanding the underlying SSH architecture becomes critical for effective troubleshooting.

Understanding SSH Connection Process

Before diving into debugging, it's important to understand how SSH establishes connections. The process follows these steps:

TCP connection establishment (Port 22 by default)
Protocol version exchange
Key exchange and server authentication
Client authentication
Session establishment

Detailed process is highlighted in another article: Configuring SSH to Access Remote Server dedicated to overview this process in a deeper level.

Each step can fail for different reasons, producing various error messages. Understanding these steps helps pinpoint where issues occur.

Common SSH Errors and Solutions

When working with SSH connections, we frequently encounter various error messages that might seem cryptic at first glance. While SSH's error reporting is intentionally terse for security reasons, each error message provides valuable clues about the underlying issue. Understanding these common errors and their potential solutions not only helps us resolve issues faster but also deepens our knowledge of SSH's security model and connection process. Many of these errors stem from incorrect configurations, permission issues, or network problems, and they often appear during system upgrades, after server migrations, or when setting up new environments. Let's explore the most frequent SSH connection issues and their systematic solutions.

Connection Refused: Understanding Server Accessibility

One of the most common SSH errors we encounter during daily operations looks deceptively simple:

ssh: connect to host example.com port 22: Connection refused

This error message indicates that our SSH server is completely unreachable, though the underlying cause requires careful investigation. Most frequently, we discover that the SSH daemon (sshd) has stopped running on the target server. This typically happens after system updates, configuration changes, or occasional service crashes. A quick check of the service status often reveals the problem:

$ sudo systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
     Active: inactive (dead) since Mon 2024-01-15 09:23:45 UTC

Pay attention that the server is shown as inactive (dead) and usually it should be started with the following command:

$ sudo systemctl start sshd

Network connectivity issues present another common culprit. Corporate or cloud provider firewalls might be blocking the default SSH port 22, requiring us to verify firewall rules. In cloud environments like AWS or GCP, this often means checking both the instance's security group and network ACLs:

$ sudo iptables -L | grep ssh
# No output indicates potential firewall blocking

Sometimes the issue stems from DNS resolution problems or incorrect IP addressing. We can validate basic network connectivity using standard networking tools:

$ ping example.com
ping: example.com: Name or service not known

$ telnet example.com 22
telnet: Unable to connect to remote host: Connection refused

If all these checks pass but the connection still fails, the server itself might be experiencing issues, possibly due to resource exhaustion or hardware problems. In such cases, server logs become our primary diagnostic tool:

$ sudo tail -f /var/log/syslog
Jan 26 10:15:32 server kernel: Out of memory: Kill process 1234 (sshd) score 28 or sacrifice child

Permission Denied

One of the most frustrating SSH errors occurs during the authentication phase, presenting itself as a seemingly simple message:

Permission denied (publickey,password)

This deceptively brief message indicates an authentication failure, though its resolution often requires careful investigation. The error message's format actually provides our first clue - the terms "publickey" and "password" in parentheses tell us which authentication methods the server attempted before denying access.

When troubleshooting this error, we often find that the username doesn't match the remote system's records. For example, while connecting to an Ubuntu server, we might mistakenly use:

$ ssh admin@ubuntu-server
Permission denied (publickey,password)

When the correct username should be 'ubuntu':

$ ssh ubuntu@ubuntu-server
Welcome to Ubuntu 22.04.2 LTS...

Private key mismatches represent another common scenario. The SSH server maintains a strict relationship between private and public key pairs, and even a slight mismatch will trigger this error. We can investigate key-related issues by enabling verbose output:

$ ssh -v ubuntu@ubuntu-server
...
debug1: Trying private key: /home/user/.ssh/id_rsa
debug1: Trying private key: /home/user/.ssh/id_ed25519
debug1: No more authentication methods to try.

This verbose output shows SSH attempting to use each private key file it finds. The sequence shows that SSH couldn't successfully authenticate with any available key file. When you see 'No more authentication methods to try', it means SSH has exhausted all configured authentication methods (in this case, trying both RSA and ED25519 keys) without success.

A particularly tricky variant occurs when all credentials appear correct, but file permissions are preventing SSH from accepting the keys. SSH enforces strict permission requirements for security reasons. We commonly see this when copying key files between systems:

$ ls -la ~/.ssh/id_ed25519
-rw-rw-r-- 1 user user 464 Nov 26 10:15 /home/user/.ssh/id_ed25519

$ ssh ubuntu@ubuntu-server
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0664 for '/home/user/.ssh/id_ed25519' are too open.

The solution requires setting appropriate permissions:

$ chmod 600 ~/.ssh/id_ed25519
$ ssh ubuntu@ubuntu-server
Welcome to Ubuntu 22.04.2 LTS...

For those cases where the server has our public key but still denies access, examining the server's auth log often reveals the underlying issue:

$ sudo tail -f /var/log/auth.log
Nov 26 10:15:32 ubuntu-server sshd[12345]: Authentication refused: bad ownership or modes for directory /home/ubuntu/.ssh

Host Key Verification Failed

During routine SSH operations, we occasionally encounter an alarming message that stops us in our tracks:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:abcdef1234567890abcdef1234567890.
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/user/.ssh/known_hosts:3

This error, while alarming, serves as a critical security feature in SSH's trust model. When we first connect to a server, SSH stores its unique fingerprint in our known_hosts file. Any subsequent changes to this fingerprint trigger this warning, protecting us from potential security breaches.

In cloud environments, this error frequently occurs after server rebuilds. For instance, when working with AWS EC2 instances:

$ ssh ubuntu@ec2-12-34-56-78.compute-1.amazonaws.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

The instance might have been terminated and recreated, generating a new host key. We can verify this through AWS console or CLI:

$ aws ec2 describe-instance-history --instance-id i-1234567890abcdef0
{
    "InstanceHistoryEvents": [
        {
            "EventType": "instanceStop",
            "EventTime": "2024-01-25T10:00:00Z"
        }
    ]
}

For known legitimate changes, we can remove the old key:

$ ssh-keygen -R ec2-12-34-56-78.compute-1.amazonaws.com
# Host ec2-12-34-56-78.compute-1.amazonaws.com found: line 3
/home/user/.ssh/known_hosts updated.

However, when this error occurs unexpectedly, particularly with production servers, it warrants immediate investigation. We can examine the server's SSH fingerprint directly:

$ ssh-keyscan -t ed25519 hostname
# hostname:22 SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.1
hostname ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI...

For added security, we can compare this with the fingerprint provided by our infrastructure provider or system administrator:

$ ssh-keygen -lf <(ssh-keyscan -t ed25519 hostname 2>/dev/null)
256 SHA256:abcdef1234567890abcdef1234567890 hostname (ED25519)

In cases involving DNS changes, we might see this error when a domain name starts pointing to a different server. A quick DNS lookup can confirm such changes:

$ dig +short hostname
93.184.216.34  # Note if this IP has changed

Slow Connections

While SSH generally provides very responsive connections, we occasionally encounter frustratingly slow sessions that impact us. These performance issues often manifest in various ways: delayed command responses, laggy terminal output, or prolonged initial connection times.

One of the most common culprits involves DNS resolution delays. When establishing an SSH connection, the server attempts to resolve the client's hostname by default. In environments with misconfigured DNS servers or slow network responses, this resolution process can add significant delays:

$ time ssh server.example.com date 
Warning: Reverse DNS lookup failed 
Tue Nov 26 10:15:32 UTC 2024 
real 0m3.245s 
user 0m0.035s 
sys 0m0.012s

The output shows three timing measurements: 'real' indicates the actual elapsed wall-clock time (3.245 seconds), while 'user' and 'sys' show CPU time spent in user and kernel mode respectively. The large difference between real time (3.245s) and CPU time (0.047s total) indicates the connection is spending most of its time waiting for DNS resolution, not processing.

We can significantly improve connection times by disabling DNS lookups in the server configuration:

$ sudo nano /etc/ssh/sshd_config
# Add or modify the following line
UseDNS no

# Restart the SSH service to apply changes
$ sudo systemctl restart sshd

# Test the connection speed again
$ time ssh server.example.com date
Tue Nov 26 10:15:32 UTC 2024
real    0m0.532s
user    0m0.034s
sys     0m0.011s

For connections over high-latency networks or when transferring large amounts of data, enabling SSH compression can yield substantial performance improvements. SSH compression becomes particularly effective when working with text-heavy sessions or transferring compressible data:

$ cat ~/.ssh/config
# Global SSH client configuration
Host *
    # Enable compression for all connections
    Compression yes
    # Use compression level 6 for optimal balance
    CompressionLevel 6

Perhaps one of the most powerful optimizations involves SSH connection multiplexing. Instead of establishing new TCP connections for each SSH session, multiplexing reuses an existing connection, dramatically reducing connection overhead. This becomes especially valuable when working with remote Git repositories or running multiple SSH sessions to the same server:

$ cat ~/.ssh/config
Host *
    # Enable automatic multiplexing
    ControlMaster auto
    # Define the control socket location
    ControlPath ~/.ssh/control/%C
    # Keep the master connection alive for an hour
    ControlPersist 1h
    # Optional: Configure keepalive to prevent timeouts
    ServerAliveInterval 60
    ServerAliveCountMax 3

We can verify multiplexing is working by examining the control socket:

$ ls -la ~/.ssh/control/
total 0
drwx------ 2 user user 100 Nov 26 10:15 .
drwx------ 8 user user 160 Nov 26 10:15 ..
srw------- 1 user user   0 Nov 26 10:15 example.com-22-user

The srw at the start of the last line indicates this is a socket file (s) with read-write permissions (rw). The size showing as 0 is normal for socket files. The filename example.com-22-user follows the format hostname-port-username, indicating an active multiplexed connection for this specific combination.

The presence of the socket file indicates an active multiplexed connection. Subsequent SSH commands to the same host will reuse this connection, resulting in nearly instantaneous session establishment:

$ time ssh server.example.com date
Tue Nov 26 10:15:32 UTC 2024
real    0m0.087s
user    0m0.012s
sys     0m0.008s

For production environments where consistent performance is critical, we might also consider adjusting TCPkeepalive settings to prevent connection drops over problematic networks:

Host production-*
    # More aggressive keepalive for production servers
    TCPKeepAlive yes
    ServerAliveInterval 30
    ServerAliveCountMax 6

Resolving SSH Key-Related Issues

SSH key problems often emerge as some of the most perplexing authentication challenges. Let's explore two critical categories of key-related issues that frequently impact SSH connections.

One of the most common SSH key errors presents itself with an alarming warning message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/home/user/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.

This error reflects SSH's strict security requirements for private key files. Common scenarios leading to this issue include copying keys from another system, extracting them from backups, or creating them with incorrect default permissions. Let's examine a typical scenario:

$ ls -la ~/.ssh/id_rsa
-rw-rw-r-- 1 user user 1876 Nov 26 10:15 /home/user/.ssh/id_rsa

The permissions shown above (664) allow group members to read the private key, creating a security vulnerability. We can resolve this by applying proper permissions:

# Secure the private key file
$ chmod 600 ~/.ssh/id_rsa
$ ls -la ~/.ssh/id_rsa
-rw------- 1 user user 1876 Nov 26 10:15 /home/user/.ssh/id_rsa

# Secure the SSH directory itself
$ chmod 700 ~/.ssh
$ ls -la ~ | grep .ssh
drwx------ 2 user user 4096 Nov 26 10:15 .ssh

Another subtle but frustrating issue occurs when SSH refuses to read seemingly valid key files:

Load key "/home/user/.ssh/id_rsa": invalid format

This error often surfaces after migrating keys between different SSH implementations or when working with keys generated by third-party tools. Let's investigate a problematic key:

$ ssh-keygen -l -f ~/.ssh/id_rsa
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: INVALID KEY FILE FORMAT DETECTED!           @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

The -l flag attempts to show the key's fingerprint and bit length. This error indicates the key file exists but isn't in a format SSH can parse. This often happens when the key file has been corrupted during transfer or when it's been modified by a text editor that changed line endings or character encoding.

The error might occur because the key is in a modern OpenSSH format while connecting to an older server, or vice versa. We can examine the key's content (being careful not to expose private key material):

$ head -n 1 ~/.ssh/id_rsa
-----BEGIN OPENSSH PRIVATE KEY-----

If we see a different header format or unexpected content, we might need to convert the key to a compatible format. The PEM format offers the widest compatibility:

# Backup the original key first
$ cp ~/.ssh/id_rsa ~/.ssh/id_rsa.backup

# Convert the key to PEM format
$ ssh-keygen -p -f ~/.ssh/id_rsa -m PEM
Key has comment 'user@hostname'
Enter new passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved with the new passphrase.

# Verify the key format
$ head -n 1 ~/.ssh/id_rsa
-----BEGIN RSA PRIVATE KEY-----

For keys that appear completely unreadable, we might need to check their encoding:

$ file ~/.ssh/id_rsa
/home/user/.ssh/id_rsa: PEM RSA private key

Sometimes, keys might become corrupted during transfer, especially when copying between Windows and Unix systems. In such cases, checking for hidden characters or incorrect line endings can help:

$ dos2unix ~/.ssh/id_rsa
dos2unix: converting file /home/user/.ssh/id_rsa to Unix format...

Advanced Debugging Techniques

When standard troubleshooting steps fall short, we often need to delve deeper into SSH's internal workings to identify and resolve complex connection issues. SSH provides sophisticated debugging capabilities that, while potentially overwhelming at first glance, offer invaluable insights into the connection process. These advanced techniques become particularly necessary when dealing with enterprise environments, complex network configurations, or when standard error messages prove insufficient for diagnosis.

During production incidents or when supporting mission-critical systems, these debugging approaches help us understand the intricate dance between client and server configurations, network interactions, and authentication mechanisms. Let's explore the advanced tools and techniques that experienced system administrators rely on for resolving challenging SSH connection problems.

Verbose Logging

The very first step which should be done - enable excessive logging. SSH's logging capabilities represent one of our most powerful diagnostic tools, offering three distinct levels of detail (-v, -vv, -vvv). Each level peels back another layer of the connection process:

# Basic debugging output
$ ssh -v user@hostname
OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to hostname port 22 [192.168.1.100]

# More detailed protocol debugging
$ ssh -vv user@hostname
debug2: resolving "hostname" port 22
debug2: ssh_connect_direct: needpriv 0
debug2: fd 3 setting O_NONBLOCK

# Maximum verbosity for complex issues
$ ssh -vvv user@hostname
debug3: send packet: type 5
debug3: receive packet: type 6
debug3: rekey after 134217728 bytes, 3600 seconds

Server-Side Logging

Understanding what's happening on SSH server gives better vision of root cause of the problems, as well as security and troubleshooting. By enabling detailed logging, we can monitor authentication attempts, track user sessions, and investigate potential security incidents with precision.

To unlock the full potential of SSH logging, at first we need to modify SSH daemon configuration. Open /etc/ssh/sshd_config and set the logging level to its most verbose setting:

LogLevel DEBUG3

SSH activity can be monitored in real-time through system logs. The log location varies by Linux distribution:

# For Debian-based systems (Ubuntu, Debian)
sudo tail -f /var/log/auth.log

# For Red Hat-based systems (RHEL, CentOS, Fedora)
sudo tail -f /var/log/secure

The logs contain detailed information about SSH connections. Here's an example of a successful login with its associated IP address:

Apr 15 14:23:21 server sshd[12345]: Accepted publickey for alice from 192.168.1.100 port 52413

Failed authentication attempts are also recorded, providing valuable security insights:

Apr 15 14:25:33 server sshd[12346]: Failed password for invalid user admin from 203.0.113.1 port 59632 ssh2

Log rotation helps manage the increased volume of data from DEBUG3 level logging. This can be configured in /etc/logrotate.d/sshd to maintain disk space while preserving historical data.

Note that verbose logging creates additional system overhead. In high-traffic production environments, consider reducing the log level after completing specific monitoring or investigation tasks.

Testing Connectivity

Before diving into complex SSH issues, establishing basic connectivity helps narrow down potential problems. Let's start by examining network paths and connections.

The netcat utility provides a straightforward way to verify if the SSH port accepts connections:

nc -zv hostname 22

The -z flag tells netcat to scan for listening daemons without sending data, while -v enables verbose output. A successful response looks like 'Connection to hostname port 22 succeeded!', while a failure might show 'Connection refused' or 'Connection timed out'. This test confirms basic TCP connectivity without attempting SSH authentication.

When connection issues arise, tracing the network path often reveals routing problems or blocked ports:

traceroute hostname

DNS resolution problems can manifest as connection failures, so checking name resolution adds another layer of verification:

dig hostname

Moving beyond basic connectivity, validating SSH configuration prevents common setup issues. The SSH client includes built-in configuration testing:

ssh -G hostname

This command displays the exact configuration that will be used when connecting to the specified host, including inherited defaults and matching Host blocks.

For server-side verification, the SSH daemon offers similar diagnostic capabilities:

sudo sshd -T

This command performs a comprehensive check of the server configuration, displaying the active settings after processing all included files and applying default values. The output helps identify misconfigurations before they impact users.

Best Practices for SSH Troubleshooting

When troubleshooting SSH issues, following a methodical approach leads to faster resolution. Starting with basic connectivity checks establishes a foundation for further investigation. Moving through permission verification and authentication methods helps isolate problems systematically. System logs and verbose SSH output often reveal the root cause of connection issues.

Maintaining clear documentation strengthens our troubleshooting capabilities. Recording configuration changes, preserving working configurations, and keeping configuration backups creates a reliable reference point when issues arise. This documentation becomes particularly valuable when dealing with complex multi-server environments.

During troubleshooting, maintaining security remains paramount. Avoiding temporary security bypasses prevents accidental exposure. Host key changes warrant careful verification, and proper file permissions must be maintained throughout the debugging process.

Conclusion

SSH debugging requires a methodical approach and understanding of both the protocol and common failure points. By following this guide, you can efficiently diagnose and resolve SSH connection issues while maintaining security. Remember that SSH's complexity is a feature, not a bug – it's designed to be secure first and convenient second.

Future maintenance of SSH connections can be simplified by implementing proper monitoring, maintaining documentation, and following security best practices. When issues do arise, a systematic debugging approach will help resolve them quickly and effectively.

Understanding SSH Key Pairs: A Developer's Guide

Eugene Zimin — Tue, 26 Nov 2024 07:22:45 +0000

In today's interconnected development world, secure authentication is not just a luxury—it's a necessity. Whether you're a seasoned DevOps engineer or a junior developer just starting your journey, understanding SSH key pairs is crucial for your daily workflow. They're the unsung heroes that keep our git pushes secure, our server access protected, and our deployments safe from prying eyes.

But let's be honest: SSH keys can be confusing. With terms like "public key infrastructure," "cryptographic algorithms," and "key fingerprints" floating around, it's easy to feel overwhelmed. This guide aims to demystify SSH key pairs, breaking down complex concepts into digestible pieces that will help you make informed decisions about your security setup.

What Are SSH Key Pairs?

SSH key pairs are cryptographic credentials consisting of two parts:

A private key that stays on your local machine (keep this secret!)
A public key that you can share freely

Think of them as a sophisticated lock and key system. The public key is like a special lock you can give to anyone, while the private key is the unique key that opens only your locks. When you connect to a remote system, it checks if your private key matches the public key it has stored—if they match, you're granted access. This eliminates the need for password-based authentication, which can be vulnerable to brute force attacks and keyloggers.

What makes SSH key pairs particularly powerful is their ability to provide secure authentication without transmitting sensitive information over the network. Your private key never leaves your machine, making it virtually impossible for attackers to intercept your credentials during the authentication process.

Types of SSH Key Pairs and Their Mathematical Foundations

The cryptographic algorithms behind SSH keys represent some of the most elegant applications of number theory and algebraic geometry in modern computing. Let's delve deep into their mathematical foundations and understand how they provide the security we rely on daily.

RSA: The Prime Numbers Guardian

RSA's brilliance lies in the elegant use of prime numbers and modular arithmetic. Let's walk through a simplified but illustrative example of how RSA actually works in practice.

Suppose we want to create a small (insecure, but educational) RSA key:

First, choose two prime numbers:
p = 61 and q = 53
Calculate n (the modulus):
n = p × q = 61 × 53 = 3,233
Calculate the totient φ(n):
φ(n) = (p-1) × (q-1) = 60 × 52 = 3,120
Choose a public exponent e (commonly 65537):
Let's use e = 17 for this example (must be coprime with φ(n))
Calculate the private exponent d:
d = e⁻¹ mod φ(n) = 2,753

This gives us:

Public key: (n=3,233, e=17)
Private key: (n=3,233, d=2,753)

Here's how the encryption works with these numbers:

Message (m) = 123
Encryption: c = m^e mod n
c = 123^17 mod 3,233 = 855

Decryption: m = c^d mod n
m = 855^2,753 mod 3,233 = 123

In real RSA implementations, we use numbers that are typically 2048 or 4096 bits long. To put this in perspective, a 4096-bit number is roughly 1,234 decimal digits long. The security comes from the fact that factoring such large numbers is computationally infeasible with current technology.

Here's how you'd generate such a key in practice:

# Create a 4096-bit RSA key with custom settings
ssh-keygen -t rsa -b 4096 \
  -C "RSA-4096_$(date +%Y%m%d)" \
  -f ~/.ssh/id_rsa_4096 \
  -N "your_secure_passphrase"

Ed25519: Elliptic Curve Elegance

Ed25519 operates on a specialized Edwards curve, defined by the equation:
-x² + y² = 1 - (121665/121666)x²y²

This curve was carefully chosen for several mathematical properties that make it both secure and efficient. Let's break down how a point multiplication works on this curve:

The base point B has coordinates:

x = 15112221349535400772501151409588531511454012693041857206046113283949847762202
y = 46316835694926478169428394003475163141307993866256225615783033603165251855960

A private key is a 256-bit scalar k
The public key is the point k·B (scalar multiplication)

Here's a simplified example of point addition on the curve:

Given two points P1(x₁,y₁) and P2(x₂,y₂):

x₃ = (x₁y₂ + y₁x₂)/(1 + dx₁x₂y₁y₂)
y₃ = (y₁y₂ - ax₁x₂)/(1 - dx₁x₂y₁y₂)

where d = -121665/121666

The actual Ed25519 implementation uses field arithmetic modulo the prime 2²⁵⁵ - 19, chosen for efficient computation on modern 64-bit processors. When you generate an Ed25519 key:

# Generate Ed25519 key with maximum entropy
ssh-keygen -t ed25519 \
  -C "Ed25519_$(hostname)_$(date +%Y%m%d)" \
  -f ~/.ssh/id_ed25519 \
  -N "$(head -c 32 /dev/urandom | base64)"

The real power of Ed25519 comes from its resistance to various implementation attacks:

Signing equation:
R = rB
S = (r + H(R,A,M)a) mod l

where:
r = H(h_b,...,h_2b-1,M)
H = SHA-512
a = private key
A = public key
M = message
l = 2²⁵² + 27742317777372353535851937790883648493

ECDSA: The NIST Curves

ECDSA uses the NIST P-curves, which are defined over prime fields. The P-256 curve is defined by:
y² = x³ - 3x + b

where b = 0x5AC635D8AA3A93E7B3EBBD55769886BC651D06B0CC53B0F63BCE3C3E27D2604B

Let's examine a point multiplication on this curve:

Start with a base point G:

Gx = 0x6B17D1F2E12C4247F8BCE6E563A440F277037D812DEB33A0F4A13945D898C296
Gy = 0x4FE342E2FE1A7F9B8EE7EB4A7C0F9E162BCE33576B315ECECBB6406837BF51F5

For a private key k, the public key Q = kG is computed through repeated point doubling and addition:

Point Addition (P + Q):
s = (y₂ - y₁)/(x₂ - x₁) mod p
x₃ = s² - x₁ - x₂ mod p
y₃ = s(x₁ - x₃) - y₁ mod p

Point Doubling (P + P):
s = (3x₁² + a)/(2y₁) mod p
x₃ = s² - 2x₁ mod p
y₃ = s(x₁ - x₃) - y₁ mod p

Generate an ECDSA key using the P-521 curve:

# Generate ECDSA key with P-521 curve
ssh-keygen -t ecdsa -b 521 \
  -C "ECDSA_P521_$(date +%Y%m%d)" \
  -f ~/.ssh/id_ecdsa_521 \
  -N "$(openssl rand -base64 32)"

Extended Security Considerations

The security of these algorithms depends on different hard mathematical problems:

RSA: Integer Factorization Problem (IFP)

Given n = pq, find p and q
Time complexity: O(exp((log n)^(1/3) * (log log n)^(2/3)))

Ed25519: Elliptic Curve Discrete Logarithm Problem (ECDLP)

Given P and Q = kP, find k
Time complexity: O(√n) using Pollard's rho algorithm

ECDSA: Same as Ed25519, but with different curve parameters

Security level comparison:
RSA 3072-bit ≈ ECDSA/Ed25519 256-bit
RSA 15360-bit ≈ ECDSA/Ed25519 512-bit

Quantum Computing Impact

The advent of quantum computers poses different threats to these algorithms. Using Shor's algorithm:

RSA factoring time complexity:
Classical: O(exp((log N)^(1/3) * (log log N)^(2/3)))
Quantum: O((log N)^3)

ECDLP solving time complexity:
Classical: O(√n)
Quantum: O((log n)^3)

This is why post-quantum cryptography is becoming increasingly important, though it's not yet implemented in standard SSH keys.

Making an Informed Choice

The choice between these algorithms goes beyond mathematics—it's about balancing security, compatibility, and performance. Ed25519 represents the future: mathematically elegant, computationally efficient, and designed with modern threats in mind. Its implementation provides consistent security properties across different platforms, making it the ideal choice for new deployments.

For systems requiring broad compatibility, RSA with 4096 bits remains a solid choice. Its mathematical foundation has withstood decades of cryptanalysis, and while it may be computationally more intensive than modern elliptic curve approaches, its security margins are well understood.

When implementing any of these algorithms, the key is to ensure proper entropy during key generation. A strong random number generator is crucial for security, as even the most mathematically secure algorithm can be compromised by poor randomness. Modern systems use hardware random number generators and entropy pools to ensure strong key generation, but it's worth being aware of this critical foundation.

SSH Config File - Forgotten Gem

Eugene Zimin — Tue, 26 Nov 2024 07:04:16 +0000

For developers and system administrators managing multiple remote servers, the conventional approach of typing lengthy SSH commands such as those incorporating identity files, usernames, and complex domain names presents a significant operational burden. Consider the following typical command structure:

ssh -i ~/.ssh/special_key.pem username@ec2-123-45-67-89.compute-1.amazonaws.com

Such verbose commands, while explicit in their intent, can be transformed into more elegant alternatives through proper configuration. The SSH config file enables concise commands like ssh staging or ssh prod, reducing cognitive load and potential typing errors. This often overlooked tool enhances SSH workflow efficiency substantially, while maintaining the robust security measures inherent in SSH protocols.

Understanding the SSH Config File

The SSH config file, generally located at ~/.ssh/config, serves as a sophisticated configuration repository for SSH connections, embodying the Unix philosophy of maintaining simple, text-based configuration files. It functions as a sophisticated bookmark system with extensive customization capabilities, allowing for the definition of aliases and default settings. This approach to configuration management reflects the fundamental principles of systems administration: clarity, maintainability, and scalability.

Basic Configuration Implementation

Consider this foundational example of an SSH config file (~/.ssh/config), which demonstrates the essential elements of host configuration. Each section defines a specific connection profile, encapsulating all necessary parameters for establishing secure connections:

Host github
    HostName github.com
    User git
    IdentityFile ~/.ssh/github_key

Host staging
    HostName ec2-123-45-67-89.compute-1.amazonaws.com
    User ubuntu
    IdentityFile ~/.ssh/staging.pem
    Port 22

This configuration reduces verbose SSH commands to simplified versions, demonstrating the principle of abstraction in system administration. The complex underlying connection details remain hidden yet accessible when needed:

ssh github
# or
ssh staging

Advanced Features and Implementations

The SSH configuration system provides several sophisticated mechanisms for managing complex connection scenarios. These advanced features demonstrate the extensive capabilities of OpenSSH's configuration framework and its ability to handle diverse operational requirements.

Wildcard Pattern Implementation

Pattern matching facilitates efficient host management through the implementation of glob-style patterns, enabling sophisticated matching rules that apply configurations across multiple hosts. This pattern-based approach significantly reduces configuration redundancy and maintains consistency across similar environments. The pattern matching system employs asterisks (*) and question marks (?) as wildcards, following similar principles to shell globbing patterns.

Basic Pattern Examples

# Development environment configurations
Host dev-*
    User development-user
    IdentityFile ~/.ssh/dev_key.pem
    ForwardAgent yes
    StrictHostKeyChecking ask
    LogLevel INFO
    Port 22000

# Staging environment configurations
Host staging-*
    User staging-user
    IdentityFile ~/.ssh/staging_key.pem
    ForwardAgent yes
    StrictHostKeyChecking ask
    LogLevel INFO
    Port 22001

# Production environment configurations
Host prod-*
    User production-user
    IdentityFile ~/.ssh/prod_key.pem
    ForwardAgent no
    StrictHostKeyChecking yes
    LogLevel ERROR
    Port 22

Advanced Pattern Matching

The pattern matching system supports complex configurations through hierarchical rule application:

# Default settings for all hosts
Host *
    Compression yes
    TCPKeepAlive yes
    ServerAliveInterval 60
    ForwardAgent no

# Regional data center configurations
Host *.eu-west-*
    ProxyCommand ssh eu-jumphost -W %h:%p
    User european-user
    IdentityFile ~/.ssh/eu_key.pem

Host *.us-east-*
    ProxyCommand ssh us-jumphost -W %h:%p
    User american-user
    IdentityFile ~/.ssh/us_key.pem

# Service-specific patterns
Host db-*
    User database-admin
    Port 5022
    StrictHostKeyChecking yes

Host app-*
    User application-admin
    Port 5023
    StrictHostKeyChecking yes

Pattern Precedence Examples

The pattern matching system follows a hierarchical precedence model, where more specific patterns override general ones. This enables sophisticated configuration layering:

# Global defaults
Host *
    ForwardAgent no
    Compression yes
    ServerAliveInterval 60

# Environment-specific overrides
Host *-prod
    ForwardAgent no
    Compression no
    ServerAliveInterval 120
    StrictHostKeyChecking yes

# Region-specific overrides
Host *-prod-eu
    ProxyCommand ssh eu-prod-bastion -W %h:%p
    IdentityFile ~/.ssh/eu_prod_key.pem

# Service-specific configurations within production
Host db-*-prod
    User database-prod-admin
    Port 5022
    IdentityFile ~/.ssh/db_prod_key.pem

Real-world Implementation Examples

Consider a multi-environment, multi-region infrastructure setup:

# Development environments
Host dev-app-* dev-db-*
    User devops
    IdentityFile ~/.ssh/dev_key.pem
    ForwardAgent yes
    StrictHostKeyChecking no

    # Development-specific settings
    LogLevel DEBUG
    Compression yes
    TCPKeepAlive yes
    ServerAliveInterval 30

# Regional production configurations
Host prod-app-eu-* prod-db-eu-*
    User prod-admin
    IdentityFile ~/.ssh/prod_eu_key.pem
    ProxyCommand ssh eu-prod-bastion -W %h:%p

    # Production-specific settings
    LogLevel ERROR
    Compression no
    TCPKeepAlive no
    ServerAliveInterval 60
    StrictHostKeyChecking yes

# Database-specific configurations
Host *-db-*
    # Database server specific settings
    Port 5022
    IPQoS throughput
    Ciphers aes256-gcm@openssh.com,aes128-gcm@openssh.com

# Application-specific configurations
Host *-app-*
    # Application server specific settings
    Port 5023
    IPQoS lowdelay
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com

# Monitoring system access
Host monitor-*
    User monitoring
    IdentityFile ~/.ssh/monitoring_key.pem
    PermitLocalCommand yes
    LocalCommand logger "Monitoring system access: %h"

This comprehensive pattern matching implementation enables:

Environment segregation (development, staging, production)
Regional configuration management
Service-specific settings
Security policy enforcement
Performance optimization per service type
Audit logging for specific access patterns

The pattern matching system proves particularly valuable in large-scale infrastructures where maintaining individual host entries would become unmanageable. Through careful pattern design, administrators can implement consistent policies while maintaining the flexibility to override specific settings where necessary.

ProxyJump Configuration for Bastion Hosts

Bastion host traversal becomes straightforward through proper configuration, implementing the security principle of defense in depth. This approach enables secure access to internal resources while maintaining strict access controls. The ProxyJump feature, introduced in OpenSSH 7.3, replaces the older ProxyCommand methodology:

# Bastion host configuration
Host bastion
    HostName bastion.company.com
    User jumpuser
    IdentityFile ~/.ssh/bastion_key
    StrictHostKeyChecking yes
    LogLevel VERBOSE

# Internal server accessed via bastion
Host internal-server
    HostName 10.0.0.5
    User appuser
    ProxyJump bastion
    IdentityFile ~/.ssh/internal_key
    ForwardAgent no

For more complex scenarios, multiple jump hosts can be specified in sequence:

# Multi-hop bastion configuration
Host internal-private
    HostName 192.168.1.100
    ProxyJump bastion1,bastion2
    User internal-user
    IdentityFile ~/.ssh/internal_key

Connection Persistence Configuration

Optimizing connection management and minimizing authentication requests through persistent connections represents a significant improvement in both security and efficiency. The ControlMaster feature enables multiple SSH sessions to share a single network connection, reducing overhead and improving response times:

Host *
    # Connection sharing configuration
    ControlMaster auto
    ControlPath ~/.ssh/control/%C
    ControlPersist 1h

    # Connection keepalive settings
    ServerAliveInterval 60
    ServerAliveCountMax 3

    # TCP keepalive and compression
    TCPKeepAlive yes
    Compression yes

This configuration implements several important features:

ControlMaster auto: Automatically creates a master connection for subsequent sharing.
ControlPath: Defines the socket file location using %C for a unique hash of connection parameters.
ControlPersist: Maintains the master connection in the background for the specified duration.

The implementation can be further enhanced with environment-specific adjustments:

# Development environments - longer persistence
Host dev-*
    ControlPersist 4h
    ServerAliveInterval 30

# Production environments - stricter settings
Host prod-*
    ControlPersist 30m
    ServerAliveInterval 90
    ServerAliveCountMax 2

Advanced Authentication Configurations

The SSH config file supports sophisticated authentication mechanisms, including multi-factor authentication and certificate-based access:

Host secure-server
    HostName secure.company.com
    User secure-user
    CertificateFile ~/.ssh/user_cert.pub
    IdentityFile ~/.ssh/secure_key
    PKCS11Provider /usr/local/lib/opensc-pkcs11.so
    RequestTTY force
    PreferredAuthentications publickey,keyboard-interactive

Port Forwarding and Tunnel Configuration

Advanced port forwarding configurations enable secure access to remote services while maintaining security boundaries:

Host tunnel-host
    HostName gateway.company.com
    User tunnel-user
    # Forward local port 8080 to remote host's port 80
    LocalForward 8080 internal.company.com:80
    # Forward remote port 5432 to local PostgreSQL instance
    RemoteForward 5432 localhost:5432
    # Dynamic SOCKS proxy on local port 1080
    DynamicForward 1080
    # Ensure tunnel stays active
    ExitOnForwardFailure yes
    ServerAliveInterval 30

Advanced Configuration Patterns

Service-Specific Key Management
The implementation of distinct keys for different services enhances security through isolation and granular access control:

   Host github.com
       IdentityFile ~/.ssh/github_key

   Host gitlab.com
       IdentityFile ~/.ssh/gitlab_key

Environment-Specific Configurations
Different environments often require distinct logging levels and access patterns, reflecting the operational requirements of various deployment stages:

   Host prod-*
       User prod-user
       LogLevel QUIET

   Host dev-*
       User dev-user
       LogLevel VERBOSE

Port Configuration by Environment
Security requirements often necessitate different port configurations across environments, implementing the principle of least privilege:

   Host staging-web
       HostName staging.company.com
       Port 2222

   Host prod-web
       HostName prod.company.com
       Port 22

Security Implementation Guidelines

The implementation of robust security measures in SSH configuration requires a methodical approach to multiple aspects of system security. Each element of the configuration must be carefully considered to maintain the integrity and confidentiality of SSH connections while ensuring operational efficiency.

File System Security

The cornerstone of SSH security begins with proper file system permissions. These permissions form the first line of defense against unauthorized access and potential security breaches:

# Set restrictive permissions on SSH directory
chmod 700 ~/.ssh

# Set proper permissions on configuration file
chmod 600 ~/.ssh/config

# Secure private keys
chmod 600 ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_ed25519

# Ensure public keys are readable
chmod 644 ~/.ssh/*.pub

# Protect known_hosts file
chmod 600 ~/.ssh/known_hosts

Key Management Protocols

The implementation of a robust key management strategy requires careful consideration of several factors:

# Example of key-specific configurations
Host production-*
    # Specify permitted key types
    PubkeyAcceptedKeyTypes ssh-ed25519,rsa-sha2-512

    # Define preferred authentication methods
    PreferredAuthentications publickey

    # Disable password authentication
    PasswordAuthentication no

    # Restrict key forwarding
    ForwardAgent no

    # Enable strict host key checking
    StrictHostKeyChecking yes

    # Use specific identity file
    IdentityFile ~/.ssh/prod_%h_ed25519

Network Security Configurations

Implementation of network-level security measures helps protect against various attack vectors:

# Enhanced security configuration
Host *
    # Prefer modern, secure ciphers
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com

    # Specify secure key exchange algorithms
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group16-sha512

    # Define secure MAC algorithms
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

    # Disable X11 forwarding
    ForwardX11 no

    # Set connection timeout
    ConnectTimeout 60

    # Enable verbose logging for security events
    LogLevel VERBOSE

Environment-Specific Security Policies

Different environments require varying levels of security controls:

# Development environment
Host dev-*
    StrictHostKeyChecking ask
    UserKnownHostsFile ~/.ssh/known_hosts.dev
    LogLevel DEBUG3

# Staging environment
Host staging-*
    StrictHostKeyChecking yes
    UserKnownHostsFile ~/.ssh/known_hosts.staging
    LogLevel VERBOSE

# Production environment
Host prod-*
    StrictHostKeyChecking yes
    UserKnownHostsFile ~/.ssh/known_hosts.prod
    LogLevel ERROR
    IdentitiesOnly yes
    MaxAuthTries 3
    NoHostAuthenticationForLocalhost no

Secret Management Integration

Modern security practices often involve integration with external secret management systems:

# Example of retrieving credentials securely
Host secure-service
    # Use environment variables for sensitive data
    Match exec "test -n '$SSH_SECRET_KEY_PATH'"
        IdentityFile $SSH_SECRET_KEY_PATH

    # Integration with external secret management
    Match exec "vault-ssh-helper --verify"
        IdentityFile ~/.ssh/vault-signed-key

Audit and Logging Configurations

Implementing comprehensive logging helps in security monitoring and incident response:

Host *
    # Enable detailed logging
    LogLevel VERBOSE

    # Log connection attempts
    Match exec "logger 'SSH connection attempt to %h'"
        LogLevel DEBUG

    # Additional security logging
    PermitLocalCommand yes
    LocalCommand logger "SSH connection established to %h by %r"

Conclusion

The SSH config file represents a powerful tool for optimizing SSH workflows and implementing robust security practices. Through proper configuration, complex SSH commands can be condensed into efficient, memorable aliases while maintaining comprehensive security standards. This approach exemplifies the balance between usability and security in systems administration.

The management of multiple servers becomes considerably more efficient through effective SSH config file utilization. Beginning with basic host configurations and progressively implementing more advanced features allows for natural skill progression and workflow enhancement, following the principle of incremental improvement in systems administration.

It is worth noting that the examples presented constitute only a subset of the available functionality. The official OpenSSH documentation provides extensive additional options for advanced configuration and customization, offering numerous possibilities for further optimization and security enhancement.

Access to Google Cloud Virtual Machine through SSH

Eugene Zimin — Tue, 26 Nov 2024 05:30:06 +0000

Earlier in another article, we provided a detailed description of how to create and set up a Virtual Machine (VM) instance in the Google Cloud Platform (GCP) using Google Compute Engine (GCE). It is accessible here - Google Cloud: Provisioning a Virtual Machine and Accessing it via SSH.

Provisioning a GCP VM allows us to create our own powerful computing environment to serve as a sandbox or workspace for our further testing and experiments. Having successfully deployed a VM in GCP, the next crucial step is to establish a secure remote connection to this virtual machine. It is required by many reasons, starting from efficient administration and management up to the monitoring and observing it. Google Cloud Platform provides a straightforward method to configure SSH access to Compute Engine instances, facilitating an encrypted communication channel. In this article, we will outline the steps required to set up SSH access to a Google Compute Engine (GCE) virtual machine.

Few Words About SSH

Secure Shell (SSH) is a cryptographic network protocol and the de facto standard for secure remote login and command execution on Unix-like operating systems. SSH provides a secure encrypted channel over an unsecured network, preventing potential eavesdropping, packet sniffing, or man-in-the-middle attacks that could compromise login credentials or sensitive data. By leveraging strong encryption algorithms and authenticated key exchange, SSH ensures data integrity and confidentiality, making it an essential tool for securely administering remote servers, transferring files, and managing networked systems over insecure networks such as the internet.

SSH operates through the use of encrypted key pairs - a public key that gets placed on the remote server, and a private key that is kept secure by the client. The keys are very large numbers derived via a one-way mathematical function, making them virtually impossible to derive if intercepted during transmission.

When initiating an SSH connection, the client and server negotiate a secure symmetrical encryption cipher and session keys to use. This key exchange utilizes the private/public keys for authentication and protection against man-in-the-middle attacks. Once the secure tunnel is established, all subsequent commands, file transfers, and communications are encrypted between the client and server.
Figure 1. Communication schema for SSH exchange

SSH supports several strong modern encryption algorithms such as AES, Blowfish, 3DES, and ciphers like ChaCha20. The specific algorithms used can be configured on both ends based on policy and requirements. SSH also provides data integrity verification through cryptographic message authentication codes like HMAC-SHA1 to detect tampering.

By default, SSH operates on TCP port 22, but can be configured to use any port. It supports various user authentication mechanisms like passwords, public keys, security keys, and more. SSH key-based authentication is considered more secure than basic password authentication.

Configuring SSH on Local Machine

Before initiating an SSH connection to our Google Cloud VM instance, we'll need to have SSH configured and set up on our local machine. The steps vary slightly depending on the operating system, as Linux and macOS usually have preinstalled SSH client which requires minimal configuration, whereas Window requires a bit more steps to make initial setup ready.

Linux and macOS

Most Linux distributions and macOS come pre-installed with OpenSSH, a free open source SSH client and server utility. To check if it's installed, we can open a terminal and run:

ssh -V

# Response should be something like the line below
# OpenSSH_9.6p1, LibreSSL 3.3.6

This should display the installed OpenSSH version. If not installed, we can get it through the operating system's package manager, for Linux it may be usually apt or yum for macOS brew is very popular.

Windows

Windows 11 has pre-installed SSH client. To check it we need to open cmd application (a.k.a Command Prompt) and run the same command:

ssh -V

If we will see a similar output we saw for Linux and macOS (some names may vary), it means we have installed SSH client and we are good to go.

A little bit more complicated situation happens if Windows doesn't have preinstalled client. In that case we may use one of the following 3rd party applications to run it:

PuTTY for Windows
Git Bash for Windows

Generating Key Pairs

To establish secure SSH connections, we need to generate cryptographic key pairs consisting of a public and private key. The ssh-keygen utility allows us to create these key pairs locally on our machine. Before we start creating a key pair it may worth to overview file schema for the it.
Figure 2. Schema of SSH key pairs

We can run ssh-keygen in a terminal/command prompt and follow the prompts. Some common options include:

-t to specify the key type (e.g. rsa, ecdsa, ed25519)
-b to set the key length in bits (e.g. 2048, 4096)
-C to add a comment to identify the key

For example:

ssh-keygen -t rsa -b 4096 -C "user@example.com"

This generates an RSA key pair with 4096 bits length associated with the user@example.com email.

There are various key types we may use. Here are the most popular ones we may want to generate:

RSA - One of the oldest and most widely used key types. Recommended minimum key length is 2048 bits, with 4096 bits being more secure.
ECDSA (Elliptic Curve Digital Signature Algorithm) - This key type is based on elliptic curve cryptography which provides equal security strength with smaller key sizes compared to RSA.
Ed25519 - A modern EdDSA (Edwards-curve Digital Signature Algorithm) key type that provides better performance and security than RSA/ECDSA. Ed25519 uses elliptic curve cryptography with a 256-bit key length.

While RSA is the most widely compatible, Ed25519 keys are considered more secure and efficient for modern systems. We can check which key types our SSH server and client support using ssh -Q key.

During key generation, we have the option to protect the private key with a passphrase. While providing a passphrase enhances security, we need to enter it every time the key is used.

After generating, both keys - private and public - will be saved in the local SSH configuration folder. Usually (for Linux and macOS systems) this folder is under the following path - ~/.ssh/. Both keys by default use their name pattern as id-<algorithm-name>[.pub] , where extension .pub is used only for public key from the pair. As an example, user may get id-rsa and id-rsa.pub if there was RSA algorithm used, or id-ed25519 and id-ed25519.pub there was Ed25519 algorithm in use during key pair generation.

Next step would be to copy the content of the public key (which has .pub extension) to the remote server, in accordance with the Figure 2. We may copy the content of the public key and paste it on the new line of the ~/.ssh/authorized_keys file for the user which we will be connecting to. We can do that by many ways, but usually we should have access to the remote server via screen sharing or through the web interface, like with Google Compute Engine in GCP, where we can paste our public key.

NOTE FOR GCP
GCP requires to add at the end of the key userName and expireOn in form of JSON. We should do that manually to get the ready for pasting. Eventually we should have something like that:
ecdsa-sha2-nistp521 AAAAE2...== user@laptop.local {"userName":"<user@yourgcp.email>","expireOn":"2024-05-06T08:43:20+0000"}

At the same time private key (without .pub extension) remains securely stored on our local machine. We must keep the private key safe and never share it.

With our key pair ready, we can now proceed to configure server to use key pair only and disallow further access using username and password.

Configuring Server Side

While using SSH keys provides a much more secure authentication mechanism compared to passwords, many cloud providers, including Google Cloud Platform, enable password-based logins to SSH servers by default. However, relying solely on password authentication for SSH connections introduces potential security risks and is considered a poor practice, so we should consider to disable it and keep it disabled.

Main reasons why we should do that lay in the security considerations. Having password authentication enabled we allow anyone to try to connect to remote server and try out to brute force the password. Sometimes password are not that strong and such an attack might be successful.

That being said the advantages of disabling Password authentication for the remote server are as follows:

Enhanced Security: SSH keys utilize strong encryption algorithms and prevent brute-force and dictionary attacks that are common threats to password-based authentication. Keys are virtually impossible to guess or decrypt.
Eliminate Weak Passwords: Enforcing key-based authentication eliminates the risk of users setting weak or easily guessable passwords, which is a common vulnerability.
Audit Trail: SSH key pairs provide better audit trails and monitoring capabilities, as each key can be associated with a specific user or system.
Compliance: Many security standards and best practices, such as PCI-DSS, HIPAA, and NIST, recommend or mandate the use of key-based authentication over passwords for remote access to servers.

At the same time nothing comes without price. If any of the parts for the key pair would be lost or corrupted, it means we may also lost the access to the server, thus we should be very careful and keep generated key pair as good as possible.

To disable password authentication for the SSH server we need to modify the SSH daemon (sshd) configuration file. To do that we need to connect to the VM instance over SSH. We will use our new generated private key to connect to the server (don't forget to replace username and address of the server).

ssh -i ~/.ssh/id_ed25519 remote-user@server-ip-address

On the remote server open the SSH daemon configuration file as a superuser:

sudo nano /etc/ssh/sshd_config # may be changed to vim

In the file editor find the line #PasswordAuthentication yes and change it to PasswordAuthentication no and save the file and exit the editor.

Finally we need to restart the SSH daemon to make our changes take effect:

sudo systemctl restart sshd # (or command for your OS)

After making this change, the SSH server will only allow key-based authentication, and password logins will be disabled.

It's important to note that we should have at least one authorized SSH key added to the VM instance before disabling password authentication. Otherwise, we may get locked out of the system.

By enforcing key-based authentication and disabling password logins, we significantly enhance the security of our SSH server and remote access to our Google Cloud virtual machines, aligning with industry best practices for secure system administration.

Continuance

The next article in the cycle is dedicated to the debugging and troubleshooting different connection errors happening during establishing SSH connections - Debugging SSH connections: A Comprehensive Guide

JWT at a Glance

Eugene Zimin — Mon, 19 Aug 2024 04:32:11 +0000

Many of us heard about JWT and while JWT has become a buzzword in tech circles, it's frequently misunderstood or confused with OAuth 2.0 and OIDC, particularly among those who use it without fully grasping its intricacies. Mixing up JWT with OAuth 2.0 and OIDC is like tossing different fruits into a blender and calling it all "smoothie." It's especially messy when folks treat these tech tools like magic wands, waving them around without peeking under the hood.

What is JWT? By itself is a small piece of data that contains information about someone or something. It's like a small container which is able to keep and transfer information between two different destinations. Or it's like a digital badge that proves who the user is and what this user is authorized to do.

Let's imagine a library where anyone could walk in and borrow books without showing any identification. It would be chaos! The library needs a way to know who's borrowing what, and to ensure that only members can borrow books. In the digital world, JWT serves a similar purpose.

Session Management Based Authorization

Traditional web applications often use sessions to keep track of who's logged in. This is like giving someone a special badge when they enter the library. Library staff can see the badge and then if they need to know details - they will check them out inside the computer. Let's take a look at the example below.
Figure 1. Regular request-response flow

This diagram illustrates the basic flow of a web application, where the browser sends a request to the server, the server processes the request (potentially interacting with a database), and then sends a response back to the browser. Nothing complex, it's just a business as usual:

User sends a request to the Service, seeking for some information.
Service goes to the Database to retrieve that data from the long term storage.
Finally Service returns that data to the user in his browser to view.

Now, let's imagine if the Service holds sensitive personal information. Without security system to verify who's knocking on the door, there's a real danger of handing out private details to the wrong hands. It's like throwing a party and accidentally inviting the whole neighborhood when you meant to host a small gathering. In the digital world, this kind of mix-up isn't just embarrassing - it's a serious security risk that could expose confidential data to anyone who comes asking. That's a scenario is a "no-go" situation!

To prevent this situation to ever happen, let's think about implementing another service, which would be in charge to authenticate users and grant them special permissions which data they allow to access and which is not. Look at the diagram below:

Figure 2. Regular request-response flow with Login Service

What is changed on the diagram? Let's take a look again. We've introduced a new service which will be in charge for user's authentication and authorization. In other words we make this service exists to ensure proper user verification before granting access to the system's resources.

Prior to allowing any interaction with the data, this service requires users to authenticate themselves through a login process. Upon successful identification, the system initiates a session and issue a session identifier, or SessionID. In this context, a session is essentially a database record in Auth Users database indicating user is online and interacting with the system.

Upon successful authentication, all the rest interaction with the system no longer require the user to re-authenticate. Instead, the user's client (browser) simply keeps and provides back with every request the session identifier, which is securely stored in the Auth Users database. This identifier serves as a temporary credential, allowing the system to recognize and authorize the user for the duration of their session.

It may be shown like on the diagram below.

Figure 3. Session Management

Let's walk through this web application flow step by step, in a way that's easy to understand:

The process begins with the browser initiating an HTTP request to the server. This should happen only after user has been authenticated and have a Session ID in hands as a response from the authentication flow.
1. HTTP request itself is comprehensive and contains at least:
  - The destination URL
  - A token in the headers, any relevant cookies and optionally - a request body
2. The token in the headers carries a Session ID - a unique identifier acting as a digital passport for the user's session.
Upon receiving the request, the Service doesn't immediately process it. Instead, it has to forwards the session information to the Auth component for verification of the request sender, i.e user.
The Auth service cross-references the Session ID with the database, essentially checking the list of sessions and its mapping to the provided Session ID. If there is a match, the Auth component retrieves and returns the associated user data, which may include any necessary data about the user, like user ID, first and last name of the user, his email and user's role and permissions.
Armed with this user information, Service can now safely decide whether user is able to interact with the system or not. Eventually, Service compiles the requested information and sends it back to the browser as an HTTP response if the user is authorized or rejects it if the user is not authorized.

That's it! As this process occurs with each interaction between user and the system, we may consider the system is safe unless the session token is not stolen or brute forced.

Is that good or bad approach? There is no straightforward answer. One of its greatest benefits is that the backend fully controls the access to all protected resources. By any chance if any request would be considered suspicious, user might be easily kicked off of the system by erasing a record in the Auth Users database with the Session ID associated to him.

On the other hand - backend needs to keep the state about user's status - whether he authenticated or not. This stateful nature of session-based authentication brings challenges.

As every request to the system necessitates a database lookup to verify the session we can easily imagine what happens if our UI sends N (where N > 0) concurrent requests to retrieve information from the backend (hello React and other SPA):

Figure 4. Potential bottleneck with the session management

Take a look at the diagram above. On every user action, for example an initial page load, the UI may send 4 requests to fill out different page sections. That may happen as those data might be provided by different API Endpoints, like information about the user should be obtained from /api/v1/users and information about their appointments from another one, like /api/v1/appointments. More importantly, those requests are usually sent concurrently, as we want rapid page rendering in the user's browser.

Now let's estimate the cost of such an approach. Every user action with that page will push us to send 4 concurrent requests to the Service. Okay, that's doable. However each of those requests will also need be authorized, which means Service should ask Auth is that possible to proceed with them as it doesn't have any other information than Session ID which is nothing more than a link in Auth Users database. In that case Service needs to issue another 4 requests to the Auth and ask it about user information and details.

Easy calculation gives us that 1 user action creates 4 requests to one service in our backend and same amount for another service on the backend. In total, these eight actions will all result in database interactions. Let's imagine there are another services which also require confirmed authorization and we will understand the full picture.

Is it too much? The answer depends on the use case - sometimes it's necessary. However, is it possible to decrease the load on the database and other services?

Stateless Authorization Management

There is another way to handle system security and authorize users without using session management. This alternative approach tackles some of the scalability headaches we saw earlier and it is often implemented using JSON Web Tokens (JWT).

As we mentioned earlier JWT is a small container which holds some information for us. Compare it with the Session ID:

Figure 5. Compare Sessions with JWT approaches

In a JWT-based system, the server no longer needs to store session information. Instead, when user authenticates, the Auth generates a JWT - a compact, self-contained token that encapsulates all necessary user information and permissions. This token is then sent back to the client and stored, typically in local storage or a cookie.

Interesting thing happens on the next stage - when user needs to reach protected resources. Using previous approach we have to trigger authorization logic on every upcoming request on a different service which is Auth in the example above. What happened when we employ JWT?

Take a look at the diagram below.

Figure 6. JWT based Authorization flow

When a user makes a request to access a protected resource, their client (typically the browser) automatically includes the JWT in the request headers. This is usually done by setting the Authorization header to "Bearer token", where token is the actual JWT.

Upon receiving the request, the Service no longer needs to consult with the Auth service for each incoming request. Instead, it can verify the JWT's integrity directly and decode the JWT on its own. This is possible because the JWT is cryptographically signed, and the Service has access to the JWT secret key or public key (in case of asymmetric signing) used to sign the token by Auth at the JWT creation time.

Once the signature is verified, the Service can decode the JWT payload. This payload contains all the necessary information about the user - their ID, roles, permissions, and any other relevant data that was included when the token was created. All of this happens without any database lookups or calls to the Auth service.

Finally with the user information available from the decoded JWT, the Service can make immediate authorization decisions. It can check if the user has the necessary permissions to access the requested resource, all without any additional network calls or database queries.

JWT Structure

To make this flow happen JWT should be able to carry on some information. That includes not only some useful information we may want to store in the token, but also data, allowing to make the flow secured.

RFC 7519 defines the JWT as follows.

JSON Web Token (JWT) is a compact, URL-safe means of representing
claims to be transferred between two parties. The claims in a JWT
are encoded as a JSON object that is used as the payload of a JSON
Web Signature (JWS) structure or as the plaintext of a JSON Web
Encryption (JWE) structure, enabling the claims to be digitally
signed or integrity protected with a Message Authentication Code
(MAC) and/or encrypted.

Using that we can describe the structure of a JWT as of (usually) consisting of three parts: a header, a payload, and a signature, separated by . (dot) symbol. The payload contains claims about the user, such as user ID, role, and expiration time. The signature, created using a secret key known only to the server, ensures the token's integrity and authenticity.

Figure 7. JWT Structure

All these 3 parts are separated by dot symbol and following in the order of:

header . payload . signature

These three components are encoded using the BASE64URL algorithm. This encoding method is closely related to standard BASE64, but with a few key differences. In BASE64URL, the output replaces the plus sign + with a minus sign -, and the forward slash / becomes an underscore _. Additionally, BASE64URL omits the typical padding found in standard BASE64, which usually consists of equal signs = at the end of the encoded string. These modifications make the encoded output more suitable for use in URLs and other contexts where certain characters might cause issues.

Payload

If the payload (middle) part is more or less clear, as it is the container for user defined data, it still worth to take a look at some predefined fields over there.

RFC 7519 standard calls data inside the payload a Climes Set and defines three classes of JWT Claim Names

The JWT Claims Set represents a JSON object whose members are the
claims conveyed by the JWT.

There are three classes of JWT Claim Names: Registered Claim Names,
Public Claim Names, and Private Claim Names.

Registered Climes are those which defined by the standard and used by applications in case they considered as mandatory by developers. Here they are:

jti (JWT ID) Claim a case-sensitive unique identifier for the JWT, designed to prevent token replay and ensure uniqueness across multiple issuers
iss (Issuer) Claim identifies the JWT's issuer using a case-sensitive StringOrURI value, with processing typically application-specific
iat (Issued At) Claim specifies the JWT's creation time, allowing for age determination of the token
sub (Subject) Claim identifies the JWT's principal using a case-sensitive StringOrURI value, which must be locally or globally unique, and typically serves as the subject of the token's claims
aud (Audience) Claim specifies the intended recipients of the JWT as a case-sensitive string or array of strings, each containing a StringOrURI value, and requires the processing principal to be identified within this claim for token acceptance
exp (Expiration Time) Claim specifies the latest valid processing time for the JWT, with a small leeway often permitted for clock skew
nbf (Not Before) Claim sets the earliest time the JWT can be processed, with a small leeway often allowed for clock skew

Different applications may use different claims from the list above, but many of them rely at least on 2 - which are iat and exp to determine the lifecycle of the token itself and when it should be regenerated.

Public Claims contains user defined information and may include there everything in JSON format. Usually developers create it with user data, which is necessary to be shown on the UI, user permissions and some additional information.

Header

The JWT header plays a critical role in the token's functionality. This component, appearing at the beginning of the token, contains metadata about the token itself. Two key elements stored in the header are the token type and the hashing algorithm used for the signature.

The token type, denoted by the typ claim, usually specifies JWT to indicate that the token is indeed a JSON Web Token. This information helps systems quickly identify and process the token appropriately. The hashing algorithm, represented by the alg claim, specifies the cryptographic algorithm employed to create the token's signature. Common algorithms are HMAC SHA256 (HS256) or RSA SHA256 (RS256), but there are many others which are possible.

Signature

This section of the JWT is supposed to provide a mechanism of verification of the token integrity and authenticity. It is important and serves only for a one reason - prevent modification of any information at any stage.

This validation of the JWT happen on the backend Service, right after the Service received user's request for data. Server actually can recreate the signature using the same secret key (which is secretly stored on the backend) and algorithm (which is defined in the JWT header section). If the newly generated signature matches the one in the token, it confirms that the contents haven't been altered since the token's creation.

Refreshing JSON Web Tokens

There is a last but not least thing to consider - JWT lifetime. After issuing a JWT to a user, we actually allowing him to access the protected data in accordance with his level of permissions. However do we want to limit it in time? In other words - do we want that user confirms his authenticity after some time or not?

Even though this question is not that obvious, the answer is straightforward and positive. User must to re-confirms his authenticity and he should do that within a reasonable amount of time. It matters because we can't exclude the situation or JWT leakage, as what goes to browser is available to the rest of the world. JWT provides safety mechanism from data modifications (by signing them and later signature verification), but they do not save from reusing them.

This brings us to the concept of JWT refreshing. While limiting the lifetime of a JWT is necessary for security reasons, it can also lead to a poor user experience if the token expires too quickly, forcing frequent re-authentication. And this is where refresh tokens come into play.

A refresh token is another special token that can be used to obtain a new JWT once the original JWT has expired. Unlike JWTs, refresh tokens are typically stored server-side only and are associated with a specific user, using some of its unique data, like a hash of the JWT itself. They have a longer lifespan than JWTs but are only used to verify one thing - the old JWT is valid even if it is expired and we can safely issue a new one. In other words it let us know that original JWT was not modified and still can be trusted.

This approach balances security and user experience. It allows for short-lived JWT reducing the window of potential token misuse, while also allowing users to remain logged in for extended periods without explicit re-authentication.

Scalability Advantages of JWT

The stateless nature of JWT provides a significant boost especially for distributed systems in terms of scalability, when compared to traditional session-based authentication systems. This makes JWT an attractive approach for modern, distributed applications, such as those deployed in cloud environments or utilizing load-balanced architectures.

In traditional session-based systems, user authentication data is stored on the server side, typically in a database or in-memory store. Each time a user makes a request, the server must retrieve this session information to verify the user's identity and permissions. As the number of concurrent users grows, this approach can lead to increased load on the session store, potentially becoming a bottleneck in the system.

JWT, on the other hand, encapsulate all necessary authentication and authorization information within the token itself. When a user authenticates, the server generates a JWT containing the user's identity and permissions, then sends this token back to the client. For subsequent requests, the client includes this token, allowing the server to verify the user's identity and permissions without querying a central session store.

JWT's stateless nature aligns well with microservices architectures. In a microservices environment, different services can easily validate the JWT and extract necessary information without maintaining complex, centralized session management systems. This decoupling of authentication state from individual services enhances the overall scalability and flexibility of the system.

Next Steps

In the next article, we'll take a look further into the authentication and authorization cased by integrating JWT and two other important protocols: OAuth 2.0 and OpenID Connect (OIDC). Stay tuned to learn how combining JWT with OAuth 2.0 and OIDC can enhance your application's security posture while maintaining the scalability benefits we've discussed!

Using Domain-Driven Design to to Create Microservice App

Eugene Zimin — Mon, 24 Jun 2024 07:18:01 +0000

Creating scalable and maintainable applications remains a constant challenge in the software development industry. As digital ecosystems grow more complex and user demands evolve rapidly, developers face increasing pressure to build systems that can adapt and expand seamlessly.

Scalability presents a multifaceted challenge. Applications must handle growing user bases, increasing data volumes, and expanding feature sets without compromising performance. This requires careful architectural decisions, efficient resource utilization, and the ability to distribute workloads effectively across multiple servers or cloud instances.

Maintainability, on the other hand, focuses on the long-term health of the codebase. As applications grow in complexity, keeping the code clean, understandable, and easily modifiable becomes crucial. This involves creating modular designs, adhering to coding standards, and implementing robust testing strategies. A maintainable codebase allows for faster bug fixes, easier feature additions, and smoother onboarding of new team members.

The intersection of scalability and maintainability often leads to trade-offs. Highly optimized systems for scalability may sacrifice code readability, while overly modular designs for maintainability might introduce performance overhead. Striking the right balance requires deep domain knowledge, technical expertise, and a forward-thinking approach to software design.

The rapid pace of technological change adds another layer of complexity. Developers must create systems that not only meet current needs but also remain flexible enough to incorporate future technologies and methodologies. This forward-looking approach is essential in preventing technical debt and ensuring the longevity of the application.

Rapid pace of technological change adds another layer of complexity. Developers must create systems that not only meet current needs but also remain flexible enough to incorporate future technologies and methodologies. This forward-looking approach is essential in preventing technical debt and ensuring the longevity of the application.

Domain-Driven Design, first introduced by Eric Evans, emphasizes aligning software design with business needs. It provides a set of patterns and practices that help architects and developers create flexible, modular systems that can evolve with changing requirements. By centering the design process around the core domain and domain logic, DDD facilitates the creation of software that truly reflects the underlying business model.

Domain-Driven Design provides a framework for creating systems that are both scalable in terms of functionality and maintainable in terms of code organization. It offers strategies for managing complexity, facilitating communication between technical and non-technical stakeholders, and creating flexible systems that can evolve with changing business needs.

Domain-Driven Design Quick Overview

Domain-Driven Design (DDD) is a software development approach that places the project's core focus on the domain and domain logic. Introduced by Eric Evans in his seminal book, DDD provides a set of principles and patterns for creating complex software systems that closely mirror the business domain they serve.

At the heart of DDD lies the concept of the ubiquitous language. This shared vocabulary, meticulously crafted by developers in collaboration with domain experts, forms the foundation of the model. It ensures that all stakeholders, from business analysts to programmers, communicate effectively using terms and concepts directly related to the domain. This alignment significantly reduces misunderstandings and helps create software that truly reflects business needs.

Bounded contexts represent another cornerstone of DDD. These are explicit boundaries within which a particular domain model applies. In large systems, different parts may have different domain models, each with its own ubiquitous language. Recognizing and defining these boundaries helps manage complexity and allows teams to work independently on different parts of the system.

Aggregates are a key tactical pattern in DDD. An aggregate is a cluster of domain objects treated as a single unit for data changes. The aggregate root, a single entity within the aggregate, ensures the consistency of changes within the aggregate and controls access to its members. This pattern is particularly useful in defining clear boundaries for transactions and maintaining data integrity.

Domain events are another crucial concept in DDD. These represent significant occurrences within the domain. By modeling important changes as events, DDD facilitates loose coupling between different parts of the system, enabling more flexible and scalable architectures.

In the context of microservices architecture, DDD proves exceptionally valuable. The bounded contexts in DDD often align well with individual microservices, providing a natural way to decompose a complex system into manageable, independently deployable services. This alignment helps in creating a modular system where each service has a clear responsibility and a well-defined interface.

The strategic patterns of DDD, such as context mapping, help in understanding and defining the relationships between different bounded contexts or microservices. This is crucial for managing the complexity that arises from distributed systems and ensuring that the overall system remains coherent.

By applying DDD principles, developers can create software that's not only aligned with business needs but also inherently more maintainable and scalable. The focus on the core domain ensures that development efforts are concentrated where they provide the most value. The clear boundaries and well-defined interfaces facilitate easier changes and extensions to the system over time.

In the following sections, we will explore how these DDD concepts can be applied to create a Twitter-like application using a two-microservice architecture. This practical example will demonstrate how DDD can guide the design of complex systems, resulting in a flexible and maintainable solution.

Defining the Bounded Contexts

In applying Domain-Driven Design (DDD) to our Twitter-like application, the first step is to identify and define the bounded contexts. Bounded contexts are crucial in DDD as they delineate the boundaries within which a particular domain model applies. For our simplified Twitter-like platform, we'll focus on two primary bounded contexts, each corresponding to a microservice: UserManagementService and MessagingService.

UserManagementService Bounded Context

The UserManagementService encompasses all aspects related to user accounts and profiles. This context is responsible for managing user identities, authentication, and user-specific information. Within this bounded context, the core concepts include:

User - the central entity representing an individual account on the platform, which contains user-specific details such as display name, bio, and profile picture, authentication information like username and password, etc.
Role - defines user role in the whole system, whether he's a messages producer or messages consumer only

The ubiquitous language within this context includes terms like "register", "login", "logout", "authentication" and "authorization". The UserManagementService operates independently, focusing solely on user-related operations without direct concern for messaging or content creation.

MessagingService Bounded Context

The MessagingService handles all aspects of content creation, distribution, and consumption. This context is more complex as it manages both the creation of messages (tweets) and the subscription system. Key concepts in this bounded context include:

Message: The core entity representing a piece of content (similar to a tweet).
Subscription: Represents the follow relationship between users.
Feed: A collection of messages from subscribed users.
Producer: A user who creates content (messages).
Consumer: A user who consumes content from their subscriptions.

The ubiquitous language here includes terms like "post message", "subscribe", "unsubscribe", "consume message". This context deals with the dynamic aspects of the platform, managing the flow of content between users.

Context Interactions

While these bounded contexts are separate, they do interact. The MessagingService needs to reference users, but it does so without delving into the internal complexities of user management. Instead, it might use a simplified representation of a user, containing only the necessary information for messaging purposes.

Figure 1. Context Interactions between bounded contexts

For instance, when a user posts a message, the MessagingService doesn't need to know about the user's password or email address. It only needs a user identifier and perhaps a display name. This separation allows each context to evolve independently while maintaining necessary connections.

This separation is further reinforced by implementing the database-per-service pattern. In this approach, each microservice maintains its own dedicated database. The UserManagementService has a database that stores comprehensive user information, including sensitive data like passwords and email addresses. On the other hand, the MessagingService database focuses on message content, user subscriptions, and a minimal set of user data required for its operations.

To maintain necessary connections between services, the MessagingService keeps a minimal replica of user data required for its operations. This typically includes the user ID only. If MessagingService requires some data related to the user, it may request that information from UserManagementService. This may be useful for checking user roles for example or user name to display.

Technically speaking, this approach introduces some data redundancy, however organized correctly, such a redundancy may be reviewed as nothing much but as foreign keys used between different tables in RDBMS.

It allows each service to operate independently most of the time, enhancing overall system resilience. It also aligns well with the DDD principle of respecting bounded context boundaries, as each service has full control over its own data model and storage.

Context Mapping

To manage the relationship between these bounded contexts, we employ context mapping. In this case, we might use a Customer-Supplier relationship, where the UserManagementService (supplier) provides necessary user data to the MessagingService (customer). This could be implemented through API calls or message queues, ensuring that each service has the information it needs without tightly coupling the two contexts.

Implementation of this context mapping involves several key aspects. First, the UserManagementService exposes a well-defined API that the MessagingService can use to retrieve essential user information. This API is designed to provide only the necessary data, such as user IDs and user roles, without exposing sensitive information. For example, when a new message is posted, the MessagingService might make an API call to the UserManagementService to verify the user's existence and his role.

Figure 2. Context Mapping between different bounded contexts

As part of the Customer-Supplier relationship, we define clear Service Level Agreements (SLAs) between the services. These SLAs specify the expected request and response structure, timeline for API calls, the frequency of event publications, and the handling of service outages.

Leveraging these patterns we will create a foundation for a modular, maintainable system. Each context has a clear responsibility and a well-defined interface, allowing for independent development and scaling. This separation also provides flexibility for future expansions, such as adding new features or integrating with external systems.

In the following sections, we'll delve deeper into each of these bounded contexts, exploring their domain models, aggregates, and services in detail.

User Management Service

The UserManagementService forms a crucial part of our Messaging application, handling all aspects related to user accounts and roles. This service embodies its own bounded context, focusing solely on user-related operations. Let's delve into the key components of this service, structured according to Domain-Driven Design principles.

Domain Model

At the core of the UserManagementService lies the User aggregate. In DDD, an aggregate is a cluster of domain objects treated as a single unit. The User aggregate encapsulates all user-related data and behavior.

The User aggregate root contains essential attributes such as userId, username, email, and password. It also includes methods for user authentication, user roles and user authorization. By centralizing these responsibilities within the User aggregate, we ensure that all user-related operations maintain data consistency and adhere to business rules.

Associated with the User aggregate are several value objects. These are immutable objects that describe characteristics of the User but have no conceptual identity. Examples include Email and Password. These value objects encapsulate validation logic, ensuring that email addresses are properly formatted in accordance with RFC 2822 and passwords meet security requirements by its length and other criteria.

The Profile value object represents user-specific details like displayName, bio, and profilePictureUrl. By modelling Profile as a separate value object, we allow for easy expansion of profile-related features without cluttering the User aggregate.

Repository

The UserRepository interface defines methods for persisting and retrieving User aggregates. This abstraction allows us to separate the domain model from the underlying data storage mechanism. The implementation of this repository interfaces with our chosen database system, in this case, MySQL.

The repository includes methods such as findById, findByUsername, and regular CRUD operations (CREATE-READ-UPDATE-DELETE). It also provides more specific query methods like findByEmail, which might be used during the user registration process to ensure email uniqueness.

Data model, which is also scoped to User aggregation only, might be represented as on the figure below.

Figure 3. User Management Data Model

Domain Services (Domain Flows)

While most business logic resides within the User aggregate, some operations involve multiple aggregates or require external resources. For these, we use domain services which may exist as separate services or might be defined as modules inside a single service. As of now we will prefer to define those operations as separate reusable modules inside one service.

The UserRegistrationModule handles the process of user registration, which might involve checking for existing users, validating input, and triggering welcome emails. This module coordinates between the User aggregate, UserRepository, and potentially external services like email providers.

The AuthModule manages user login processes, including password verification and generation of authentication tokens, as well as handles requests to identify user roles. This module works closely with the User aggregate but keeps the authentication and authorization logic separate, allowing for easier updates to authentication and authorization mechanisms in the future.

ApplicationGatewayModule is responsible for communication with other external components and services. In some other sources it is also called a Controller, however we try to include here only logic, responsible for terminating incoming traffic, data serialization, input validation and forming responses to return them to the requesters. This module acts like a communication facade between different external services and internal modules, providing SLA for them, and at the same time keeping flexibility of internal modules.

The last module we may want to use is DataAccessLayerModule. We already defined the data model, now we need to determine the way to communicate with it. Using direct calls to the database from any point of the code is possible but may not be considered as a good idea because of many reasons, main ones are correlated to the high coupling and low cohesion between data model and application logic. Using this module we may decompose and abstract access our logic from accessing data. Instead of creating a specific SQL query to select specific user and access his permissions, we may simply create a method, called getUserWithRoles, and hide the implementation behind it. In that case we reach high consistency by calling the only one method, which we will be a single point to access data.

Figure 4. Block Diagram of Domain Modules

Infrastructure Concerns

While not strictly part of the domain model, the UserManagementService also addresses several infrastructure concerns. These include implementing security measures like password hashing and token-based authentication, setting up database connections and ORM mappings, and configuring API endpoints.

The service also implements event publishing for significant domain events. When a user updates their profile, for instance, the service publishes a UserProfileUpdated event. Other services, like our further considering NotificationService, can subscribe to these events to maintain consistency across the system and deliver different state changes.

By structuring the UserManagementService according to DDD principles, we create a robust, maintainable service that clearly encapsulates all user-related functionality. This design allows for easy extension of user management features and provides a solid foundation for our Messaging application.

MessagingService

The MessagingService forms the core of our Messaging application's content creation and distribution system. This service encompasses its own bounded context, managing messages (tweets) and subscriptions. Let's explore the key components of this service, structured according to Domain-Driven Design principles.

Domain Model

The central aggregate in the MessagingService is the Message. This aggregate encapsulates the content created by users, along with metadata such as creation time and author information. The Message aggregate root contains attributes like messageID, content, authorID (which is a userID), and createdAt. It also includes methods for creating, editing (if allowed), and deleting messages.

Another crucial aggregate is the Subscription, representing the follow relationship between users. The Subscription aggregate contains a producerID and a subscriberID, which are no more than userID correlated to the appropriate data from the UserManagementService, along with subscription status and creation date. This aggregate is responsible for managing the relationships between content producers and consumers.

Value objects in this context might include MessageContent, which encapsulates the actual text or media content of a message and its author ID, to establish and determine the subscription status.

Repository

The MessagingService might include several repositories in case we would like to split it into standalone services to manage messages and subscriptions. In this article we consider that functionality as closely related and manage it under the single umbrella of the MesagingService and within the single database for this service.

The MessageRepository handles persistence and retrieval of Message aggregates. It includes methods like regular CRUD, findByMessageId, findByAuthorId, and findBySubscriberId. We may also implement another method allowing users to filter messages by the time when they were created, which may be another convenient use case, and call this method as findByCreatedAt.

As it is seen, this database includes all those entities related to the messages and subscriptions. It's schema might be represented like on the figure below.

Figure 5. Messaging Service Data Model

Domain Services (Domain Flows)

In this service we may define the same amount of modules as in the UserManagementService - 4, where 2 of them would be identical - DataAccessLayerModule and ApplicationGatewayModule, as these modules' responsibilities are generic and service agnostic. These 2 modules are intended provide border modules for data communication between MessagingService other ones. Eventually their goal is to convert and adjust incoming or outgoing data with its internal representation happening inside this Domain.

MessagingModule in this case this is the core module for the whole service as it manages messages creation, retrieval and editing (if supported). This modules accept data from different sources, including user input, like message's text, user data, obtained from UserManagementService by using User ID, provided by user and using internal logic performs the requested operation - create, retrieve or edit (if supported) message.

The SubscriptionModule handles the complex logic of managing user subscriptions, including creating new subscriptions, handling unsubscribes, and managing subscription statuses like muting or blocking.

Figure 6. Block Diagram of Domain Modules

Infrastructure Concerns

The MessagingService implements several infrastructure-level features to support its operations, as concerns for this service should be different than for the UserManagementService. These include setting up message queues for handling high volumes of new messages, implementing caching mechanisms for frequently accessed subscriptions, and managing database connections for efficient data retrieval.

The service also may publish domain events such as MessageCreated and SubscriptionChanged to be consumed by other parts of the system or external services for features like notifications or analytics.

By structuring the MessagingService according to DDD principles, we create a robust and scalable system for handling the core functionality of our Messaging application. This design allows for efficient management of messages, subscriptions, while providing clear boundaries for future extensions and optimizations.

Scalability and Performance Considerations

As our application grows in popularity, scalability and performance become critical concerns. This section explores strategies to ensure our microservices architecture can handle increasing loads while maintaining responsiveness and reliability. Generally speaking those techniques below are not critical for application functionality, however they become those when the system starts experiencing the high load.

Horizontal Scaling

Both the UserManagementService and MessagingService are designed to be stateless, allowing for easy horizontal scaling. As user traffic increases, we can deploy multiple instances of each service behind a load balancer. This approach distributes the load across multiple servers, improving overall system capacity and resilience.

Database Scaling

As the volume of data grows, database performance can become a bottleneck. To address this, we may implement several strategies for that.

For the UserManagementService, we establish a system of read replicas, as this service load assumes prevail retrieval operations over writing ones. This setup allows us to distribute read operations across multiple database instances, significantly reducing the load on the primary database. By directing read-heavy operations to these replicas, we ensure that the primary database can focus on write operations, thereby improving overall system performance and responsiveness.

The MessagingService, which deals with a substantially larger volume of data, requires a more robust scaling solution. Here, we may implement database sharding, a technique that involves partitioning data across multiple database instances based on user ID or message ID ranges. This approach not only distributes the data itself but also spreads the query load across multiple servers. As a result, we can handle a much larger number of concurrent operations and store a greater volume of data than would be possible with a single database instance.

In addition to these structural changes, we should place a strong emphasis on query optimization and indexing. Our database administrators and developers work in tandem to continuously monitor query performance, identifying frequently used query patterns and ensuring that appropriate indexes are in place to support them. This ongoing process of refinement helps maintain database efficiency even as the volume and complexity of data grow.

Caching Strategies

Caching plays a pivotal role in performance optimization strategy, serving as a crucial layer between our services and the database. In the UserManagementService, we implement a sophisticated caching mechanism for user profile data. We may utilize any in-memory storage, like Redis, for high-performance in-memory data store, where we may cache frequently accessed user information. This approach significantly reduces the number of database queries required for common operations, such as retrieving user details for display or authentication purposes.

The MessagingService employs a more complex, multi-tiered caching strategy to handle the high-volume, real-time nature of social media feeds. We may use a combination of in-memory caching with Caffeine for the most recent and frequently accessed feed entries, providing near-instantaneous access to this hot data. For older or less frequently accessed feed data, we leverage Redis as a second-level cache. This tiered approach allows us to balance the need for ultra-fast access to recent content with the ability to efficiently store and retrieve a larger volume of historical data.

As our system scales horizontally, maintaining cache consistency across multiple service instances becomes critical. To address this, we configure Redis in cluster mode, ensuring that our caching layer is both distributed and consistent. This setup allows us to scale our caching infrastructure in tandem with our application services, maintaining performance as user numbers grow.

Eventual Consistency and CAP Theorem Considerations

In designing our distributed system, we've had to grapple with the fundamental trade-offs described by the CAP theorem, which states that in the event of a network partition, a distributed system must choose between consistency and availability. Given the nature of our application, where the immediacy of information flow is a key feature, we've opted to prioritize availability and partition tolerance over strict consistency in many scenarios.

Figure 7. CAP Theorem Problem

This decision manifests in our embrace of eventual consistency. For instance, when a user updates their profile information in the UserManagementService, we acknowledge and apply this change immediately in that service. However, we accept that there may be a short delay before this update is reflected in the MessagingService or in cached data across the system. During this brief period, different parts of the system may have slightly different views of the user's data.

While this approach introduces the possibility of temporary inconsistencies, it allows our system to remain highly available and responsive, even in the face of network issues or high load. We mitigate the impact of these inconsistencies through careful system design, such as including timestamps with data updates to help resolve conflicts, and by setting appropriate user expectations about the speed of data propagation.

This eventual consistency model extends to other areas of our system as well, such as follower counts or message distribution. By relaxing the requirement for immediate, system-wide consistency, we can process high volumes of operations more efficiently, leading to a more scalable and responsive system overall. However, we're also careful to identify those areas of our application where strong consistency is necessary, such as in financial transactions or security-critical operations, and design those components accordingly.

Conclusion

The journey of creating a Twitter-like application using Domain-Driven Design (DDD) and microservices architecture offers valuable insights into building complex, scalable systems. Through our exploration of the UserManagementService and MessagingService, we've demonstrated how DDD principles can guide the development of a robust, maintainable, and flexible application architecture.

By focusing on clearly defined bounded contexts, we've created services that are independently deployable and scalable, yet cohesive in their overall functionality. The use of aggregates, repositories, and domain services has allowed us to encapsulate complex business logic within each service, promoting code that is both more understandable and more closely aligned with the underlying business domain.

Our approach to data management, including the implementation of the database-per-service pattern and sophisticated caching strategies, showcases how modern distributed systems can handle large volumes of data while maintaining performance and responsiveness. The adoption of event-driven architecture for inter-service communication highlights the power of loose coupling in creating systems that are both resilient and adaptable to change.

Throughout this process, we've had to make important architectural decisions, balancing concerns such as data consistency, scalability, and system complexity. Our choice to embrace eventual consistency in certain areas of the application demonstrates the nuanced thinking required when designing distributed systems, taking into account the trade-offs described by the CAP theorem.

While our implementation focuses on core functionalities, omitting features like notifications and commenting, the architecture we've designed provides a solid foundation for future expansion. The principles and patterns we've applied can be extended to accommodate new features and services as the application grows.

It's important to note that the journey doesn't end with implementation. Continuous monitoring, performance optimization, and iterative improvement are crucial for maintaining and evolving such a system. As user behavior changes and new technologies emerge, our Twitter-like application must be ready to adapt and scale accordingly.

The combination of Domain-Driven Design and microservices architecture offers a powerful approach to building complex, scalable applications. By focusing on clear domain boundaries, embracing eventual consistency where appropriate, and leveraging modern technologies for data management and inter-service communication, we can create systems that are not only capable of handling current demands but are also well-positioned to evolve with future needs.

DEV Community: Eugene Zimin

Drawing the Blueprint: Flowchart, Functional Diagram, and Sequence Diagram

Lesson 3 of Build a Twitter Clone - A Practical Guide to Software Modelling

Introduction - The Foundation Is Ready

A Quick Reminder: The Three-Lens Progression

Lens 1: Behaviour - Drawing the Flowchart

Building it step by step

The flowchart

What this diagram tells us - and what it doesn't

Lens 2: Structure - Reading the Functional Diagram off the Flowchart

The method: scan every step, ask one question

The functional diagram

What this diagram is - and isn't

Lens 3: Interaction - The Sequence Diagram

From components to lifelines

Reading the flow

The sequence diagram

What this diagram shows that the others couldn't

Limitations and Open Questions

What's Next

The Blueprint Beneath the Blueprint: Designing Data Model and Choosing Its Database

Lesson 2 of Build a Twitter Clone - A Practical Guide to Software Modelling

Introduction - Data Before Diagrams

One System, Two Databases - and Why That's the Right Call

The problem with putting everything in one place

The principle: each domain owns its data

The idea behind the split: bounded contexts

Reading the Use Cases for Data Clues

Post a message

View the timeline

Register an account

Naming the Entities

Defining the Fields

A note on identifiers

users - the identity record

Supporting the User: roles and sessions

messages - the content record

subscriptions - the social graph

How the Entities Relate

What the Diagrams Will Now Be Built On

Conclusion

From Idea to Blueprint: Turning a Vague App Concept into Something You Can Actually Build

Lesson 1 of Build a Twitter Clone — A Practical Guide to Software Modelling

Introduction — From Idea to Blueprint

Why an idea isn't a plan

Diagrams: thinking you can point at

What this series is — and what this lesson does

A deliberately small project

Meet the Project: Bird

The hardest decision is what to leave out

One use case becomes our worked example

Three Lenses, One Progression

The three questions

Why the order is not arbitrary

The thread you'll be able to trace

Closing this lesson's opening half

API Authentication: Part III. JWT Tokens

Why API Keys Aren't Always Enough

The hidden cost of a roundtrip

The core idea behind JWT

A Few Words of History

The world before: the session cookie

The pressure that created JWT

RFC 7519: the standard

From standard to ubiquity

Why the history matters

The Anatomy of a JWT

Three parts, two dots

Why it looks like gibberish: Base64URL

Part 1: The header

Part 2: The payload

Part 3: The signature

Building one by hand

The production path: use a library

Claims: What a Token Actually Says

A claim is just a statement

Registered claims: the standard vocabulary

The temporal claims: iat, nbf, and exp

Custom claims: your own data

Avoiding name collisions

`users` - the identity record

Supporting the `User`: roles and sessions

`messages` - the content record

`subscriptions` - the social graph

Meet the Project: `Bird`

The temporal claims: `iat`, `nbf`, and `exp`

The algorithm confusion attack: `alg: none`