DEV Community: Ansh Dhanani

Distributed Systems and Raft Consensus

Ansh Dhanani — Fri, 09 Oct 2026 17:53:34 +0000

A single process is relatively easy to reason about. It has one memory space, one execution flow, and usually one place to inspect when something goes wrong.

Now put the application on several machines. They communicate over a network, store copies of data, and keep serving requests while machines restart or connections fail. You get more capacity and potentially better availability, but you also get a new set of problems: messages can be delayed, machines can disappear, and two nodes can disagree about what happened.

now, for anyone first time, It can be tough to visualize how this actually works in practice, but don't worry we will break it all down step by step. By the end of this guide, you won't need to look anywhere else.

This is the world of distributed systems.

Raft is one way to solve a particularly important part of that problem: getting a group of machines to agree on an ordered sequence of changes, even when some machines fail.

The central problem is not making every machine identical at every instant. It is making the system behave consistently despite delays and failures.

1. What is a distributed system?
2. Why distributed systems are hard
3. Common distributed system building blocks
4. Replication and consistency
5. The CAP theorem
6. What is consensus?
7. Why Raft?
8. Raft cluster architecture
9. Leader election
10. Log replication
11. Terms, log matching, and safety
12. Committing entries
13. Client requests and linearizability
14. What happens when nodes fail?
15. Snapshots and log compaction
16. Membership changes
17. A worked example
18. Raft versus other approaches
19. Where Raft is used
20. Observability and debugging
21. Common mistakes
22. Quick reference
23. Further reading

1. What is a distributed system?

A distributed system is a collection of independent computers that communicate over a network and cooperate to provide a service.

Each machine has its own CPU, memory, operating system, and local view of the world. There is no single shared clock or shared memory that all machines can access instantly.

A simple example is a web application with:

A load balancer receiving incoming requests.
Several application servers processing them.
A database storing application data.
A cache reducing repeated database work.
A message broker handling background jobs.
Monitoring services collecting metrics and logs.

                     Users
                       |
                       v
                 Load Balancer
                  /    |    \
                 v     v     v
              App A  App B  App C
                 \     |     /
                  \    |    /
                   v   v   v
                  Database
                      |
                      v
                 Persistent Disk

These components run separately, but together they provide one application.

Why not use one powerful machine?

A single machine can be a sensible starting point. Distributed systems add complexity, so distributing everything without a reason is usually a bad trade.

Multiple machines can help when you need:

More capacity: split work across several machines.
Fault tolerance: keep serving requests when a machine fails.
Geographic distribution: place services closer to users.
Isolation: separate workloads with different resource needs.
Independent scaling: scale a busy component without scaling everything else.

But adding machines does not automatically make a system reliable. It introduces network failures, coordination, partial failures, replication lag, and operational overhead.

The important idea: partial failure

In a local program, if a function returns, throws an exception, or crashes, the process can often determine what happened.

In a distributed system, a request may be sent to another machine and then time out. The caller cannot immediately know whether:

The remote machine never received the request.
The machine received it but has not processed it yet.
The machine processed it, but the response was lost.
The machine crashed during processing.

Client                 Server
  |                       |
  |------ Request ------->|
  |                       | Process request
  |                       | Save result
  |<----- Response -------X  Response lost
  |
  | Timeout
  |
  | "Did the operation happen?"

The client sees a timeout. The server may have completed the operation successfully.

This is a partial failure: one part of the system cannot tell whether another part is working, failed, or simply slow.

A timeout is evidence that a response did not arrive in time. It is not proof that the operation did not happen.

This distinction matters for payments, job processing, database writes, and any operation that should not be performed twice.

2. Why distributed systems are hard

The hard part is not sending a message. The hard part is deciding what to do when messages and machines do not behave as expected.

Network delay is unpredictable

A message might arrive in milliseconds, arrive much later, or never arrive because a connection has broken. A slow network can look similar to a failed machine.

There is no universal timeout that perfectly distinguishes a slow node from a dead node. A short timeout detects failures quickly but may trigger unnecessary elections. A long timeout avoids some false suspicions but delays recovery.

Machines can fail independently

Consider a cluster with three nodes:

Node A: healthy
Node B: crashed
Node C: healthy

The cluster may continue operating if its protocol permits a majority of nodes to make progress.

Now consider a network partition:

        Partition
   A --------X-------- B
   |                   |
   |                   |
   C                   D

Nodes on one side may communicate normally with each other but cannot reach the other side. Both sides might believe the other side has failed.

A safe protocol must prevent both sides from independently making conflicting decisions that are treated as committed.

There is no perfect global clock

Machine A's clock can differ from Machine B's clock. Even if both clocks are synchronized periodically, there can be drift and synchronization error.

Therefore, timestamps alone are not a reliable way to establish the exact order of events across machines.

Distributed systems often use logical ordering, such as sequence numbers, log indexes, and protocol terms, to reason about events.

Messages can be duplicated or reordered

Depending on the protocol and retry behavior, a request may be retried after a timeout. An application may receive the same logical operation more than once.

A robust design may need:

Request identifiers.
Idempotency keys.
Deduplication records.
Sequence numbers.
Explicit ordering rules.

The system has more than one state

A distributed application may have several different notions of state:

Local memory on each process.
Durable state on disk.
Replicated state on other nodes.
The state currently visible to clients.
State held by caches or downstream services.

These states may temporarily differ. The protocol must define which differences are allowed and when a change becomes authoritative.

Safety and liveness

These two terms appear throughout distributed systems.

Property	Meaning	Example
Safety	Nothing bad happens	Two leaders do not commit conflicting values for the same log position
Liveness	Something good eventually happens	A healthy majority can elect a leader and make progress

A protocol can preserve safety by refusing to do anything when it cannot establish agreement. That may be correct but unavailable.

A protocol that always accepts writes may remain responsive but risk conflicting histories.

Good distributed protocols define which guarantees they preserve and under what assumptions.

3. Common distributed system building blocks

Consensus is important, but it is only one tool in a larger toolbox.

Load balancing

A load balancer distributes requests across multiple servers.

             Request
                |
                v
          Load Balancer
           /    |    \
          v     v     v
        App A  App B  App C

Load balancing can improve capacity and availability, but it does not automatically synchronize application state. If App A stores a shopping cart only in its local memory, App B may not know about that cart.

Replication

Replication stores copies of data on multiple nodes.

             Write
               |
               v
          Primary Node
           /        \
          v          v
      Replica B   Replica C

Replication can improve durability, read capacity, and failover. The exact guarantees depend on how writes are acknowledged and how replicas are kept in sync.

Partitioning or sharding

Partitioning divides a dataset into pieces, often by key.

Users 1-1000       -> Shard A
Users 1001-2000    -> Shard B
Users 2001-3000    -> Shard C

Sharding increases capacity by spreading data across nodes. It introduces challenges around routing, rebalancing, and operations that touch multiple shards.

Message queues

A queue decouples producers from consumers.

Order API -> Message Queue -> Email Worker
                         \-> Billing Worker
                         \-> Analytics Worker

The producer can enqueue work without waiting for every consumer to finish. The system must still account for retries, duplicate delivery, ordering, and poison messages.

Consensus

Consensus helps a group of nodes agree on a value or a sequence of values.

It is useful when the system needs one agreed decision, such as:

Which node is the leader?
Which configuration is active?
What is the committed order of updates?
Which log entries may be applied to the state machine?

Raft is a consensus algorithm designed around replicated logs.

4. Replication and consistency

Replication means keeping multiple copies of data. The central design question is when a write is considered successful and what a reader is allowed to observe.

Synchronous replication

A write is acknowledged only after the required replicas have confirmed it.

Client
  |
  v
Leader
  |------ Write ------> Replica B
  |------ Write ------> Replica C
  |<----- Ack ---------|
  |<----- Ack --------------------|
  |
  v
Client receives success

The actual acknowledgement rule depends on the system. Waiting for more replicas can improve durability, but it may increase latency and reduce availability when replicas are unreachable.

Asynchronous replication

The primary acknowledges a write before all replicas have received it.

Client -> Primary -> Client success
             |
             | replication continues later
             v
          Replica

This can reduce write latency, but a primary failure may cause recently acknowledged data to be missing from a replica that is promoted.

Strong and eventual consistency

Strong consistency is a broad phrase used for systems that provide stronger guarantees about what reads can observe. The exact guarantee should be named rather than assumed.

For example, linearizability requires each operation to appear as if it took effect at one instant between its invocation and response, while respecting real-time ordering of non-overlapping operations.

Eventual consistency means replicas can temporarily disagree, but if updates stop and communication continues under the system's assumptions, replicas eventually converge.

Neither label automatically tells you every detail about transactions, durability, or failure behavior. Read the actual consistency model.

A concrete example

Suppose an account has a balance of ₹1,000.

A client sends a debit of ₹200. The primary updates its copy to ₹800 and acknowledges the request. An asynchronous replica still has ₹1,000.

If the primary crashes and the stale replica is promoted, a read might return ₹1,000. Depending on the design, the acknowledged debit may have been lost.

With consensus-backed replicated state, a write can be acknowledged only after the protocol's commit condition is met. This reduces the risk of promoting a node whose log is missing a committed entry.

Consensus does not automatically make every application operation correct. The application still needs appropriate transaction logic, validation, and request retry handling.

5. The CAP theorem

CAP describes a trade-off that appears when a distributed data system experiences a network partition.

Consistency (C): in the usual CAP formulation, operations behave as if there is a single up-to-date copy.
Availability (A): every request to a non-failing node receives a response, even if that response cannot include the latest write.
Partition tolerance (P): the system continues operating despite messages being lost or delayed between groups of nodes.

When a partition occurs, a system cannot guarantee both linearizable consistency and availability for every request. It must sometimes reject or delay operations on one side to preserve consistency.

                 Network partition
              Node A       Node B
                 |             |
                 X  no link    X

Consistency-first:
  One side may reject writes until agreement is possible.

Availability-first:
  Both sides may accept writes, risking conflicting updates.

A few important qualifications:

CAP is about behavior during a partition, not a universal rule that every system is simply "CP" or "AP" at all times.
Real systems make choices per operation, data type, and failure condition.
A timeout does not prove a partition. It is how a client or node detects that progress is not happening quickly enough.
CAP does not mean performance, latency, or durability are irrelevant. It addresses a specific consistency and availability trade-off under partition.

Raft generally chooses safety over accepting conflicting writes. A Raft group needs a majority to elect a leader and commit new entries. A minority partition cannot independently commit entries.

6. What is consensus?

Consensus is the problem of getting multiple nodes to agree on a value despite failures and unreliable communication.

Imagine five machines must agree on the next configuration version. Each machine may have a different initial view, and some machines may crash or be unreachable.

A consensus protocol aims to ensure that:

Agreement: nodes do not decide on conflicting values.
Validity: a decided value must satisfy the protocol's validity rules.
Termination: under the protocol's assumptions, nodes eventually decide.

Exact definitions differ between consensus formulations. In replicated state machines, consensus is usually applied repeatedly to establish an ordered sequence of commands.

Why a simple vote is not enough

Suppose three nodes vote on whether to write a value. If a response is delayed, one node might assume a peer has failed. If each side of a partition can decide independently, they might accept conflicting results.

A consensus protocol needs explicit rules for:

Who may propose a value.
How many nodes must acknowledge it.
How to handle competing proposals.
What happens after a leader or voter crashes.
How a newly elected leader learns which values were already decided.

Raft combines leader election with replicated log rules to answer these questions.

Majority quorum

For a cluster of (N) voting nodes, a majority quorum is:

[
Q = \left\lfloor \frac{N}{2} \right\rfloor + 1
]

Voting nodes	Majority required	Nodes that can fail while a majority remains
1	1	0
3	2	1
5	3	2
7	4	3

A majority is important because any two majorities in the same fixed voting configuration overlap in at least one node.

For five nodes, any majority contains three nodes. Two groups of three selected from the same five nodes must share at least one member.

This overlap helps preserve safety, but it is not the whole proof. Raft also relies on terms, voting restrictions, and log freshness rules to ensure a newly elected leader cannot overwrite committed history.

For a fixed Raft group, losing the majority means losing the ability to commit new entries. The minority may still run, but it cannot safely act as an independent leader.

7. Why Raft?

Raft is a consensus algorithm introduced as an alternative to Paxos with an emphasis on understandability. It is commonly used to build replicated state machines.

The core idea is straightforward:

Elect one leader.
Send client commands to that leader.
Replicate the leader's log entries to followers.
Commit entries when the protocol's quorum and log rules permit.
Apply committed entries to each node's state machine in order.

                 Client write
                      |
                      v
                +-----------+
                |  Leader   |
                +-----------+
                 /         \
                v           v
          +-----------+ +-----------+
          | Follower  | | Follower  |
          +-----------+ +-----------+

The leader coordinates log replication. Followers do not independently accept a competing log as committed.

What Raft does not do

Raft is not:

A load balancer.
A database query engine.
A replacement for networking.
A guarantee that every request always succeeds.
A general-purpose distributed transaction protocol across unrelated services.
A way to make a cluster tolerate unlimited failures.

It provides a foundation for agreement on replicated log entries. The application built on top of it defines what those entries mean.

The replicated state machine model

A state machine takes an ordered sequence of commands and produces state transitions.

For example:

Initial state: balance = 1000

Command 1: Deposit(500)
State:     balance = 1500

Command 2: Withdraw(200)
State:     balance = 1300

Command 3: Deposit(100)
State:     balance = 1400

If every node starts from the same state and applies the same deterministic commands in the same order, every node reaches the same result.

This is the key idea behind a replicated state machine.

             Replicated log
          [C1] [C2] [C3] [C4]
             |    |    |    |
             v    v    v    v
        +-----------------------+
        | Apply commands in     |
        | the same order        |
        +-----------------------+
             |             |
             v             v
         State A        State B

         Same log + same deterministic
         transitions = same logical state

If a command depends on nondeterministic local behavior, such as reading the current clock independently on each node, nodes may diverge. Such values should be captured in the replicated command or otherwise handled consistently.

8. Raft cluster architecture

Raft describes three main node roles.

Leader

The leader:

Accepts client commands.
Appends commands to its local log.
Replicates log entries to followers.
Tracks replication progress.
Determines when entries can be committed.
Sends heartbeats to maintain leadership.

There should be at most one leader whose authority is accepted for a given term. During elections or partitions, nodes may temporarily have different beliefs about leadership, but Raft's safety rules prevent conflicting leaders from committing conflicting entries.

Follower

A follower:

Responds to leader heartbeats.
Accepts log entries from the leader.
Votes in elections when eligible.
Starts an election if it does not hear from a valid leader within its election timeout.

Most of the time, a follower is waiting for messages or persisting replicated state.

Candidate

A candidate is a node trying to become leader. It starts an election, increments its term, votes for itself, and requests votes from other nodes.

If it receives votes from a majority, it becomes leader. If it learns of a newer term, it steps down. If no candidate wins, another election may begin after a timeout.

Node state

A Raft implementation stores persistent and volatile state. The exact internal implementation varies, but the classic Raft description includes the following.

Persistent state, which must survive a crash:

Field	Meaning
`currentTerm`	Highest term the node has observed
`votedFor`	Candidate that received this node's vote in the current term, if any
`log[]`	Replicated log entries

Volatile state:

Field	Meaning
`commitIndex`	Highest log index known to be committed
`lastApplied`	Highest log index applied to the state machine

The leader also tracks replication progress for each follower, commonly with fields like nextIndex[] and matchIndex[].

nextIndex[follower] is the next log index the leader plans to send to that follower.
matchIndex[follower] is the highest index the leader knows that follower has replicated.

These indexes are not interchangeable. nextIndex is a replication guess that can move backward after a rejection. matchIndex records confirmed replication progress and is used when calculating commitment.

Terms

Raft divides time into terms, numbered with increasing integers. A term begins with an election and may contain a leader.

Term 1             Term 2             Term 3
--------           --------           --------
Leader A           Leader C           Leader B
   |                   |                  |
   v                   v                  v
 log entries        log entries       log entries

A term can end without a successful leader being elected. Several candidates may time out and start elections in successive terms.

Terms act like a logical clock for leadership changes. If a node receives a message with a term higher than its current term, it updates its term and returns to follower state.

The three main RPCs

The Raft paper describes three core remote procedure calls:

RPC	Purpose
`RequestVote`	Ask peers to vote during an election
`AppendEntries`	Replicate log entries and send heartbeats
`InstallSnapshot`	Transfer a snapshot to a follower that needs compacted history

An empty AppendEntries request can act as a heartbeat. Implementations may use additional internal messages, but these are the central Raft RPCs.

9. Leader election

Leader election happens when a follower stops receiving valid heartbeats for long enough.

Election flow

Follower timeout
      |
      v
Increment currentTerm
      |
      v
Become Candidate
      |
      v
Vote for self
      |
      v
Send RequestVote to peers
      |
      v
Receive responses
      |
      +-------------------------------+
      |                               |
      v                               v
Majority votes                    Higher term seen
      |                               |
      v                               v
Become Leader                   Become Follower

If the candidate receives votes from a majority, it becomes leader for that term.

If it discovers a higher term, it steps down. If it does not win, the election may time out and restart.

What is in a RequestVote?

Conceptually, a vote request includes:

The candidate's current term.
The candidate's identity.
The index of its last log entry.
The term of its last log entry.

The receiving node grants its vote only if the request satisfies the protocol rules. Important checks include whether the candidate's term is current enough, whether the voter has already voted for someone else in that term, and whether the candidate's log is at least as up to date as the voter's log.

The last check is essential. A candidate cannot win merely by collecting votes if its log is too stale.

Why random election timeouts?

If every follower starts an election at exactly the same time, votes can split across candidates. No candidate may receive a majority.

Randomized election timeouts reduce the chance that several nodes start elections together.

For example, a system might configure election timeouts in a range such as 300 to 600 milliseconds. Those numbers are illustrative, not universal recommendations. Real values depend on network latency, disk persistence, scheduling pauses, and the implementation.

Node A timeout: 420 ms
Node B timeout: 570 ms
Node C timeout: 360 ms

Node C times out first and starts an election.
If it gets a majority, the other nodes hear from the new leader
and do not need to start competing elections.

Heartbeats

A leader periodically sends empty AppendEntries messages to followers.

These heartbeats:

Tell followers the leader is still active.
Carry the leader's current term.
Help followers reset their election timers.
Can communicate the leader's commit index.

Heartbeats are not proof that every follower has replicated every entry. They establish contact and help maintain leadership. Log replication still needs to succeed.

Why election timeouts are not failure detectors

If a follower times out, it suspects the leader may be unavailable. The leader could actually be healthy but delayed by CPU starvation, a long pause, a network problem, or slow disk I/O.

Raft can tolerate this uncertainty because safety does not depend on the timeout being perfect. Timeouts affect how quickly a new leader is elected and how often elections happen. Terms, voting rules, and log restrictions protect safety.

10. Log replication

Once elected, the leader handles client commands by appending them to its log and replicating them to followers.

Each log entry contains a command and the term in which that entry was created. Entries also have an index, which defines their position in the log.

Example:

Index	Term	Command
1	1	`SetMode("safe")`
2	1	`CreateUser(42)`
3	2	`Deposit(500)`
4	2	`Withdraw(100)`

The same index and term information helps Raft check whether two logs agree.

The normal write path

Suppose a client sends Deposit(500) to the leader.

Client          Leader          Follower B       Follower C
  |                |                 |                 |
  | Deposit(500)   |                 |                 |
  |--------------->|                 |                 |
  |                | Append locally  |                 |
  |                |------ AppendEntries ------------->|
  |                |----------------->|                 |
  |                |                 | Persist entry    |
  |                |<----- success --|                 |
  |                |<---------------------- success ---|
  |                |                 |                 |
  |                | Majority has replicated entry     |
  |                | Mark entry committed, if rules allow
  |<---------------|                 |                 |
  | Success        |                 |                 |

The diagram shows the main idea. A production implementation must handle persistence ordering, retries, rejections, terms, and commit rules correctly.

Step by step

The leader receives the command.
The leader appends it to its local log.
The leader sends AppendEntries requests to followers.
Followers check that the incoming entries match the preceding log position and term.
Followers append missing entries and persist the required state.
The leader tracks successful replication.
Once the relevant commitment rule is satisfied, the entry becomes committed.
The leader applies committed commands to its state machine and informs followers of the commit index.
The client receives a response according to the implementation's acknowledgement contract.

A leader does not need every node to respond before making progress. It needs the required quorum.

AppendEntries consistency check

An AppendEntries request includes the index and term of the entry immediately preceding the new entries. The follower checks whether its log contains an entry at that index with the same term.

If it does not, the follower rejects the request. The leader adjusts its replication position and retries with an earlier prefix until the logs match, then sends the missing entries.

This is how a new leader repairs divergent, uncommitted log suffixes.

Logs can differ temporarily

A follower may be behind the leader. It may also contain entries from an older leader that were never committed.

Raft does not require every log to match at every moment. It requires committed history to be preserved and provides a mechanism for bringing follower logs into agreement.

11. Terms, log matching, and safety

The most important Raft properties are not just about copying entries. They are about ensuring that copied entries represent one safe history.

Log index and term

An entry has an index and a term.

The index identifies its position in the log.
The term identifies the leadership term in which it was created.

Two entries with the same index and term must contain the same command under Raft's log-matching property. If two logs contain an entry with the same index and term, all preceding entries in those logs must also match.

This allows a leader to establish a matching prefix before repairing a follower's log.

Election restriction

A candidate's log must be at least as up to date as the log of a node granting it a vote.

The comparison uses the term of the last log entry first. A log whose last entry has a higher term is considered more up to date. If the last terms are equal, the log with the greater last index is more up to date.

This restriction helps ensure a leader elected by a majority contains the history required to preserve committed entries.

Why a majority alone is not the full explanation

It is tempting to say, "A majority replicated the entry, so it is safe forever." The full Raft reasoning is more specific.

A majority matters because future majorities overlap with the earlier majority. But the node in that overlap must also follow the voting and log-freshness rules. Those rules help ensure a future leader has the necessary committed history.

The protocol's safety argument depends on the combination of:

Majority quorums.
Persistent terms and votes.
The log-matching property.
Election restrictions.
The rule for advancing the commit index.

Important Raft safety properties

Property	Practical meaning
Election Safety	At most one leader can be elected in a given term
Leader Append-Only	A leader does not overwrite or delete entries from its own log
Log Matching	Matching index and term imply matching entries and prefixes
Leader Completeness	A leader elected in a later term contains entries committed in earlier terms
State Machine Safety	No two nodes apply different commands at the same log index

These properties work together. A correct implementation must preserve the required state and ordering rules, including across crashes.

Persistent state and crashes

Suppose a node grants a vote, then crashes and restarts. If it forgets that vote, it could vote for another candidate in the same term. That could break election safety.

This is why Raft's persistent state must be stored reliably before the node responds in ways that depend on it. Details such as write-ahead logging, fsync behavior, and storage guarantees are implementation concerns, but they are not optional details for a correct implementation.

12. Committing entries

An entry is committed when Raft's commitment rules establish that it is part of the agreed history. Committed entries are safe to apply to the state machine.

The leader tracks which entries have been replicated on a majority. For each follower, matchIndex records the highest entry index known to be replicated there.

For a fixed configuration of five voters, an entry replicated on three voters has majority replication. However, the leader's commit rule has an important term restriction.

The current-term rule

In the standard Raft algorithm, a leader advances commitIndex directly based on majority replication only when the candidate entry is from the leader's current term.

Why? An entry from an earlier term may have been replicated on a majority at some point but still be vulnerable under the protocol's rules if treated as committed only by counting replicas. Committing a current-term entry establishes the required history, and earlier entries in the matching prefix become committed along with it.

This is a subtle but important distinction. Do not reduce Raft's commit rule to "any entry on a majority is always committed."

Example with five nodes

Suppose the cluster has five voters:

Leader A
Follower B
Follower C
Follower D
Follower E

A majority is three nodes.

If A, B, and C have replicated a current-term entry and the required protocol conditions are met, the leader can commit it. D and E can catch up later.

If A can only communicate with B, it has two nodes and cannot commit new entries. It must not treat its own local log as a committed decision.

Committed versus applied

These terms mean different things:

Replicated: an entry exists in more than one node's log.
Committed: the protocol has established that the entry is part of the agreed history.
Applied: the node has executed the command against its state machine.

An entry can be committed but not yet applied if the node is still processing earlier entries or recovering after a restart.

Log entries:  [1] [2] [3] [4] [5]
                         ^
                    commitIndex

Applied entries: [1] [2] [3]
                         ^
                    lastApplied

Here, entry 4 may be committed but not yet applied. The state machine must apply entries in order.

Why clients should care

If a server returns success before the operation is safely committed, a leader failure may cause the client to believe a write succeeded even though it is not part of the durable agreed history.

The API's success response should have a clearly defined meaning. In a consensus-backed service, a common contract is to return success after the command is committed and applied, or after the implementation's documented equivalent point.

13. Client requests and linearizability

Clients usually send writes to the leader. A follower may redirect the client to the leader or proxy the request, depending on the implementation.

A basic write path looks like this:

Client
  |
  v
Known endpoint
  |
  +---- follower? ----> redirect or proxy
  |
  v
Leader
  |
  v
Replicate and commit
  |
  v
Apply command
  |
  v
Return result

What if the leader changes during a request?

Imagine a client sends a write to Leader A. A loses contact with the majority. Before the client receives a response, a new leader is elected.

The client may time out and retry. The original operation might have committed, or it might not have. A timeout alone does not resolve the uncertainty.

Applications can address this with:

Unique operation IDs.
Idempotency keys.
A replicated record of completed requests.
Explicit status lookup APIs.
Carefully defined retry behavior.

For example, instead of sending Deposit(500) as an unidentifiable command every time, a client could send Deposit(request_id="r-123", amount=500). The state machine can remember the request ID and return the prior result if the same operation is retried.

The deduplication record must be stored and applied consistently with the operation. Keeping it only in local memory would not survive failover.

Linearizability

Linearizability is a consistency guarantee that makes each operation appear to take effect at one instant between its start and completion, while respecting the real-time order of operations that do not overlap.

A replicated state machine can provide linearizable writes when the complete implementation follows the necessary protocol and client-serving rules.

Reads need care too. A leader that has been isolated from a majority might not know it has lost leadership. Serving a read from stale local state could violate linearizability.

Implementations use techniques such as quorum-confirmed reads or a correctly implemented ReadIndex mechanism. Lease-based reads are another option, but they require assumptions about timing and clock behavior to be satisfied.

Do not assume that every read served by a node calling itself leader is automatically linearizable.

14. What happens when nodes fail?

A distributed system should be designed around failure scenarios, not only the happy path.

One follower crashes in a three-node cluster

A: Leader, healthy
B: Follower, crashed
C: Follower, healthy

A and C are a majority of two out of three. The cluster can continue committing entries if they can communicate and the other required conditions hold.

B can restart later and catch up from the leader.

The leader crashes

Before failure:

       A (Leader)
       /        \
      B          C

A crashes
    X

B and C stop hearing heartbeats.
One times out and starts an election.
A majority elects a new leader.

The cluster may pause while a new leader is elected. Requests during that interval can time out or fail. Raft aims to preserve safety while allowing progress once a majority can communicate.

A minority partition

For five voters, imagine a partition splits the cluster into groups of three and two.

Partition 1                  Partition 2
+------------------+         +------------------+
| A  B  C          |    X    | D  E             |
| majority: 3      |         | minority: 2      |
+------------------+         +------------------+

The three-node side can elect a leader and commit new entries. The two-node side cannot commit new entries because it cannot reach a majority.

If the minority still believes an old leader is active, it may accept a connection or hold uncommitted local state depending on the implementation, but it cannot safely commit new entries independently.

When the partition heals, the current leader reconciles followers' uncommitted log suffixes.

All nodes lose their majority

If only one node in a three-node cluster is reachable, it cannot commit new entries. It may have a local log and may still be running, but it cannot establish quorum.

This is intentional. Rejecting or delaying writes is preferable to accepting conflicting histories.

A slow disk

A node can be alive at the process level but unable to persist entries quickly. This may delay acknowledgements, cause timeouts, and trigger elections.

For a consensus implementation, disk latency is not merely a storage performance concern. Persistent state and log entries are part of the correctness protocol.

Restart and recovery

After restart, a node recovers persistent state, including its term, vote, and log. It learns the current leader and reconciles its log.

A node must not treat uncommitted local entries as committed simply because they survived on disk. Commitment depends on the protocol's rules and information from the current cluster.

Failure tolerance by cluster size

Voters	Majority	Maximum unavailable voters while retaining a majority
3	2	1
5	3	2
7	4	3

More voters increase failure tolerance but also increase the number of machines involved in elections and the potential communication overhead. In many deployments, three or five voters are common choices. The right number depends on failure domains, latency, operational requirements, and implementation guidance.

A five-node cluster in one rack does not protect against losing the entire rack. Place replicas across independent failure domains when the infrastructure and latency requirements allow it.

15. Snapshots and log compaction

A replicated log grows as commands are added. Keeping every command forever makes storage usage grow without bound.

Raft implementations can use snapshots to compact old log entries.

A snapshot captures the state machine state through a particular log index and term. Once the snapshot is safely stored, entries covered by it can be discarded according to the implementation's rules.

Before snapshot:

Log: [1] [2] [3] [4] [5] [6] [7] [8] [9]

Snapshot covers entries through index 6.

After compaction:

Snapshot at index 6
Log:                  [7] [8] [9]

The snapshot must include enough metadata to establish the last included index and term. That allows Raft to continue matching logs after older entries have been removed.

What if a follower is far behind?

If a follower's missing entries have already been compacted from the leader's log, the leader cannot repair it by sending those old entries one by one. It may need to send a snapshot using InstallSnapshot.

The follower installs the snapshot according to the protocol and then continues receiving log entries after the snapshot's index.

Snapshot considerations

Snapshot data must correspond to a consistent state machine position.
The last included index and term must be retained.
Installation should not expose partially written state.
The implementation must handle crashes during snapshot creation or installation.
Snapshot size and frequency affect disk usage, recovery time, and network traffic.

Snapshots are a storage and recovery mechanism. They do not replace the consensus protocol.

16. Membership changes

What if a three-node cluster needs to become a five-node cluster? Changing membership is not just a matter of adding two machines to a list.

If old and new configurations used unrelated quorum rules at the same time, different groups could potentially make conflicting decisions.

Raft handles membership changes through a protocol designed to preserve quorum safety during transitions. The original Raft approach uses joint consensus, where the system temporarily requires agreement under both the old and new configurations.

Conceptually:

Old configuration: {A, B, C}

Transition:
  Joint configuration includes old and new membership rules.

New configuration: {A, B, C, D, E}

During the joint phase, decisions require the necessary majorities of both configurations. Once the transition is committed, the new configuration becomes active.

Implementations may use different, carefully designed membership-change procedures. Follow the specific implementation's documentation rather than manually changing voter lists while the cluster is running.

Why adding a node is not enough

A new node usually needs to:

Be provisioned with the correct identity and configuration.
Join through the supported membership-change mechanism.
Receive the log or a snapshot.
Catch up with the cluster.
Become a voter only when the protocol permits it.

A node that is present on the network is not automatically a safe voting member.

17. A worked example

Let's walk through a three-node cluster that stores a simple account balance.

Initial state:

Node A: Leader
Node B: Follower
Node C: Follower

Balance = 1000
Commit index = 4

All three nodes have applied log entries through index 4.

Step 1: A client requests a deposit

The client sends:

{
  "requestId": "deposit-001",
  "type": "Deposit",
  "amount": 500
}

The leader appends a new log entry:

Index	Term	Command	Status
5	3	`Deposit(500)`	Appended on A

The entry is not committed merely because A wrote it locally.

Step 2: The leader replicates the entry

A sends AppendEntries to B and C.

Suppose B acknowledges after persisting the entry. The entry now exists on A and B.

A: [1] [2] [3] [4] [5]
B: [1] [2] [3] [4] [5]
C: [1] [2] [3] [4]

Two of three voters have the entry. If it is from the current term and the required Raft conditions hold, A can advance the commit index to 5.

C can be temporarily behind.

Step 3: The entry is committed and applied

A applies Deposit(500):

Balance: 1000 + 500 = 1500

A can return the result according to its client acknowledgement contract.

Later, A informs C of the updated commit index, and C applies the same command after replicating it.

A: Balance = 1500
B: Balance = 1500
C: Balance = 1500

All nodes apply the same committed command in the same order.

Step 4: The leader fails

A crashes before C catches up.

B and C remain. If they can communicate, they form a majority and can elect a new leader. B already has the committed entry at index 5, so the election restriction helps prevent a candidate missing committed history from winning.

After the new leader is established, the cluster continues from the agreed history.

Step 5: The client retries after a timeout

Suppose the client never received the success response from A. It cannot know whether the deposit was committed.

It retries with requestId: "deposit-001".

If the state machine records request IDs with their results, the new leader can recognize the retry and return the previous result instead of applying the deposit twice. Without a deduplication design, retrying a non-idempotent command can double the balance.

This is an application-level concern built on top of Raft. Consensus orders commands, but it does not automatically know that two requests from a client represent the same intended operation.

18. Raft versus other approaches

Raft is not the only way to replicate data or coordinate machines. Different tools solve different problems.

Approach	Main idea	Useful when	Important limitation
Primary-replica replication	One primary accepts writes and sends updates to replicas	Read scaling and straightforward failover	Guarantees depend on acknowledgement and failover policy
Raft	Leader replicates a log using majority-based consensus	Strongly coordinated replicated state	Cannot commit without a quorum
Paxos family	Consensus based on proposals, promises, and acceptance	Building fault-tolerant agreement protocols	Variants and implementations can be difficult to reason about
Gossip	Nodes spread information through repeated peer exchanges	Membership, failure suspicion, and eventually convergent metadata	Does not by itself provide a single committed log
Two-phase commit	Coordinator asks participants to prepare and then commit or abort	Coordinating a transaction across participants	Can block under coordinator failures without additional mechanisms
CRDTs	Data types merge concurrent updates using defined rules	Some multi-writer, eventually consistent data	Not suitable for every invariant or transaction

These approaches are not all direct replacements for one another. For example, Raft consensus and two-phase commit solve different problems. A system can use consensus to maintain metadata and still need a transaction protocol for coordinating separate resources.

Raft and Paxos

Both are families of consensus algorithms. Raft's presentation centers on a leader and a replicated log, making its operation easier for many engineers to follow. Paxos has several variants and is foundational to distributed consensus research and systems.

It is not accurate to say that Raft is universally faster or safer than Paxos. Performance depends on implementation, workload, persistence, network topology, batching, and configuration.

Raft and gossip

Gossip is useful for spreading membership and health information. Nodes exchange observations with peers, and information gradually propagates.

But gossip alone does not guarantee that a group has committed one value at one log index. Systems may use gossip for discovery or monitoring while relying on consensus for authoritative decisions.

Raft and distributed transactions

Raft can replicate the state of one state machine. It does not automatically make an operation spanning several independent Raft groups atomic.

For example, transferring money between accounts that live in different shards may require a separate transaction design. Consensus within each shard does not, by itself, guarantee that both sides of the transfer commit together.

19. Where Raft is used

Raft is useful when a service needs a reliable, agreed sequence of changes.

Examples include:

Cluster configuration and membership metadata.
Distributed key-value stores.
Coordination services.
Replicated metadata for storage systems.
Leader election and replicated control-plane state.

Some well-known systems and projects use Raft or Raft-derived designs, including etcd and Consul. Their complete behavior includes additional components, APIs, storage layers, and operational choices, so understanding Raft is a foundation rather than a complete guide to operating those products.

Why Kubernetes uses etcd

Kubernetes stores cluster state in etcd. That state includes information such as Kubernetes objects and configuration used by the control plane.

etcd uses the Raft consensus algorithm to replicate its data among members. This helps the cluster maintain a consistent view of stored state when a member fails, as long as the remaining members can form a quorum.

A simplified view:

kubectl / API clients
         |
         v
  Kubernetes API Server
         |
         v
       etcd
   +------+------+ 
   |      |      |
   v      v      v
 Member Member Member
   A      B      C

   Raft replicates the log
   and coordinates commitment.

A Kubernetes worker node does not normally run its own etcd member just because it is a worker. In a typical self-managed control plane, etcd members run on designated control-plane or dedicated machines according to the cluster design.

Why this matters for platform engineering

When etcd loses quorum, the Kubernetes API may be unable to persist new state changes. Existing workloads might continue running for a while, but operations that require updating cluster state can fail.

This is a useful operational distinction: a cluster's workloads, control plane, and state store are related, but they are not the same component.

20. Observability and debugging

Consensus protocols are easier to operate when you can see what the nodes are doing. A cluster that appears "up" at the process level may still be unable to make progress.

Useful signals

Signal	What it can reveal
Leader changes	Unstable elections or node/network problems
Election count	Repeated timeouts and leadership instability
Request latency	Slow network, disk, queueing, or overloaded nodes
Commit latency	Time between proposal and commitment
Replication lag	Followers falling behind the leader
Disk latency	Slow persistence affecting progress
WAL size	Growth in the write-ahead log
Snapshot duration	Cost of compaction and recovery
Quorum availability	Whether enough voting members can communicate

Metric names vary by implementation. Use the official metrics documentation for the exact service you are operating.

Symptoms and possible causes

Symptom	Possible causes to investigate
Repeated leader changes	Election timeout too low for actual latency, packet loss, CPU pauses, slow disk
Writes time out	No quorum, overloaded leader, slow follower acknowledgements, disk bottleneck
One follower falls behind	Network issue, slow storage, repeated rejection while catching up
Restart takes a long time	Large log, snapshot work, slow storage, recovery overhead
High request latency	Batching behavior, network round trips, fsync latency, queueing

These are starting points for investigation, not definitive diagnoses.

A practical debugging order

Check which node is leader and whether leadership is stable.
Confirm that a majority of voters can communicate.
Inspect node restarts, CPU saturation, memory pressure, and disk latency.
Check network latency and packet loss between members.
Look at replication progress and log or snapshot activity.
Compare client latency with commit latency.
Read the service logs around term changes, rejected appends, and storage errors.

Do not fix election instability by blindly increasing timeouts. First determine whether the cluster has slow disks, overloaded nodes, network problems, or an unrealistic timeout configuration.

Failure testing

A lab cluster is a good place to test failure behavior. With a disposable environment, you can try:

Stopping one follower.
Restarting a follower after it has missed entries.
Interrupting network connectivity between members.
Restarting the leader.
Observing what happens when a majority is unavailable.
Restoring connectivity and watching lagging nodes catch up.

For each test, record what happened to write success, latency, leader identity, and recovery time. Do not run disruptive tests on a production cluster without a reviewed plan.

21. Common mistakes

Treating timeout as proof of failure

A timeout means the expected response did not arrive in time. The remote operation may still have succeeded.

Better: design retry behavior around operation IDs and idempotency where needed.

Assuming replicas are always identical

Replicas can have different uncommitted log suffixes or different applied positions. Raft allows temporary differences while preserving committed history.

Better: distinguish replication, commitment, and application.

Thinking the leader can always accept writes

A leader without a majority cannot commit new entries. It may still be running and reachable by some clients, but it cannot safely progress independently.

Better: monitor quorum and define client behavior when quorum is unavailable.

Assuming a majority alone explains all safety

Quorums are central, but Raft also needs voting restrictions, log matching, terms, persistence, and the current-term commitment rule.

Better: understand the full path from election through log replication to commitment.

Forgetting durable storage

If a node acknowledges protocol actions but loses required persistent state after a crash, safety assumptions can be violated.

Better: follow the implementation's storage and persistence requirements. Treat fsync and write ordering as correctness concerns, not just performance tuning.

Serving every read from the leader without checking

A node that believes it is leader may be isolated or stale. Read guarantees depend on the read protocol.

Better: use the implementation's documented linearizable-read mechanism when linearizable reads are required.

Using Raft for everything

Consensus adds coordination and requires a quorum. It is not needed for every cache, analytics pipeline, or stateless web service.

Better: use consensus where a single agreed history is needed, and simpler designs where weaker guarantees are sufficient.

Running an even number of voters to "get more availability"

A four-voter cluster still needs three votes. It can tolerate only one unavailable voter, just like a three-voter cluster, while requiring more members to participate.

Better: choose the number and placement of voters based on failure domains and the system's documented recommendations.

What to remember

Raft is not about making every node do everything independently. It is about agreeing on one ordered history, then applying that history consistently.

The leader coordinates, the followers replicate, the quorum supports commitment, and the state machine applies committed commands. Elections and retries help the system recover, while the safety rules prevent temporary failures from turning into conflicting committed histories.

Here is a clear, alphabetically organized glossary of key terms covered in your post, designed to be placed at the very end of your guide for a quick and easy reader review:

22. Quick Reference: Key Terms and Meanings

Asynchronous Replication: A write mode where the primary node acknowledges a request immediately after saving it locally, without waiting for replicas to confirm receipt. Faster, but risks data loss if the primary fails.

CAP Theorem: The rule stating that a distributed system can guarantee at most two of three properties during a network partition: Consistency, Availability, and Partition Tolerance.

Clock Drift: The gradual discrepancy that develops between physical clocks on different machines over time, proving that physical timestamps alone cannot reliably order distributed events.

Consensus: The fundamental problem of getting multiple independent nodes to agree on a single value or an ordered sequence of operations despite failures and message delays.

Distributed System: A collection of independent computers connected over a network that communicate and coordinate to provide a single, unified service.

Eventual Consistency: A weak consistency model where replicas temporarily diverge during updates but automatically converge over time once activity ceases.

Idempotency: A property where an operation can be applied multiple times without changing the result beyond the initial application, preventing bugs from duplicate network retries.

Leader Election: The process by which follower nodes timeout and vote to choose a new authoritative coordinator when the existing leader fails.

Linearizability: A strong consistency model requiring all operations to appear atomic and instantaneous at a precise point in time between their invocation and response.

Log Compaction (Snapshots): Truncating old log entries by saving the current state machine state to disk to prevent unbounded storage growth.

Log Replication: The primary mechanism where the leader accepts client commands, appends them to its local log, and forces all follower nodes to match its history.

Majority Quorum: The minimum number of operational nodes (⌊N/2⌋+1) required to safely make cluster decisions and commit logs.

Partial Failure: A confusing state where one part of a system cannot determine whether another component is slow, crashed, or working properly.

Partitioning (Sharding): Splitting a large dataset into distinct, manageable pieces distributed across multiple nodes to scale storage and throughput.

Replication: Storing copies of data across multiple nodes to enhance durability, read capacity, and fault tolerance.

Safety: A system guarantee ensuring that "nothing bad happens" (e.g., two leaders never commit conflicting values for the same log position).

Split-Vote: An election failure scenario where votes are split evenly among multiple candidates, resolved in Raft using randomized timeouts.

Synchronous Replication: A write mode where the primary node waits for a required majority of replicas to confirm a write before acknowledging success to the client.

Term: Raft's logical clock counter used to divide time into discrete election epochs and detect obsolete leaders.

Timeout: Evidence that a network response did not arrive in time, which indicates a potential failure but does not prove the underlying operation failed.

23. Further reading

These are good places to go deeper into the protocol and its real implementations.

The original Raft paper is worth reading after you understand the basic roles, terms, log replication, and commitment rules. The figures and safety arguments explain why the protocol's restrictions exist, not just what messages get sent.

The End

Distributed systems become easier to reason about when you stop assuming that every machine knows the same thing at the same time.

Messages get delayed. Nodes fail. Logs temporarily differ. Raft gives a cluster a disciplined way to elect a leader, replicate commands, and decide which entries are safe to apply.

The interesting part is how much depends on the details: quorum overlap, durable state, log freshness, commitment rules, and how clients behave when responses are lost. Those details are what turn a few machines connected over a network into a system that can be trusted.

i touched AWS and stuff didn't break (mostly)

Ansh Dhanani — Wed, 27 May 2026 09:51:37 +0000

so. i finally sat down and actually did AWS. not just watched tutorials, not just read docs actually clicked buttons, broke things, fixed them, and now i have opinions.

this is a writeup of everything i learned practically in this last 1 week, in the order i learned it, with zero fluff. if you're someone who learns by doing and wants a no-nonsense walkthrough of core AWS concepts you're in the right place.

let's go.

spinning up my first EC2 instance

EC2 is basically "rent a computer from Amazon and it runs 24/7 somewhere in a data center." that's the whole thing. everything else is just configuration.

i launched a t3.micro in ap-south-1 (Mumbai) free tier, which is perfect for learning. named it my web server because i'm creative like that.

what actually matters when launching: the AMI (think: OS), the instance type (think: hardware specs), and your key pair (SSH access). everything else can be changed later. don't overthink it.

once it's running, you get a public IPv4. EC2 Instance Connect lets you SSH directly from the browser which is actually very clean for beginners. no terminal setup needed.

one thing that tripped me up the instance showed "Running" but that doesn't mean your application is running. the VM is up, your code is not. different things.

deploying a "website" (lol)

my first deployed website on AWS was literally this:

Hi, i am Ansh. and this is my first AWS based self deployed website. WOHOOOO!!! let's gooo!!!

one line of HTML. serving it with nginx. accessible on port 80 over HTTP. it's terrible and i loved it.

hitting that raw IP in the browser and seeing that text load that was a real moment. first cloud-deployed thing i've ever built. it's plain text and i don't care, it counts.

the actual commands to get nginx running:

# update, install, start
sudo apt-get update -y
sudo apt-get install nginx -y
sudo systemctl start nginx
sudo systemctl enable nginx

# drop your HTML here
sudo nano /var/www/html/index.html

that's literally it. the web server is now live. the hard part was not the code it was getting the networking right, which brings us to...

security groups - what even is a port?

security groups are basically firewall rules for your EC2 instance. when i first tried hitting my instance's IP in the browser, nothing loaded. timeout. because i hadn't opened port 80 (HTTP).

i set up a security group called ssh-http-web-secgrp with two inbound rules:

1. SSH on port 22 so i can connect to the instance via terminal. source: 0.0.0.0/0 (anywhere) for learning, should be your IP in prod.

2. HTTP on port 80 so the world can access the web server. source: the load balancer's security group, not the open internet. that is how security is built. we only allow what is needed, in this case the security group of the load balancer which is named LB-SG.

actually important thing: security groups are stateful. if you allow inbound on port 80, the response traffic goes out automatically. you don't need a matching outbound rule. this confused me for a bit.

launch templates + auto scaling groups (aka AWS spawning servers for me automatically)

this was the point where AWS stopped feeling like "just a Linux VM in the cloud" and started feeling like actual cloud engineering.

after setting up the load balancer, i wanted the infrastructure to react automatically when traffic increased. not manually launching EC2 instances every time the CPU cried for help.

so i created a Launch Template and connected it to an Auto Scaling Group (ASG).

the mental model finally clicked:

Launch Template = blueprint for creating EC2 instances
Auto Scaling Group = service that decides when to create or terminate them

the template stored everything:

Ubuntu AMI
t3.micro
security groups
key pair
nginx setup
networking config

basically a reusable recipe for servers.

instead of manually configuring instances forever, AWS can now just clone new ones automatically whenever needed.

my ASG config looked like this:

minimum instances: 1
desired instances: 1
maximum instances: 3

which basically means:

always keep at least 1 server alive
normally run 1
but if traffic spikes, scale up to 3

and honestly, this is where cloud finally became interesting.

the architecture looked like this

internet
    ↓
Application Load Balancer
    ↓
Auto Scaling Group
   ↙          ↘
web-1        web-2
ap-south-1c ap-south-1a

traffic comes into the ALB.

the ALB distributes requests across healthy EC2 instances.

CloudWatch monitors metrics like CPU usage.

if usage crosses a threshold, the ASG launches another instance automatically using the Launch Template.

very simple idea conceptually. insanely powerful in practice.

making AWS panic on purpose

then came the fun part.

i SSH'd into web-1 and intentionally tortured the CPU using the stress tool:

sudo apt-get install stress -y

stress --cpu 2 --timeout 600 &
stress --cpu 1 --timeout 600 &
stress --cpu 2 --timeout 60

my scaling policy was configured around:

if CPU > 40%
→ launch another instance

and then i just watched the dashboards.

watching auto scaling happen live

the load balancer monitoring started showing request spikes and activity increasing as the stress test ran.

CPU utilization climbed hard from basically idle.

network traffic increased too because requests were flowing through the ALB into the instances.

and then suddenly another EC2 instance appeared automatically.

i did not press "Launch Instance."

AWS launched it for me.

this tiny terminal command triggered the whole scaling workflow:

high CPU load
    ↓
CloudWatch detects >40%
    ↓
Auto Scaling policy triggers
    ↓
Launch Template used
    ↓
new EC2 instance created
    ↓
Load Balancer adds healthy target

that feedback loop is the foundation of scalable cloud systems.

the thing that changed how i think about servers

before this, my brain thought scaling meant:

"buy a bigger machine"

AWS teaches the opposite mindset.

instead of:

1 huge server

you do:

many smaller servers

horizontal scaling.

because smaller machines are replaceable.

if one dies? the ASG launches another.

if traffic spikes? launch more.

if traffic drops? terminate extras and save money.

that's why load balancers matter so much. users don't connect directly to servers anymore. they connect to the ALB, and the ALB decides where traffic goes.

instances become disposable infrastructure.

and that is a VERY different way of thinking compared to traditional hosting.

honestly the coolest part

the craziest thing is this entire setup was built using just:

EC2
CloudWatch
Load Balancer
Launch Template
Auto Scaling Group

no Kubernetes.

no Docker orchestration.

no Terraform.

just core AWS services working together.

and suddenly the infrastructure can:

scale itself
distribute traffic
recover automatically
survive instance failures

that's the first moment AWS actually felt like "the cloud" to me instead of just "Ubuntu running somewhere else."

IAM - learning what "not authorized" feels like

IAM (Identity and Access Management) is AWS's permission system. users, roles, policies. this one bit me directly.

i was poking around with a second AWS account and tried to view EC2 instances in eu-north-1 (Stockholm). got this:

You are not authorized to perform this operation.
User: arn:aws:iam::590128028909:user/Ansh-2 is not authorized
to perform: ec2:DescribeInstances because no identity-based
policy allows the ec2:DescribeInstances action

makes total sense in hindsight. the user had no IAM policies attached. by default in AWS you get nothing. zero permissions. you have to explicitly grant everything.

this is the opposite of how most beginners think about it. they expect "allow everything by default, block what's dangerous." AWS does "block everything by default, allow what you need." way more secure.

the IAM mental model:

root account = god mode. don't use it for daily stuff.
IAM user = a specific identity with attached policies.
policy = JSON document saying "this action on this resource is allowed/denied."
role = like a user but assumed temporarily (e.g., an EC2 instance acting as a user to access S3).

the ARN (Amazon Resource Name) in that error message is how AWS identifies every resource uniquely. arn:aws:iam::590128028909:user/Ansh-2 = account ID + service + resource.

you'll see ARNs everywhere.

S3 - storing stuff in the cloud like a normal person

S3 (Simple Storage Service) is object storage. not a file system, not a database just a bucket that holds objects (files). infinitely scalable, stupidly cheap, and the backbone of half the internet.

i created a bucket called first-s3-bucket-by-ansh in ap-south-1 and uploaded one image:

maykl-dzhekson-michael-jackson.jpg · 62.2 KB · Standard storage class

(yes that's how it was spelled. yes i kept it.)

first attempt to access it via the public URL, got this:

<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied</Message>
</Error>

because S3 buckets are private by default. again, AWS defaults to locked down.

to make an object publicly accessible you need to:

Disable "Block Public Access" at the bucket level.
Add a bucket policy that allows s3:GetObject for Principal: "*" on the objects.
or just hit "Make public" on the individual object. faster for learning, wrong for production.

after sorting that out, the image loaded. michael jackson silhouette on a black background, served from my S3 bucket and accessible from anywhere in the world.

the URL structure is:

https://[bucket-name].s3.[region].amazonaws.com/[object-key]

S3 is everywhere: static website hosting, ML model weights, application logs, database backups, CDN origin... you'll use S3 in basically every AWS architecture. understand it early.

tldr / what i actually learned

not just "what buttons to click" the mental models that stuck:

AWS defaults to deny. no permissions, no access. you build up from zero.
Availability zones are not optional. run across at least two if anything needs to stay up.
Security groups are stateful firewalls. inbound rules are enough for most setups.
Load balancers + auto scaling = resilience. don't make your instance bigger, make more of them.
IAM is the foundation. understanding ARNs, policies, and roles unlocks everything else.
S3 is not just file storage. it's an architecture primitive.

what's next: RDS for a proper database, CloudFront for CDN, and Route 53 for a real domain instead of raw IP addresses.

also i need to terminate these instances before my free tier runs out.

- Ansh Dhanani

I Accidentally Found a Better Way to Get effective results from AI Agents

Ansh Dhanani — Fri, 15 May 2026 07:32:41 +0000

The Problem I Kept Running Into

Recently I was experimenting with different ways of using AI agents on larger projects and I found a workflow that gives surprisingly better results than normal prompting.

Usually when people use tools like Claude or ChatGPT for coding, they paste a few snippets or files and ask questions directly. That works fine for smaller problems, but once the project becomes bigger, the outputs start getting inconsistent. Sometimes the AI misses obvious architectural problems, sometimes it suggests things that don’t fit the system at all, and sometimes it completely loses track of how different parts of the codebase interact.

After testing this for a while, I realized the issue is usually not the model itself.

The issue is context.

Most current AI coding workflows rely on fragmented context. The model only sees disconnected chunks of the system, so it never develops a real understanding of execution flow, runtime behavior, dependency direction or how services interact together.

The Workflow I Started Using

So I tried something different.

I wrote a small script that scans my repository and bundles important files into one large file. Not literally every file, because that creates too much noise, but the parts that actually explain how the system works. Things like schemas, orchestration logic, parsers, graph systems, runtime handling, caching, workers, dependency management and other core logic.

Then instead of directly asking Claude to analyze the codebase, I first use ChatGPT to generate a much deeper investigation prompt.

Not generic prompts like:

“find bugs”

But prompts focused on things like:

architectural analysis
performance bottlenecks
scaling risks
hidden coupling
unnecessary abstractions
memory pressure
concurrency issues
runtime inefficiencies

Then I paste both the bundled code file and the generated investigation prompt into Claude.

The difference in output quality is honestly huge.

Why I Think This Works

The model starts reasoning about the system much more coherently because it can actually see relationships between different layers of the application.

It starts identifying patterns that are difficult to notice when looking at isolated files — duplicated logic, circular dependencies, architectural inconsistencies, bad retry handling, unnecessary complexity and hidden performance issues.

What became interesting to me is that this doesn’t really feel like “better prompting”.

It feels more like giving the model enough architectural visibility to reason properly.

I think a lot of current AI tools behave like they’re trying to understand a large system while only seeing tiny windows into it. Once the model can access enough of the architecture at once, the quality of reasoning changes significantly.

Where This Approach Breaks

I also noticed there’s a limit to this approach.

If you dump too much irrelevant information into context — logs, generated files, lockfiles, vendor code, build artifacts and other noisy files — the quality drops again. The model starts focusing on the wrong things or missing important details completely.

So now I’ve been experimenting more with what I’d call context engineering instead of just prompt engineering.

Another thing that improved results a lot was feeding structural information before the raw code itself. Things like dependency graphs, import relationships, architecture overviews and execution flow maps help the model build a mental representation of the system before reading implementation details.

That alone improved the consistency of outputs quite a bit.

Context Engineering > Prompt Engineering

Honestly, I think this is where AI-assisted development tools are heading.

Not just bigger models, but better context systems.

Right now most coding agents still behave like they’re reading code through a keyhole. They only see tiny sections at a time, which makes deeper reasoning difficult on larger projects.

Once the model can understand more of the system at once, the analysis becomes much more useful for:

debugging
architecture reviews
performance analysis
scaling research
understanding unfamiliar codebases

This workflow still has limitations and it definitely doesn’t magically solve hallucinations or reasoning errors.

But compared to normal snippet-based prompting, the results have been noticeably better.

Final Thoughts

I’m curious if other people are experimenting with similar workflows too, because it feels like we’re still very early in figuring out how AI should actually interact with large codebases.

A lot of people are focused on better models, but I think context handling is becoming just as important.

And honestly, whoever solves context engineering properly is probably going to build extremely powerful developer tools.

Building My First Self-Hosted Infrastructure Server on Linux

Ansh Dhanani — Wed, 13 May 2026 16:23:21 +0000

I always thought Linux looked cool from the outside, but I never really understood why developers became obsessed with it until I actually started using it properly.

For most of my life, I was just a normal Windows 10 user. My workflow was simple VS Code, browsers, localhost development, and deploying apps without thinking too much about what was happening underneath. Everything worked, but after building more projects, especially AI and web applications, I started feeling disconnected from the actual systems running my applications.

I knew how to build projects.

I didn’t know how infrastructure worked.

That’s when I started exploring Linux seriously.

Discovering Omarchy

After trying different Linux distributions, I found Omarchy and instantly liked it. The system felt extremely fast, minimal, and surprisingly clean. Compared to Windows, everything felt more focused. The terminal experience, window management, keyboard shortcuts, and multiple pane workflow completely changed how I worked.

The coolest part was how efficient everything felt once you got comfortable with it.

I used Omarchy for around two months before eventually breaking the entire setup. One bad configuration later, none of the UI loaded anymore and I was left staring at a shell screen with absolutely no idea how to recover the system properly.

That moment honestly taught me something important.

Linux gives you freedom, but it also expects responsibility.

Even after breaking everything though, I still loved the experience enough to come back again. This time I decided to run Omarchy inside VirtualBox on Windows 10 so I could safely experiment without destroying my main environment again.

Setting Up the VM

Once the virtual machine was running, I finally had a playground where I could:

experiment freely
break things safely
learn networking
build servers manually
understand infrastructure properly

The goal was simple:

Build a small self-hosted infrastructure setup using:

Docker
Nginx
load balancing
networking
benchmarking

Installing Nginx and Creating the Project

Docker was already included with Omarchy, so I mainly needed Nginx.

sudo pacman -S nginx

Then I created the project folder:

mkdir nginx-lb
cd nginx-lb

Inside the folder I created:

3 web apps
an Nginx config
a Docker Compose setup

The structure looked like this:

Each app simply had an index.html file.

app1/index.html

<h1>App 1</h1>

app2/index.html

<h1>App 2</h1>

app3/index.html

<h1>App 3</h1>

Simple, but enough to visually verify load balancing.

Docker Compose Setup

Then I created the Docker Compose configuration using nano.

Omarchy already comes with NeoVim pre-installed, and honestly the setup looks extremely cool. Once you get comfortable with it, the workflow feels insanely fast. But in real server environments, especially when you're quickly editing configs over SSH, you'll often see people using something simple like nano.

It’s quick, lightweight, always available, and efficient for fast configuration changes.

version: '3'

services:
  app1:
    image: nginx
    volumes:
      - ./app1:/usr/share/nginx/html

  app2:
    image: nginx
    volumes:
      - ./app2:/usr/share/nginx/html

  app3:
    image: nginx
    volumes:
      - ./app3:/usr/share/nginx/html

  nginx-lb:
    image: nginx
    container_name: nginx-lb
    ports:
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf

At this point I finally started understanding why containers are so important.

Everything became isolated, reproducible, and easy to manage.

Understanding Load Balancing Through Nginx

This was probably the coolest part of the entire project.

The Nginx configuration looked like this:

events {}

http {

    upstream backend {
        server app1:80;
        server app2:80;
        server app3:80;
    }

    server {
        listen 80;

        location / {
            proxy_pass http://backend;
        }
    }

}

That tiny file completely changed how I viewed backend systems.

Instead of one server handling everything, Nginx distributes requests between multiple backend containers.

This is load balancing.

Not theory anymore.

An actual working system running inside my VM.

Running the Containers

Once everything was ready, I started the infrastructure:

docker compose up -d

The containers started successfully:

docker ps

And when I opened localhost inside the VM, different apps started appearing depending on which backend handled the request.

That moment felt genuinely satisfying because I wasn’t just running applications anymore.

I was running infrastructure.

NAT vs Bridged Adapter

This part confused me for hours.

The setup worked perfectly inside the VM, but Windows couldn’t access it initially.

That’s when I learned about NAT and Bridged networking.

NAT + Port Forwarding

NAT creates an isolated internal network for the VM.

The host machine cannot directly access the virtual machine unless ports are forwarded manually.

So I configured port forwarding in VirtualBox like this:

Host Port: 8080
Guest Port: 80

Which basically meant:

Windows localhost:8080 -> VM port 80

After configuring this properly, I could finally access the Nginx load balancer directly from Windows.

Bridged Adapter

Bridged mode felt even cooler.

Instead of staying inside an isolated virtual network, the VM became a real device on my local network and got its own IP address.

Something like:

ip a

Example IP:

10.93.5.4

Now I could directly access the server from Windows using:

http://10.93.5.4

That honestly felt insane the first time it worked.

Networking finally stopped feeling abstract.

Stress Testing With wrk

Now came the fun part.

Benchmarking the infrastructure.

I used wrk to generate traffic and test how the setup behaved with different numbers of backend containers behind the Nginx load balancer.

The command I used was:

wrk -t4 -c100 -d30 http://127.0.0.1

Where:

-t4 → 4 threads
-c100 → 100 concurrent connections
-d30 → run test for 30 seconds

The interesting part was comparing the performance between:

1 backend container
2 backend containers
3 backend containers

1 Container Test

Running 30s test @ http://127.0.0.1
4 threads and 100 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev
Latency    21.78ms   28.04ms 238.82ms   86.21%
Req/Sec     1.36k     1.06k    8.34k    75.55%

141456 requests in 30.05s, 34.13MB read

Socket errors: connect 0, read 0, write 0, timeout 102

Requests/sec:   4706.85
Transfer/sec:      1.14MB

With a single backend container, the setup still handled a surprisingly large amount of traffic. But the latency was noticeably higher and there were some request timeouts under heavy load.

2 Container Test

Running 30s test @ http://127.0.0.1
4 threads and 100 connections

Thread Stats   Avg      Stdev     Max   +/- Stdev
Latency    10.39ms   10.07ms 146.04ms   89.23%
Req/Sec     1.76k     1.01k    6.95k    82.19%

110705 requests in 30.10s, 26.71MB read

Socket errors: connect 0, read 0, write 0, timeout 100

Requests/sec:   3678.10
Transfer/sec:      0.89MB

Adding a second backend container reduced the average latency significantly. The load balancer was now distributing traffic across multiple services instead of pushing everything into a single container.

Even though the requests/sec result was lower during this run, the overall responsiveness and latency consistency improved.

3 Container Test

Thread Stats   Avg      Stdev     Max   +/- Stdev
Latency    13.85ms   11.87ms 247.23ms   87.54%
Req/Sec     1.74k   672.33     6.35k    75.85%

207100 requests in 30.03s, 49.97MB read

Requests/sec:   6895.35
Transfer/sec:      1.66MB

With three backend containers running behind Nginx, the setup handled the highest total request throughput.

This was the point where load balancing finally felt real to me.

Instead of one application trying to handle everything alone, requests were being distributed across multiple backend services. Even inside a small VM setup, you could clearly see how scaling horizontally changes system behavior.

What I Learned From Benchmarking

The most interesting part wasn’t just the numbers.

It was understanding the concept behind them.

Before this, “scaling” always sounded like some massive enterprise topic only huge companies cared about. But after running these tests manually, I finally understood the core idea:

Instead of making one server infinitely stronger, you can distribute traffic across multiple smaller services.

That’s basically the foundation of modern infrastructure.

What I Learned

The biggest thing I learned from this entire process is how much complexity modern platforms hide from developers.

When we deploy apps today, we rarely think about:

reverse proxies
ports
traffic routing
containers
networking
load balancing

But building things manually teaches an entirely different level of understanding.

I also realized infrastructure engineering feels very different from normal application development. A lot of the work is debugging, configuration, networking, and slowly understanding how systems communicate with each other.

Sometimes nothing works for hours.

Sometimes you break your entire Linux setup.

Sometimes networking feels cursed.

But when everything finally connects together correctly, it feels incredibly rewarding.

And honestly, building a self-hosted infrastructure server inside a Linux VM might be one of the coolest things I’ve learned so far.

Shipping Faster in Full-Stack Development (2026) with AI Orchestration

Ansh Dhanani — Wed, 28 Jan 2026 10:57:34 +0000

Why it matters?

lets say we have two guys. Where one is a giga chad coder guy who writes all code by himself and if he get stuck he just see docs. second is my type of guy who is a vibe-coder but not any other vibe-coder a Efficiency-level-V vibe-coder (I'll tell you why).

so here is a thing, when they both start a big project. the giga chad starts from scratch and write everything which is not a bad thing also he knows every whys of his code, but after all this talk about giga chad if you see at the other one, he had already completed 90% of project with all best practices.

how did he do that?

that's what you'll learn in this blog later.

so who is better?

there was a time when AI was not that impactful and people said its better to write slow and by own than from an AI assistance. but time is changed now yesterday an "open source model Kimi k2" by Moonshot AI just released and benchmark said it even better than claude opus 4.5. let me remind you claude opus 4.5 is a devil of coding complex logic.

so, what i'm trying to say is, if you really know how to orchestrate this models you can actually ship products faster and more importantly robust.

here's how you can do it too.

if you are a college student, you can get github pro using you college id and use some monthly free premium credits of claude sonnet 4.5 for free (thanks github). it also give other models but claude sonnet 4.5 is best among them.

now you have claude waiting to be used.

Lets dive into the best practices i used for best results.

Point 1: learn how to prompt efficiently.

when i was in early stage of learning ai agents for coding, i used to tell chatgpt "hey chatgpt fix this error." back then, it wasn't even able to get the problem to start with.

as i keep playing with it i come to know that you have to give full context about the problem or the thing you it to do. you have to keep in mind you have to give one prompt in which you have to write the clear problem and what path to follow and what not to too.

for example:

bad one:
“Fix the backend bug.”

Better prompt (structured):
“In our Express + Prisma backend, the POST /users endpoint fails when email already exists. Prisma throws a unique constraint error. We want to return a 409 response instead of crashing the server. Suggest a fix and include proper error handling patterns.”

remember to give full context weather it is about the error on browser console, typescript error ,compilation fails, refactor (very problematic) everything that you think it should know about everything. as a developer you must know how every thing works and why you choose what and all. you can't just let it do anything it wants, it will be just another useless slop and waste of credits.

you should utilized each and every feature of you IDE. many people says cursor is better. let me tell you i have tried cursor, windsurf, lazy vim, emacs(Salute to those who use this), Jetbrain(nice one) and of course my personal favorite VS code. and i can tell you that it doesn't actually matters that much.

talking about vs code features, you must use include files or folder in prompt feature(present in most IDEs) you want to change. you must be aware what you are give access, turns out you just expose your .env secrets and now openai asking you for 2421$ lol. also there are bunch of nice extentions which are very useful, you can check that out too.

Point 2: Never start from scratch.

if you are in any hackathon or starting any projects by your own (here i'm not talking about yt toutorials projects).

IT IS BETTER TO START WITH TEMPLATE.

you might be thinking that this is laziness. but let me tell you something, lazy smart coders are the best category of coders. They are the one who find short and best pathways using their laziness. funny but its fact.

so, you might ask me Ansh where can we find clean templates for starters,
as a developer if we talk about nextjs specifically, vercel have pretty nice templates. but you can find you perfect match from github too (i know it is little slow process but sometimes you can get a gem).

I’m participating in a hackathon at IIT Gandhinagar with my team, and we needed to create an admin dashboard for our problem statement. What I did was use a ready-made shadcn admin dashboard template. I then used Claude Desktop and connected it with Desktop MCP, writing a big prompt that detailed everything we wanted in a markdown.md file — basically everything about the problem, the tech stack, and the libraries to use. I kept an eye on it to make sure nothing went wrong. Now we have a clean project that doesn’t break.

Use Librarys.

let's say you have a very nice project you working on, and you have to add a feature, the problem is you don't know how to do it.

depending upon size and complexity of the feature, you sometime might not need any library, but if you have a big feature and you are supposed to handle many states at ones or anything complex, you must use a library rather than just asking ai to impliment it. it happens with me countless times that i asked it to implement a simple feature and it create all hooks and library fuctions inside component file, while i am having a separate clean structure to handle both types of files, but the ai doesn't know about my folder structure at all it just know that one component.

conclusion

The giga chad's way works for adding critical features to already going projects.

For project foundation and clean start you must follow efficient vibe coder way.

Thanks for reading.

HTTP Protocol Explained: The Complete Practical Guide for Developers

Ansh Dhanani — Sun, 28 Dec 2025 12:33:59 +0000

The HTTP Guide for Developers

A practical guide to HTTP. No fluff-just what you need to build better applications and debug faster.

What is HTTP?
Why HTTP Matters
HTTP Versions
HTTP Messages
HTTP Headers
HTTP Methods
Status Codes
Idempotency
CORS
Caching
Content Negotiation
Compression
Persistent Connections
Streaming
HTTPS

What is HTTP?

HTTP (HyperText Transfer Protocol) is the fundamental language that powers the entire web. Think of it as the universal translator that allows your browser, mobile apps, and backend services to communicate with servers across the internet. Every time you load a website, check your email, or scroll through social media, HTTP is working behind the scenes, shuttling data back and forth in a structured, predictable way.

At its core, HTTP operates on a beautifully simple principle: one device asks for something (the client), and another device responds with that thing (the server). This request-response model is what makes the web work. When you type a URL into your browser or your app fetches data from an API, that's an HTTP request flying across the network. The server processes that request and sends back an HTTP response containing the data you asked for-or an error message if something went wrong.

Here's what a basic interaction looks like:

Your Browser  →  "GET /users/123 HTTP/1.1 headers..."  →  Server
Your Browser  ←  "HTTP/1.1 200 OK headers... {data}"   ←  Server

The beauty of HTTP is that it's text-based (in HTTP/1.x), which means humans can actually read and understand what's being sent. This makes debugging infinitely easier compared to binary protocols where you're staring at gibberish. You can literally open your browser's developer tools right now, go to the Network tab, and watch HTTP messages flying back and forth in real-time. Try it it's oddly satisfying.

Understanding Statelessness

Now here's where HTTP gets interesting: it's stateless, meaning the server has zero memory of previous requests. Each request is treated as a completely fresh interaction, as if the server has never met you before. This might sound like a limitation at first and honestly, it can be frustrating when you're starting out but it's actually one of HTTP's greatest strengths.

Why? Because statelessness makes scaling trivial. If every request is independent, any server in a cluster can handle any request. You don't need "sticky sessions" or complex state synchronization between servers. User sends Request A to Server 1? Cool. Request B goes to Server 2? Also fine. The servers don't need to coordinate because there's no state to keep track of.

Let's see this in action:

Request 1: GET /cart HTTP/1.1
Response: "What cart? And first of all, who are you?"

Request 2: GET /cart HTTP/1.1  
Response: "Still don't know you. Who dis?"

So how do websites "remember" you? They cheat. We add state back using cookies or tokens. The server gives you a unique identifier (like a session ID), and you send it with every request:

GET /cart HTTP/1.1
Cookie: session_id=abc123

Response: "Ah yes, session abc123! Here's your cart with 3 items."

The server looks up your session ID in a database (like Redis) and retrieves your cart data. Boom stateful behavior built on a stateless protocol. It's like showing your ID card every time you enter a building instead of the security guard remembering your face.

The Request-Response Contract

HTTP follows a strict contract: the client always initiates, the server always responds. The server never sends unsolicited data (that's what WebSockets are for). This asymmetry keeps things simple and predictable. Every HTTP conversation follows the same dance:

Client opens connection (usually TCP on port 80 or 443)
Client sends request ("I want this resource")
Server processes request (database queries, business logic, etc.)
Server sends response ("Here's what you asked for" or "Nope, can't do that")
Connection closes (or stays open for more requests)

This predictable pattern is why HTTP is so reliable. There's no ambiguity about who's talking when, no complex state machines to debug. Request comes in, response goes out. Simple. Elegant. Effective.

Why HTTP Matters

Imagine trying to build the internet without a standard protocol. Every website would invent its own communication rules, every API would require a custom client, and developers would spend more time writing protocol adapters than building actual features. It would be absolute chaos like trying to have a conversation where everyone speaks a different language with no translators available.

HTTP solved this problem by giving us a universal standard. Whether you're talking to GitHub's API, Stripe's payment system, or your own backend, the protocol is identical. Same request format, same response structure, same status codes, same headers. This standardization is so fundamental to the web that we take it for granted, but it's genuinely revolutionary. One fetch() call works for thousands of different services:

// Same exact pattern, completely different services
fetch('https://api.github.com/users/octocat')
fetch('https://api.stripe.com/v1/charges')  
fetch('https://yourcompany.com/api/orders')

No special configuration, no custom libraries, no protocol negotiation just HTTP. This is the magic of standardization.

The Hidden Benefits You Get for Free

When you use HTTP, you're not just getting a protocol you're getting an entire ecosystem that has evolved over 30+ years. Built-in caching means browsers and CDNs automatically cache your responses if you set the right headers. You don't need to build a caching layer; it's already there, waiting for you to use it:

Cache-Control: public, max-age=3600

That single header tells every intermediary between your server and the user to cache this response for an hour. Millions of requests saved, zero extra code written. That's leverage.

Then there's the tooling ecosystem. Because HTTP is standardized, we have incredible tools that work with any HTTP service: browser DevTools show you every request with full headers and timing information, Postman lets you craft complex API requests with a GUI, cURL gives you command-line superpowers, and monitoring tools like DataDog can analyze HTTP traffic patterns across your entire infrastructure. All of this exists because HTTP is standardized.

Statelessness: The Scalability Superpower

Here's where HTTP's stateless design becomes a massive advantage. In the old days, servers used to maintain session state in memory. This worked fine for small applications, but it created a nightmare for scaling. If Server A knew about your session but Server B didn't, load balancers had to implement "sticky sessions" to route all your requests to Server A. This meant uneven load distribution, difficult failover, and complex infrastructure.

HTTP's statelessness cuts through this complexity like a hot knife through butter. Because each request is independent, any server can handle any request. Your first request goes to Server 1 in California? Cool. Your next request goes to Server 2 in Virginia? Also fine. Neither server needs to know about the other, and both can process your request by looking up your session token in a shared database.

This architecture is simple, reliable, and scales horizontally. Need more capacity? Spin up more servers. No complex state synchronization, no session migration, no headaches. This is why modern web applications can serve millions of users HTTP's stateless design makes it almost trivially easy to scale.

When you combine all these benefits universal compatibility, free caching, incredible tooling, and effortless scaling you start to understand why HTTP has dominated the web for three decades and shows no signs of going away. It's not just a protocol; it's the foundation that makes the entire internet economy possible.

HTTP Versions: Evolution Through Real Problems

What an actual http request/response look like?

HTTP has evolved through three major versions, each solving critical performance bottlenecks that became apparent as the web grew. Understanding this evolution isn't just history it's understanding the performance characteristics and limitations of the systems you're building today.

HTTP/1.1 (1997) - The Foundation

HTTP/1.1 was the protocol that took the web mainstream. It introduced persistent connections (keep-alive), which meant you could send multiple requests over a single TCP connection instead of opening a new connection for every request. This was huge establishing a TCP connection involves a three-way handshake that takes at least one round-trip time, and with the old HTTP/1.0 model, a page with 50 resources meant 50 TCP handshakes. That's a lot of wasted time and bandwidth.

With persistent connections, you connect once and reuse that connection:

TCP 3 way handshake

Connect (TCP handshake: 50-100ms)
Request 1 → Response 1
Request 2 → Response 2  
Request 3 → Response 3
[... many more requests ...]
Disconnect

HTTP/1.1 also gave us chunked transfer encoding, which lets servers start sending data before they know the total size. This is critical for streaming use cases imagine trying to stream a video if you had to buffer the entire file first to calculate Content-Length. Chunked encoding solved this by letting the server send data in pieces with each piece prefixed by its size.

But HTTP/1.1 had a fatal flaw: head-of-line blocking. Requests were still processed sequentially over each connection. Even though you had persistent connections, you couldn't truly parallelize requests. If Request 1 took 5 seconds, Request 2 had to wait those full 5 seconds before it could even start. Browsers worked around this by opening multiple connections (typically 6 per domain), but this was a hack, not a solution.

HTTP/2 (2015) - The Performance Revolution

HTTP/2 fundamentally changed how data flows over a connection with multiplexing the ability to send multiple requests and receive multiple responses simultaneously over a single connection. Instead of requests waiting in line, they all fly at once:

┌──────────────────────────────────┐
│   Single TCP Connection          │
├──────────────────────────────────┤
│ Stream 1: GET /style.css     ───▶│
│ Stream 2: GET /script.js     ───▶│
│ Stream 3: GET /image.png     ───▶│
│ Stream 1: 200 OK [CSS data]  ◀───│
│ Stream 3: 200 OK [IMG data]  ◀───│ (came back first!)
│ Stream 2: 200 OK [JS data]   ◀───│
└──────────────────────────────────┘

Notice how responses can come back out of order Stream 3 finishes before Stream 2, and that's fine. No more head-of-line blocking at the HTTP layer. This change alone typically improves page load times by 30-50%.

HTTP/2 also introduced header compression using HPACK. HTTP/1.1 sent the same headers repeatedly with every request, wasting kilobytes:

GET /api/users/1 HTTP/1.1
Host: api.example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36...
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
Accept: application/json
Cookie: session=abc123; prefs=dark_mode; ...

GET /api/users/2 HTTP/1.1
Host: api.example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36...
[... exact same headers repeated ...]

HTTP/2 compresses these headers and remembers them across requests, so subsequent requests only send the differences. This saves significant bandwidth, especially for APIs where you make many requests with identical headers.

The catch: HTTP/2 still runs over TCP, and TCP has its own head-of-line blocking at the packet level. If a single TCP packet is lost, everything behind it stalls while that packet is retransmitted even unrelated streams. This is especially painful on mobile networks with packet loss.

HTTP/3 (2022) - Breaking Free from TCP

HTTP/3 made the radical decision to abandon TCP entirely and use QUIC over UDP instead. This sounds crazy UDP is the "unreliable" protocol, right? But QUIC implements reliability features on top of UDP while fixing TCP's fundamental limitations.

The key innovation is per-stream head-of-line blocking. In HTTP/2, a lost packet blocks all streams. In HTTP/3, a lost packet only blocks the stream it belongs to:

Stream 1: [Packet 1] [LOST!] [Packet 3] → Stalls waiting for Packet 2
Stream 2: [Packet 1] [Packet 2] [Packet 3] → Keeps flowing!
Stream 3: [Packet 1] [Packet 2] → Keeps flowing!

This is massive for mobile users. On spotty connections with 5% packet loss, HTTP/3 can be dramatically faster than HTTP/2 because lost packets don't stall unrelated requests.

HTTP/3 also has built-in encryption with TLS 1.3 integrated into QUIC. There's no separate TLS handshake it's all one negotiation. Even better, HTTP/3 supports 0-RTT resumption, meaning if you've connected before, your first request can include data immediately without waiting for handshakes. This is perfect for mobile apps that frequently reconnect.

Finally, HTTP/3 has connection migration. When your phone switches from WiFi to cellular, TCP connections die and need to be reestablished. QUIC connections use connection IDs instead of IP addresses, so the connection seamlessly migrates to the new network. Your video keeps streaming, your API calls keep flowing no interruption.

Current reality: HTTP/3 is used by Google (all properties), Facebook, Cloudflare, and other major services. Browser support is universal. But server adoption is still growing because it requires new infrastructure. Most applications still run on HTTP/1.1 or HTTP/2, and that's fine they work well. HTTP/3 is an optimization, not a requirement.

What This Means for You

Here's the practical takeaway: you usually don't choose the HTTP version your hosting provider, CDN, or load balancer handles it. Your application code doesn't change between versions. The same Express/Flask/Spring Boot code works with all versions.

What matters is understanding that:

HTTP/1.1 is sequential (slow with many requests)
HTTP/2 is parallel (fast with many requests)
HTTP/3 is resilient (fast even on bad networks)

When debugging performance issues, check which version is being used. If you're on HTTP/1.1 and making 100 requests to load a page, upgrading to HTTP/2 might be your biggest performance win. If your mobile users complain about slow connections, HTTP/3 might help. But if you're making 5 API calls and they're already fast, the version doesn't matter much.

HTTP Messages

Every HTTP interaction uses this format:

┌──────────────┐
│ Start Line   │ ← Method + Path or Status
├──────────────┤
│ Headers      │ ← Metadata
│ Header: value│
├──────────────┤
│ (blank line) │ ← Required separator
├──────────────┤
│ Body         │ ← Optional data
└──────────────┘

Request Example

POST /api/users HTTP/1.1
Host: api.example.com
Content-Type: application/json
Content-Length: 27

{"name":"Alice","age":30}

Parts:

Request line: POST /api/users HTTP/1.1
Headers: Host, Content-Type, Content-Length
Blank line: Required!
Body: JSON data

Response Example

HTTP/1.1 201 Created
Content-Type: application/json
Location: /api/users/123

{"id":123,"name":"Alice"}

Parts:

Status line: HTTP/1.1 201 Created
Headers: Content-Type, Location
Blank line
Body: Created resource

HTTP Headers

Headers are key-value pairs that provide metadata.

Critical Request Headers

Host (required)

Host: api.example.com

Which server to route to

Authorization

Authorization: Bearer eyJhbGc...

Authentication credentials

Content-Type

Content-Type: application/json

Format of request body

Accept

Accept: application/json

What response formats you'll accept

Cookie

Cookie: session_id=abc123

Send stored cookies

Critical Response Headers

Content-Type

Content-Type: application/json

Format of response body

Set-Cookie

Set-Cookie: session_id=abc123; HttpOnly; Secure

Store cookie on client

Cache-Control

Cache-Control: public, max-age=3600

Caching instructions

Location

Location: /api/users/123

Redirect or created resource URL

Security Headers (Must-Have)

Strict-Transport-Security

Strict-Transport-Security: max-age=31536000

Force HTTPS

Content-Security-Policy

Content-Security-Policy: default-src 'self'

Prevent XSS attacks

X-Content-Type-Options

X-Content-Type-Options: nosniff

Prevent MIME sniffing

X-Frame-Options

X-Frame-Options: DENY

Prevent clickjacking

Cookie Attributes (Security)

Set-Cookie: session=abc; HttpOnly; Secure; SameSite=Strict

HttpOnly: JavaScript can't access (prevents XSS)
Secure: HTTPS only (prevents MITM)
SameSite=Strict: Blocks CSRF attacks

HTTP Headers, Reference Table

Headers provide metadata and control behavior.

Header	Explanation	Common Values	Usage
Host	Target server	`api.example.com`	Routing
Authorization	Credentials	`Bearer <token>`, `Basic <base64>`	Authentication
Content-Type	Body format	`application/json`, `text/html`, `multipart/form-data`	Parsing
Accept	Desired response	`application/json`, `text/html`, `/`	Content negotiation
Accept-Language	Preferred language	`en-US`, `fr-FR`, `hi-IN`	Localization
User-Agent	Client info	Browser, curl, Postman	Analytics, debugging
Cookie	Client state	`session_id=abc123`	Sessions
Set-Cookie	Store cookie	`HttpOnly; Secure; SameSite=Strict`	Authentication
Cache-Control	Caching	`no-cache`, `max-age=3600`, `public`	Performance
Location	Redirect or new resource	`/api/users/123`	Navigation
Strict-Transport-Security	Force HTTPS	`max-age=31536000`	Security
Content-Security-Policy	Prevent XSS	`default-src 'self'`	Security
X-Content-Type-Options	Prevent MIME sniffing	`nosniff`	Security
X-Frame-Options	Clickjacking protection	`DENY`, `SAMEORIGIN`	Security
Transfer-Encoding	Body transfer	`chunked`, `gzip`	Streaming/Compression
Connection	Connection persistence	`keep-alive`, `close`	Performance
Access-Control-Allow-Origin	CORS origin	`*`, `https://example.com`	Cross-origin requests
Access-Control-Allow-Methods	Allowed HTTP methods	`GET, POST, PUT, DELETE, OPTIONS`	CORS control
Access-Control-Allow-Headers	Allowed headers in CORS	`Authorization, Content-Type`	CORS control

HTTP Methods

Method	Purpose	Safe	Idempotent	Body
GET	Retrieve resource	✅	✅	❌
POST	Create resource	❌	❌	✅
PUT	Replace resource	❌	✅	✅
PATCH	Update resource	❌	⚠️	✅
DELETE	Remove resource	❌	✅	❌
OPTIONS	Check capabilities	✅	✅	❌

Status Codes

### you are not supposed to remember all.

Mostly used Codes.

Code	Meaning	When to Use
200	OK	Standard success
201	Created	Resource created
204	No Content	Success without body
301	Moved Permanently	Permanent redirect
304	Not Modified	Use cached resource
400	Bad Request	Invalid input
401	Unauthorized	Auth required
403	Forbidden	No permission
404	Not Found	Resource missing
429	Too Many Requests	Rate limited
500	Internal Server Error	Server issue
502	Bad Gateway	Upstream issue
503	Service Unavailable	Temporary downtime

CORS

CORS (Cross-Origin Resource Sharing): Browser security that blocks cross-origin requests by default.

The Problem

// You're on https://frontend.com
fetch('https://api.backend.com/data')
// ❌ Blocked by browser!

The Solution

Server sends CORS headers:

Access-Control-Allow-Origin: https://frontend.com
Access-Control-Allow-Methods: GET, POST
Access-Control-Allow-Headers: Content-Type

Simple vs Preflight Requests

Simple request (no preflight):

GET /api/data
Origin: https://frontend.com

Response:
Access-Control-Allow-Origin: https://frontend.com

Preflight request (checks first):

OPTIONS /api/data
Origin: https://frontend.com
Access-Control-Request-Method: POST
Access-Control-Request-Headers: Content-Type

Response:
Access-Control-Allow-Origin: https://frontend.com
Access-Control-Allow-Methods: POST
Access-Control-Allow-Headers: Content-Type
Access-Control-Max-Age: 86400

Then actual POST request proceeds...

Triggers preflight:

Methods: PUT, DELETE, PATCH
Custom headers
Content-Type: application/json

Common CORS Setup

// Node.js/Express
app.use((req, res, next) => {
  res.header('Access-Control-Allow-Origin', 'https://frontend.com');
  res.header('Access-Control-Allow-Methods', 'GET,POST,PUT,DELETE');
  res.header('Access-Control-Allow-Headers', 'Content-Type,Authorization');
  res.header('Access-Control-Allow-Credentials', 'true');

  if (req.method === 'OPTIONS') {
    return res.sendStatus(200);
  }
  next();
});

Caching

Caching saves bandwidth and improves speed.

Cache-Control Header

Don't cache (sensitive data):

Cache-Control: private, no-store

Cache for 1 hour:

Cache-Control: public, max-age=3600

Cache but revalidate:

Cache-Control: public, max-age=0, must-revalidate

Cache forever (static assets):

Cache-Control: public, max-age=31536000, immutable

ETag (Smart Caching)

First request:

GET /api/users/123

200 OK
ETag: "abc123"
{"id":123,"name":"Alice"}

Later request:

GET /api/users/123
If-None-Match: "abc123"

304 Not Modified
(no body - use cache)

If data changed:

200 OK
ETag: "xyz789"
{"id":123,"name":"Alice Updated"}

Cache Strategy by Content Type

# HTML pages
Cache-Control: public, max-age=0, must-revalidate

# API responses
Cache-Control: private, no-cache

# Static assets (CSS, JS, images with hash in filename)
Cache-Control: public, max-age=31536000, immutable

# User-specific data
Cache-Control: private, max-age=300

# Sensitive data
Cache-Control: private, no-store

Content Negotiation

Client and server agree on format/language.

Accept Header

GET /api/users
Accept: application/json

Response:
Content-Type: application/json
{"users":[...]}

GET /page
Accept: text/html

Response:
Content-Type: text/html
<html>...</html>

Accept-Language

Accept-Language: es-MX, es;q=0.9, en;q=0.8

Translation: "I prefer Mexican Spanish, any Spanish is fine, English as backup"

Quality Values (q)

Accept: application/json, application/xml;q=0.9, */*;q=0.8

q=1.0 (default): Preferred
q=0.9: Less preferred
q=0.8: Least preferred

Compression

Compression reduces payload size significantly.

Accept-Encoding

GET /api/data
Accept-Encoding: gzip, br

Response:
Content-Encoding: gzip
[compressed data]

Compression Algorithms

gzip (universal support)

Content-Encoding: gzip

Brotli (better compression, modern browsers)

Content-Encoding: br

Real Impact

Original: 1.2 MB
Gzipped:  300 KB
Savings:  75%

Always compress: Text, JSON, HTML, CSS, JavaScript

Don't compress: Images (already compressed), videos

Persistent Connections

Keep connections open for multiple requests.

HTTP/1.1 Keep-Alive

Connection: keep-alive

┌─────────────────────────────────┐
│ Connect once                     │
│ Request 1 → Response 1           │
│ Request 2 → Response 2           │
│ Request 3 → Response 3           │
│ Disconnect                       │
└─────────────────────────────────┘

vs Old way:

Connect → Request 1 → Response 1 → Disconnect
Connect → Request 2 → Response 2 → Disconnect
Connect → Request 3 → Response 3 → Disconnect

Benefit: Eliminates TCP handshake overhead (50-100ms per connection)

HTTP/2 & HTTP/3

Persistent connections built-in. No special headers needed.

Streaming

Send data without knowing total size.

Chunked Transfer Encoding

Transfer-Encoding: chunked

1a\r\n
This is the first chunk\r\n
14\r\n
This is the second\r\n
0\r\n
\r\n

Each chunk: size in hex + \r\n + data + \r\n

Last chunk: 0\r\n\r\n

Real-World Uses

Streaming API responses:

// Server-sent events
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Transfer-Encoding', 'chunked');

setInterval(() => {
  res.write(`data: ${JSON.stringify({time: Date.now()})}\n\n`);
}, 1000);

Streaming logs:

curl https://api.example.com/logs/stream
# Receives log lines as they're generated

Video streaming:
Sends chunks as they're encoded/available

HTTPS

HTTPS = HTTP + TLS encryption

Why HTTPS?

Without HTTPS:

You → "password123" → [Anyone listening] → Server

With HTTPS:

You → [encrypted gibberish] → Server

Even if intercepted, data is unreadable.

What HTTPS Provides

Encryption: Data can't be read
Authentication: Server is who they claim
Integrity: Data can't be modified

TLS Handshake (Simplified)

Client → Server: "Let's use HTTPS"
Server → Client: "Here's my certificate"
Client verifies certificate
Client → Server: [Encrypted session key]
Both use session key for encryption

Certificate Verification

Browser checks:

Certificate not expired?
Issued by trusted CA?
Domain matches?
Not revoked?

If all pass: 🔒 Green lock

If fail: ⚠️ Security warning

HTTP → HTTPS Upgrade

Force HTTPS:

HTTP/1.1 301 Moved Permanently
Location: https://example.com
Strict-Transport-Security: max-age=31536000

HSTS (HTTP Strict Transport Security):

Strict-Transport-Security: max-age=31536000; includeSubDomains

Browser automatically uses HTTPS for all requests to this domain.

Debugging HTTP

Browser DevTools

F12 → Network Tab

Shows:

All requests/responses
Headers
Timing
Payload size

cURL (Command Line)

# Basic request
curl https://api.example.com/users

# Show headers
curl -v https://api.example.com/users

# POST with data
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name":"Alice"}'

# Follow redirects
curl -L https://example.com

# Save response
curl -o output.json https://api.example.com/data

Common Issues

CORS errors:

Check: Access-Control-Allow-Origin header
Fix: Add header on server

401 Unauthorized:

Check: Authorization header present and valid
Fix: Include valid token

304 Not Modified confusion:

Remember: 304 means use cached version (not an error!)

Unexpected status codes:

Check: What server is actually returning
Use: curl -v to see full response

Quick Reference

Essential Headers Cheat Sheet

Request:

Host: api.example.com              (required)
Authorization: Bearer <token>       (auth)
Content-Type: application/json      (body format)
Accept: application/json            (wanted format)

Response:

Content-Type: application/json      (body format)
Cache-Control: max-age=3600         (caching)
Access-Control-Allow-Origin: *      (CORS)

Security:

Strict-Transport-Security: max-age=31536000
Content-Security-Policy: default-src 'self'
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

Best Practices Summary

API Design

Use appropriate HTTP methods
Return meaningful status codes
Include proper headers (Content-Type, etc.)
Implement caching where appropriate
Version your API (URL or header)

Security

Always use HTTPS in production
Set security headers (HSTS, CSP, etc.)
Use HttpOnly, Secure cookies
Implement rate limiting
Validate all inputs

Performance

Enable compression (gzip/brotli)
Use appropriate caching headers
Leverage HTTP/2 if available
Minimize payload size
Use persistent connections

Debugging

Check Network tab first
Use curl for API testing
Include request IDs for tracing
Log headers in development
Test with real network conditions

Remember: HTTP is simple at its core requests and responses with headers and bodies. Everything else is optimization and security built on top of that foundation.

Master these fundamentals, and you'll understand what's happening in any HTTP-based system.